0

LONG TIME RUNNING DEVICES STOP CONNECTING TO MQTT
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

LONG TIME RUNNING DEVICES STOP CONNECTING TO MQTT

by weaVaer on Thu Sep 12, 2019 2:15 pm

We have a couple of Huzzah feather 8266s that has been running quite a while on Adafruit IO without any problem at all.
Looking into Adafruit IO dashboard I can see no new record created after ~2019/09/10 11:46:13am (Adafruit IO dashboard time)
on any feeds and this is true for all devices.
Examining trace from Serial output shows that attempting to connect MQTT <io.adafruit.com> : [ERROR] => (-1)Connection failed
The AIO key is valid when checked with a curl script.
Other than nothing has been changed on the devices, no new sketch etc for nearly a year on them.
ESP 8266 core : \packages\esp8266\hardware\esp8266\2.3.0\
Adafruit MQTT Library version=0.20.1
Adafruit IO Arduino library version=1.0.1

We suspect a platform change on your side for ESP8266 core 2.5.0 support has broken something and you are not backwards compatible with something.
And a recent notice here about upgrading things just happen to coincide with our situation.
(It is a well known fact that all the libs that is using pre 2.5.x core have been effected from the upgrade to 2.5.x. Many things dont work and breaks with weird memory errors, crashes etc. if you try to run your code on a 2.5.x core installed.)
But we don't understand how you needed to change the MQTT connect function of your service such that a ESP8266 with a sketch compiled with older versions of ESP8266 core and Adafruit libraries can not connect to your service any more.

Also kindly take note that we didn't have any problems when the core went up to 2.4.x after we installed our sketches on the devices nearly a year ago on a premium account.
(we stayed at 2.3.0)

Need your support in this case.
Thanks

weaVaer
 
Posts: 36
Joined: Mon May 01, 2017 1:40 pm

Re: LONG TIME RUNNING DEVICES STOP CONNECTING TO MQTT

by brubell on Thu Sep 12, 2019 2:50 pm

Hi weaVaer, let's get those devices back on Adafruit IO!

This is likely not an issue with the MQTT server or your hardware - but possibly an issue with your account settings. Could you navigate to http://io.adafruit.com/support and click the Contact Support button to get in direct contact with the Adafruit IO team:
contact_io.png
contact_io.png (89.97 KiB) Viewed 102 times


Also, you listed a few different versions of the ESP8266 BSP (core) in your post. Which version (or versions) is giving you trouble with connecting to Adafruit IO? 2.3.x?

- brent r.

brubell
 
Posts: 496
Joined: Fri Jul 17, 2015 10:33 pm

Re: LONG TIME RUNNING DEVICES STOP CONNECTING TO MQTT

by weaVaer on Fri Sep 13, 2019 2:24 am

Thank you, will do that.

p.s. : Current ESP8266 core version on our sketches which recently ceased to work altogether is 2.3.0 (SDK version 2.0.0)

weaVaer
 
Posts: 36
Joined: Mon May 01, 2017 1:40 pm

Re: LONG TIME RUNNING DEVICES STOP CONNECTING TO MQTT

by abachman on Fri Sep 13, 2019 11:40 am

Hi weaVaer,


Can you post a summarized version of some code using the old libraries that isn't working anymore?

We deployed an update to our MQTT server briefly on 2019/09/10 (Tuesday) and then reverted it after about 45 minutes due to some issues with the server process rebooting unexpectedly. On Thursday we redeployed the fixed broker and it's running stable now. If your devices were having trouble late Tuesday through Wednesday, then it's likely it's a problem on your end.

We meticulously track statistics on our MQTT brokers (connections, publish events, successful auth, failed auth, errors, etc.) and everything is running normally, statistically speaking. For example, this is the running active connections graph spread over the last week, you can kind of see two blips where we deployed, but it's otherwise stable:
Screen Shot 2019-09-13 at 11.15.21 AM.png
Screen Shot 2019-09-13 at 11.15.21 AM.png (34.74 KiB) Viewed 66 times


The updates weren't intended to change MQTT support at all, but it's possible one of the underlying libraries we're using brought the service into closer compliance with the MQTT spec. If we can replicate the connection failure then we'd be able to trace the issue and take a look at our broker.

If you have a lot of devices, it's possible they were attempting to reconnect fast enough to lock your account out of the MQTT server, but that lock only lasts a minute. They'd also be locking every device out, not just devices with older firmware.

I have a sketch running successfully right now just using the ESP board pkg @ 2.3.0 and Adafruit MQTT library @ 0.20.1:
Code: Select all | TOGGLE FULL SIZE
#define WLAN_SSID       "---"
#define WLAN_PASS       "---"
#define AIO_SERVER      "io.adafruit.com"
#define AIO_SERVERPORT  1883
#define AIO_USERNAME  "---"
#define AIO_KEY       "---"
#define AIO_FEED_KEY  "example.counter-1"

#include <ESP8266WiFi.h>
#include "Adafruit_MQTT.h"
#include "Adafruit_MQTT_Client.h"

WiFiClient client;
Adafruit_MQTT_Client mqtt(&client, AIO_SERVER, AIO_SERVERPORT, AIO_USERNAME, AIO_USERNAME, AIO_KEY);
Adafruit_MQTT_Subscribe counter = Adafruit_MQTT_Subscribe(&mqtt, AIO_USERNAME "/feeds/" AIO_FEED_KEY, MQTT_QOS_1);

void setup() {
  Serial.begin(115200);
  delay(10);
  WiFi.begin(WLAN_SSID, WLAN_PASS);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
  }
  Serial.println("wifi connected");
  counter.setCallback([](char *value, uint16_t len) {
    Serial.print("received <- ");
    Serial.println(value);
  });
  mqtt.subscribe(&counter);
}

void loop() {
  MQTT_connect();
  mqtt.processPackets(10000);
  if(!mqtt.ping()) {
    mqtt.disconnect();
  }
}

void MQTT_connect() {
  int8_t ret;
  if (mqtt.connected()) {
    return;
  }
  Serial.print("Connecting to MQTT... ");
  uint8_t retries = 100;
  while ((ret = mqtt.connect()) != 0) { // connect will return 0 for connected
       Serial.println(mqtt.connectErrorString(ret));
       Serial.println("Retrying MQTT connection in 10 seconds...");
       mqtt.disconnect();
       delay(10000);  // wait 10 seconds
       retries--;
       if (retries == 0) {
         while (1);
       }
  }
}


If you have a sample of the way you're integrating the old Adafruit IO Arduino library (@ 1.0.1 it was HTTP only), I'd like to add that too to verify it.

Bottom line: we can't replicate it and our data isn't showing a widespread issue, unfortunately :( If you can get in touch with us through the support form Brent linked to, it would be helpful to get some more detailed account info to help us trace the issue. Things like usernames the devices are connecting under, the IP address they're connecting from, the IO key they're attempting to use, ... basically stuff that isn't okay for a public forum.


- adam b.

abachman
 
Posts: 328
Joined: Mon Feb 01, 2010 12:48 pm

Re: LONG TIME RUNNING DEVICES STOP CONNECTING TO MQTT

by weaVaer on Fri Sep 13, 2019 2:34 pm

Thanks for your prompt response.
Heard you loud and clear.
Switching channels.

@abachman

p.s. about the implementation of 1.0.1, yeah that might be where the problem lies.
I was using a custom wrap-up of this library (to make it work my way :) back then, more than a year ago.
These were a couple of experimental devices anyway, they didn't see a firmware upgrade and just kept on working until a couple of days ago.
I'll follow this lead, TY, and will gladly share the implementation if desired.

weaVaer
 
Posts: 36
Joined: Mon May 01, 2017 1:40 pm

Re: LONG TIME RUNNING DEVICES STOP CONNECTING TO MQTT

by weaVaer on Mon Sep 16, 2019 5:00 pm

The sketch below demonstrates that Adafruit MQTT connections on a ESP8266 WiFiClientSecure client fails since 9/10/2019.

ESP 8266 core : 2.3.0 (SDK version 2.0.0)
Adafruit MQTT Library version : 0.20.1

Code: Select all | TOGGLE FULL SIZE

/////////////////////////////////////////////////////////////////////////////////////

    #define USE_SSL true  // if this is 'true' then we will use 'WiFiClientSecure' & port:8883
                          // instead of 'WiFiClient' and port:1883, which will cause
                          // mqtt.connect() to fail with error (-1)
                          // then mqtt.disconnect() also fails with the message
                          //  {Connection failed in sendPacket}{Unable to send disconnect packet}

/// WiFi parameters /////////////////////////////////////////////////////////////////
   
    #define WLAN_SSID       ""  // <= fill in your wifi stuff
    #define WLAN_PASS       ""  // <= and here

/// Adafruit parameters /////////////////////////////////////////////////////////////   
   
    #define AIO_SERVER      "io.adafruit.com"
   
    #if (USE_SSL)
      #define AIO_SERVERPORT  8883
    #else
      #define AIO_SERVERPORT  1883
    #endif 
   
    #define AIO_USERNAME    ""  // <= fill in your Adafruit account stuff
    #define AIO_KEY         ""  // <= and here
    #define AIO_FEED_KEY    ""  // <= and here

/// libraries //////////////////////////////////////////////////////////////////////   

    // ESP8266 core : 2.3.0 (SDK version 2.0.0)
    #include <ESP8266WiFi.h>
   
    // Adafruit MQTT Library version : 0.20.1
    #include "Adafruit_MQTT.h"
    #include "Adafruit_MQTT_Client.h"

////////////////////////////////////////////////////////////////////////////////////   
   
    #if (USE_SSL)
      WiFiClientSecure client;
    #else
      WiFiClient client;
    #endif 
   
    Adafruit_MQTT_Client mqtt(&client, AIO_SERVER, AIO_SERVERPORT, AIO_USERNAME, AIO_USERNAME, AIO_KEY);
    Adafruit_MQTT_Subscribe counter = Adafruit_MQTT_Subscribe(&mqtt, AIO_USERNAME "/feeds/" AIO_FEED_KEY, MQTT_QOS_1);


////////////////////////////////////////////////////////////////////////////////////   
   

    void setup() {
      Serial.begin(115200);
      delay(100);
      Serial.println();
     
      WiFi.begin(WLAN_SSID, WLAN_PASS);
      while (WiFi.status() != WL_CONNECTED) {
        delay(500);
      }
      Serial.println("# wifi connected");
      Serial.println();
     
      counter.setCallback([](char *value, uint16_t len) {
        Serial.print("received <- ");
        Serial.println(value);
      });
      mqtt.subscribe(&counter);
      Serial.println("# mqtt subscribed");
      Serial.println();
   }


    void loop() {
      MQTT_connect();
      Serial.print("# ");
      mqtt.processPackets(5000);
      if(!mqtt.ping()) {
        mqtt.disconnect();
      }
    }



    void MQTT_connect() {
      int8_t ret;
      if (mqtt.connected()) {
        return;
      }
      Serial.print("# Connecting to MQTT (");
      #if (USE_SSL)
        Serial.print("with SSL)");
      #else
        Serial.print("no SSL)");
      #endif   
      Serial.print(" ... ");
     
      uint8_t retries = 100;
      while ((ret = mqtt.connect()) != 0) { // connect will return 0 for connected
           Serial.println(mqtt.connectErrorString(ret));
           Serial.println();
           Serial.println("# Retrying MQTT connection in 10 seconds...");
           mqtt.disconnect();
           Serial.println();           
           delay(10000);  // wait 10 seconds
           retries--;
           if (retries == 0) {
             while (1);
           }
      }
      Serial.println(" connected");
      Serial.println();
    }

weaVaer
 
Posts: 36
Joined: Mon May 01, 2017 1:40 pm

Please be positive and constructive with your questions and comments.