0

Suddenly only first AIO Connection works
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Suddenly only first AIO Connection works

by haku15 on Fri Nov 02, 2018 8:07 pm

Hi,
my Feather M0 WiFi has been succesfully posting data every few minutes to AIO Feeds for many weeks until sometime in Sept or Oct 2018. Then it started posting only on the first connection to AIO, with following connections returning the AIO_STATUS "Network connection failed." (AIO_NET_CONNECT_FAILED).
I use Adafruit_IO_Arduino 2.7.1 and lowPowerMode for the WINC1500 (but I also tried deactiving that)
The code to post to AIO is following; for troubleshooting, I just loop through this method every 10 sec:
Code: Select all | TOGGLE FULL SIZE
const unsigned long AIO_TIMEOUT = 40000;

void sendDataToIO(){

  unsigned long aioTimer = 0;
 
  io.connect();
  aioTimer = millis();
  while(io.status() < AIO_CONNECTED && millis() - aioTimer < AIO_TIMEOUT) { 
    #ifdef DEBUG_ON
    Serial.print(".");
    #endif
    delay(200); 
  }
  #ifdef DEBUG_ON
  Serial.print(F("aio_timer:"));
  Serial.println(aioTimer);
  Serial.println(F("aio status"));
  // we are connected
  Serial.println();
  Serial.println(io.statusText());
  #endif

  if(io.status() >= AIO_CONNECTED){
    io.run();
 
    extTempFeed->save(temp_ext);
    extHumFeed->save(hum_ext);
    extVcapFeed->save(capVoltage);
    intTempFeed->save(inTempRounded);
 
   
  }
   delay(500); //allow wifi to finish
   
}

Any idea?
Thanks for you help
Sincerely

haku15
 
Posts: 67
Joined: Mon Jun 25, 2012 12:45 pm

Re: Suddenly only first AIO Connection works

by haku15 on Mon Nov 05, 2018 2:03 pm

I did further testing: it looks like, consecutive io.connect() are not working anymore.
So I modified the code, to check if io.status() < AIO_CONNECTED: if yes, then io.connect(), otherwise just execute io.run().
The problem is not yet solved: the change above works if I post data every 10 secs, but not every 15minutes.
The wifi module is always on ( low power mode) and the ping that I execute to google.com after the AIO data post is always successfull, so the problems is not the wifi module.
Free RAM is also plenty available.

Code: Select all | TOGGLE FULL SIZE
void sendDataToIO(){

  unsigned long aioTimer = 0;

  if(io.status() < AIO_CONNECTED){
    io.connect();
    aioTimer = millis();
    while(io.status() < AIO_CONNECTED && millis() - aioTimer < AIO_TIMEOUT) { 
      delay(200); 
    }
  }

  if(io.status() >= AIO_CONNECTED){

    io.run();
 
    extTempFeed->save(temp_ext);
    extHumFeed->save(hum_ext);
    extVcapFeed->save(capVoltage);
    intTempFeed->save(inTempRounded);
 
  }
    //ping test below
     String hostName = "www.google.com";
     int pingResult;
     pingResult = WiFi.ping(hostName);
     if (pingResult >= 0) {
        Serial.print("SUCCESS! RTT = ");
        Serial.print(pingResult);
       Serial.println(" ms");
     } else {
       Serial.print("FAILED! Error code: ");
       Serial.println(pingResult);
    }

    Serial.println(F("Free RAM = "));
    Serial.println(freeMemory(), DEC);  // print how much RAM is available.
 
   delay(500); //allow wifi to finish
}

haku15
 
Posts: 67
Joined: Mon Jun 25, 2012 12:45 pm

Re: Suddenly only first AIO Connection works

by brubell on Tue Nov 06, 2018 10:13 am

Code: Select all | TOGGLE FULL SIZE
io.run()
might be what's causing the issue. It keeps the client connected to io.adafruit and is should be placed at the top of the main loop (it should get polled).

Could you post your full code (not just the function)?

brubell
 
Posts: 189
Joined: Fri Jul 17, 2015 10:33 pm

Re: Suddenly only first AIO Connection works

by haku15 on Tue Nov 06, 2018 7:32 pm

Thanks for the help, Brubell.
I reduced the whole code to this for troubleshooting:
Code: Select all | TOGGLE FULL SIZE
#include "AdafruitIO_WiFi.h"

#define DEBUG_ON //uncomment to activate Serial and USB attach

#define IO_USERNAME    "username"
#define IO_KEY         "key"

#define WIFI_SSID       "ssid"
#define WIFI_PASS       "password"

AdafruitIO_WiFi io(IO_USERNAME, IO_KEY, WIFI_SSID, WIFI_PASS);

#ifdef DEBUG_ON
#define halt(s) { Serial.println(F( s )); while(1);  }
#endif
#define halt(s) {  while(1);  }

AdafruitIO_Feed *extTempFeed = io.feed("project.temp");

const unsigned long AIO_TIMEOUT = 80000; //timeout (secs) Adafruit IO Connect.

int temp_ext= 4;

void setup() {

  delay(9000); //wait for usb and for sketch upload

  #ifdef DEBUG_ON
  Serial.begin(115200);
  #endif
 
}

void loop() {

  sendDataToIO();
  for(byte i=0; i<6; i++){   //change max i value to change data send interval
   
     Serial.println("for");
     delay(60000);
  }
 
}

void sendDataToIO(){

  unsigned long aioTimer = 0;

    if(io.status() < AIO_CONNECTED){
      io.connect();
      aioTimer = millis();
      while(io.status() < AIO_CONNECTED && millis() - aioTimer < AIO_TIMEOUT) { 
        #ifdef DEBUG_ON
        Serial.print(".");
        #endif
        delay(500); 
      }
      #ifdef DEBUG_ON
      Serial.print(F("aio_timer:"));
      Serial.println(aioTimer);
      Serial.println(F("aio status"));
      // we are connected
      Serial.println();
      Serial.println(io.statusText());
      #endif
    }
 
  if(io.status() == AIO_CONNECTED){
    #ifdef DEBUG_ON
      Serial.println(F("io.run"));
    #endif
    io.run();
 
    extTempFeed->save(temp_ext);
 
  }
    String hostName = "www.google.com";
    int pingResult;
    pingResult = WiFi.ping(hostName);
    if (pingResult >= 0) {
      Serial.print("SUCCESS! RTT = ");
      Serial.print(pingResult);
      Serial.println(" ms");
    }
    else {
      Serial.print("FAILED! Error code: ");
      Serial.println(pingResult);
    }
}


If the data send interval is lower than MQTT_CONN_KEEPALIVE (=5min, standard value set in Adafruit_MQTT.h of Adafruit_MQTT library), all the data are received in AIO.
If the interval is bigger than MQTT_CONN_KEEPALIVE, AIO receives only every other data. For example, with interval=6 min and start at 23:37, AIO shows data at 23:37, (nothing at 23:43), 23:49, (nothing at 23:55). Same story with interval 8 minutes. It seems the bigger the interval, the more data loss, but I have yet to verify this.

Here is the test with 6 min Interval (on the right is the Console output):
test_send_6min.png
Test result
test_send_6min.png (81.58 KiB) Viewed 125 times


Libs versions:
Adafruit MQTT: 0.17
WiFi 101: 0.15
Adafruit IO Arduino: 2.7.1
Arduino HTTP Client: 0.2.0

Could it be that some changes have been done in AIO in Sept/Oct that requires libraries update? I know some of them are a bit old, but I expect them to be functional utill there's an info in AIO Blog, that some user action is required, right?
The method that I posted in the first link has been sending data succesfully for months until Sept/Oct.

Thanks again for your help.

haku15
 
Posts: 67
Joined: Mon Jun 25, 2012 12:45 pm

Re: Suddenly only first AIO Connection works

by abachman on Wed Nov 07, 2018 7:17 pm

hi haku15,


Have you tried putting all the IO object creation inside a repeatable function? Since the io instance is only initialized once when the sketch starts, there may be some internal state that's breaking down on repeated connection attempts. I don't actually know if this will work, since the IO object also wraps WiFi global state.

Also, sending an HTTP request during an open MQTT session may cause contention in the underlying TCP/IP networking stack. I've noticed dropped MQTT packets especially when trying to intermingle HTTP requests, even in very simple sketches.

If the data send interval is lower than MQTT_CONN_KEEPALIVE (=5min, standard value set in Adafruit_MQTT.h of Adafruit_MQTT library), all the data are received in AIO. If the interval is bigger than MQTT_CONN_KEEPALIVE, AIO receives only every other data.


That's seems like it may be logical behavior for long-lived TCP connections like this. Our MQTT broker sends a heartbeat message about once a minute, but if it never get a response and there is no activity from the client, it eventually closes the connection on its end. The problem with networking code at this level is that neither end knows that the other end has gone away until it tries to send something. In this case, (if I'm reading your code and understanding you correctly) your sketch goes back to a connection it assumes is open and tries to send another packet. The packet gets dropped, the connection is recognized as actually closed, which means the next status check shows as disconnected and it can successfully reconnect. That would explain the "first data is lost" problem, at least.

If your goal is to wait a long time then wake up and send a message, you might consider writing your sketch so that it will reset after the timeout is complete and then start from the beginning. With the M0, you could try the SleepyDog library for sleep & reset functionality.

NOTE: some of this is speculative, you seem to be doing pretty good research already yourself. If you find that some particular code you'd like to write isn't documented and isn't demonstrated in any of the libraries you're using, it may just not work and you may be the first to have tried, unfortunately :(


- adam b.

abachman
 
Posts: 242
Joined: Mon Feb 01, 2010 12:48 pm

Re: Suddenly only first AIO Connection works

by haku15 on Wed Nov 07, 2018 7:23 pm

More Tests:
- made the sketch with 15 min data send interval run 2.5hours and also here every other data is missing (20:30 ok, 20:45 no, 21:00 ok,...). I don't hit any rate limiting in AIO: I have no other devices sending data to AIO.
- I've updated the libraries to the latest version:
Adafruit MQTT: 0.20.3
WiFi 101: 0.15.2
Adafruit IO Arduino: 2.7.19
Arduino HTTP Client: 0.3.1
then I repeated the 4 and 6 minutes interval tests: same problem.
- I set MQTT_CONN_KEEPALIVE in Adafruit_MQTT.h to 20min instead of 5. I run the 6 min interval test and same problem of half data missing.
So the problem seems to not be dependent on that parameter, but happens nonetheless if the interval of sending data to AIO is >= 5minutes.
All this tests were with WINC1500 LowPowerMode not activated.
WINC1500 Version B, 19.5.2 FW.
I don't know what else to try from my side...

haku15
 
Posts: 67
Joined: Mon Jun 25, 2012 12:45 pm

Re: Suddenly only first AIO Connection works

by abachman on Wed Nov 07, 2018 10:00 pm

Ah, the MQTT_CONN_KEEPALIVE value is for your client's own use, but it's anticipating the server's actual KEEPALIVE interval. I suspect the server is closing the connection on its end, but the way long quiet TCP connections like this work, your client doesn't find out that the server went away until it tries and fails to send data.

Bottom line, if you're not calling io.run() every minute or so, you're likely to disconnect eventually. Even if you call it and everything else in your code is perfect, anything that interrupts the network connection anywhere between the device and the IO servers will cause it to drop. Here's your sample code restructured to put the io.connect in setup() and io.run in loop(), with the long delay between posts. It's power hungry if you're on battery, but will stay connected if you're on a steady power supply. note: this code compiles, I haven't tested it

Code: Select all | TOGGLE FULL SIZE
#include "AdafruitIO_WiFi.h"

#define DEBUG_ON //uncomment to activate Serial and USB attach

#define IO_USERNAME    "username"
#define IO_KEY         "key"

#define WIFI_SSID       "ssid"
#define WIFI_PASS       "password"

AdafruitIO_WiFi io(IO_USERNAME, IO_KEY, WIFI_SSID, WIFI_PASS);

#ifdef DEBUG_ON
#define halt(s) { Serial.println(F( s )); while(1);  }
#endif
#define halt(s) {  while(1);  }

AdafruitIO_Feed *extTempFeed = io.feed("project.temp");

int temp_ext = 4;

void setup() {
#ifdef DEBUG_ON
  Serial.begin(115200);
#endif

  io.connect();
  while (io.status() < AIO_CONNECTED) {
#ifdef DEBUG_ON
    Serial.print(".");
#endif
    delay(500);
  }

#ifdef DEBUG_ON
  Serial.println(F("connected!"));
  Serial.print(F("aio status "));
  Serial.println(io.statusText());
#endif
}

long last_sent_at = 0;
long send_interval = 60000;

void loop() {
  io.run();

  long now = millis();
  if (now > last_sent_at + send_interval) {
#ifdef DEBUG_ON
    Serial.print("sending");
#endif
    sendDataToIO();
  }
}

void sendDataToIO() {
  extTempFeed->save(temp_ext);
#ifdef DEBUG_ON
  delay(1000);
  String hostName = "www.google.com";
  int pingResult;
  pingResult = WiFi.ping(hostName);
  if (pingResult >= 0) {
    Serial.print("SUCCESS! RTT = ");
    Serial.print(pingResult);
    Serial.println(" ms");
  } else {
    Serial.print("FAILED! Error code: ");
    Serial.println(pingResult);
  }
#endif
}


Alternatively, are you able to rewrite the sketch with a hard reset after your long delay?

In pseudo-code, something like:
Code: Select all | TOGGLE FULL SIZE
setup io object

function setup:
  setup sensors
  start watchdog timer counting down from 15 minutes
  io connect

function loop:
  io run
 
  read sensor
  publish sensor value to feed

  while (1) {} // loop until watchdog resets sketch


I've got a similar setup in a temp/humidity monitor I keep running all the time. I call Watchdog.reset() inside the main loop() function, so if / when the sketch gets stuck somewhere weird or is trying to reconnect forever, reset() never gets called and the whole sketch starts over from the beginning. This might be a better fit in your case, since it will reset the io object from scratch. This example also doesn't turn off the radio, so it's relatively high power.

If you're sticking with the Adafruit_IO_Arduino library, it's safest to stick to the standard form:
Code: Select all | TOGGLE FULL SIZE
void setup() {
  io.connect();
}

void loop() {
  io.run();
}
Anything outside that, especially dealing with sleep or low power modes, and you might find Adafruit_MQTT or another lower-level MQTT library to be a better fit.

abachman
 
Posts: 242
Joined: Mon Feb 01, 2010 12:48 pm

Re: Suddenly only first AIO Connection works

by haku15 on Thu Nov 08, 2018 6:51 pm

Hi Abachman,
thank you very much for the detailed explanations.
My previous post wasn't a reply to you first answer, because we posted almost at the same time and saw your msg only later.
I implemented your suggestions in both the "troubleshooting" sketch as well as in the complete sketch and they work.
Basically I call io.connect() just once at the beginning, refresh the io connection every minute with io.run() and post the sensors data to AIO every 15 minute with save().

Still it 'd have been interesting to know, why my previous code (see first post) had worked till Sept/Oct 2018 and why io.connect() cannot be executed repeatedly every 15 minutes anymore.
I presume, some configuration was changed on AIO MQTT broker.

Thanks again
Sincerely

haku15
 
Posts: 67
Joined: Mon Jun 25, 2012 12:45 pm

Please be positive and constructive with your questions and comments.