0

Not All Data Is Being Logged
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Not All Data Is Being Logged

by 1CM69 on Mon Jul 22, 2019 9:35 am

Hi,
very new to Arduino, Adafruit IO, ESP8266 etc but I seem to have hit a stumbling block that I cannot seem to fathom out.

I currently have a testbed setup of an ESP8266 with OLED attached & it reads and displays data from a DHT22 sensor.

I am also sending this data to ThingSpeak & Adafruit IO every 900 seconds (15mins).

What seems to be happening is that ThinkSpeak does indeed receive each set of data every 15 mins, Adafruit IO only updates every 30 mins.

Could someone take a quick look at my code & tell me where I have gone wrong.

Kind regards..,

Kirk

Code below, all personal info is ****** out.

Code: Select all | TOGGLE FULL SIZE
#include <ESP8266WiFi.h>
#include "DHT.h"
#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
#include <Adafruit_Sensor.h>
#include "ThingSpeak.h"
#include "AdafruitIO_WiFi.h"

unsigned long myChannelNumber = ******;
const char * myWriteAPIKey = "******";


#define SCREEN_WIDTH 128 // OLED display width, in pixels
#define SCREEN_HEIGHT 64 // OLED display height, in pixels

Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, -1);

#define DHTTYPE DHT22   // DHT 22  (AM2302), AM2321

const char* ssid = "******";  // Enter SSID here
const char* password = "******";  //Enter Password here

#define AIO_USERNAME    "******"
#define AIO_KEY         "******"

AdafruitIO_WiFi io(AIO_USERNAME, AIO_KEY, ssid, password);

AdafruitIO_Feed *BathTempFeed = io.feed("bathroom-temperature");
AdafruitIO_Feed *BathHumFeed = io.feed("bathroom-humidity");

WiFiClient  client;

uint8_t DHTPin = D5;
               
DHT dht(DHTPin, DHTTYPE);               

void setup() {

  pinMode(DHTPin, INPUT);

  dht.begin();

  ThingSpeak.begin(client);         

  WiFi.begin(ssid, password);

   //check wi-fi is connected to wi-fi network
  while (WiFi.status() != WL_CONNECTED) {
  delay(1000);
  }
 
  io.connect();

 if(!display.begin(SSD1306_SWITCHCAPVCC, 0x3C)) {
    Serial.println(F("SSD1306 allocation failed"));
    for(;;);
  }
  delay(2000);
  display.clearDisplay();
  display.setTextColor(WHITE);
}

void loop() {
 
  float t = dht.readTemperature();
  int h = dht.readHumidity();
 
  // clear display
  display.clearDisplay();
 
  // display temperature
  display.setTextSize(1);
  display.setCursor(0,0);
  display.print("Temperature: ");
  display.setTextSize(2);
  display.setCursor(0,10);
  display.print(t);
  display.print(" ");
  display.setTextSize(1);
  display.cp437(true);
  display.write(167);
  display.setTextSize(2);
  display.print("C");
 
  // display humidity
  display.setTextSize(1);
  display.setCursor(0, 35);
  display.print("Humidity: ");
  display.setTextSize(2);
  display.setCursor(0, 45);
  display.print(h);
  display.print(" %");
 
  display.display();

  io.run();
 
  BathTempFeed->save(t);
  BathHumFeed->save(h);

  ThingSpeak.setField(1, t);
  ThingSpeak.setField(2, h);

  ThingSpeak.writeFields(myChannelNumber, myWriteAPIKey);

  delay(900000); //true delay 15 mins
  //delay(10000); //for testing 10 secs
}


1CM69
 
Posts: 4
Joined: Mon Jul 22, 2019 9:10 am

Re: Not All Data Is Being Logged

by abachman on Mon Jul 22, 2019 11:24 am

Hi 1CM69,


First thing that sticks out to me is that IO is using MQTT to save data and that means the server expects a heartbeat message every ~5 minutes or it assumes the connection was lost. In the IO Arduino library's case, io.run() handles that heartbeat / ping message for you, but that means you have to call io.run() at least once every 5 minutes to keep the connection alive. IO may be publishing every other time through because it fails and reconnects, then succeeds the next time, then times out again.

The easiest fix for IO is to call loop() as often as possible, but only perform publishing actions every 15 minutes (or however long). You could do that by using an unsigned long variable to track the last_published_at time and then only publish when millis() passes a specified delay. For example:
Code: Select all | TOGGLE FULL SIZE
unsigned long last_published_at = 0;
unsigned long publish_delay = 15 * 60 * 1000; // in milliseconds

void loop() {
  io.run();

  unsigned long now = millis();
  if (now > last_published_at + publish_delay) {
    doPublish();
    last_published_at = now;
  }

  // only pause for a second on each loop
  delay(1000);
}


This also has the benefit of letting your device do other work inside the loop() function. For example, if you wanted to add servos or LED animations (neopixels or dotstars), those will play a lot nicer with a sketch that doesn't include long delays.

As for why the ThingSpeak publishes are working, it could be that the ThingSpeak library is using HTTP? Two things there: 1) it's stateless and reconnects as needed, so timeouts aren't a problem and 2) running multiple long client connections (e.g., two MQTT connections or one MQTT and HTTP) inside the same project on a microcontroller board like the ESP8266 is problematic. There's only one process available to the system, so it could be that calling ThingSpeak is disconnecting IO. I'm not familiar with their library, but both IO and ThingSpeak probably want full control of the WiFi hardware.

My advice for further troubleshooting is to: 1) shorten the delay to less than 5 minutes however you can and 2) either use 1 networking library at a time to see how each works on its own or move all the connection code into loop() and reconnect IO every time before you publish.


- adam b.

abachman
 
Posts: 338
Joined: Mon Feb 01, 2010 12:48 pm

Re: Not All Data Is Being Logged

by 1CM69 on Mon Jul 22, 2019 1:48 pm

abachman wrote:Hi 1CM69,


First thing that sticks out to me is that IO is using MQTT to save data and that means the server expects a heartbeat message every ~5 minutes or it assumes the connection was lost. In the IO Arduino library's case, io.run() handles that heartbeat / ping message for you, but that means you have to call io.run() at least once every 5 minutes to keep the connection alive. IO may be publishing every other time through because it fails and reconnects, then succeeds the next time, then times out again.

The easiest fix for IO is to call loop() as often as possible, but only perform publishing actions every 15 minutes (or however long). You could do that by using an unsigned long variable to track the last_published_at time and then only publish when millis() passes a specified delay. For example:
Code: Select all | TOGGLE FULL SIZE
unsigned long last_published_at = 0;
unsigned long publish_delay = 15 * 60 * 1000; // in milliseconds

void loop() {
  io.run();

  unsigned long now = millis();
  if (now > last_published_at + publish_delay) {
    doPublish();
    last_published_at = now;
  }

  // only pause for a second on each loop
  delay(1000);
}


This also has the benefit of letting your device do other work inside the loop() function. For example, if you wanted to add servos or LED animations (neopixels or dotstars), those will play a lot nicer with a sketch that doesn't include long delays.

As for why the ThingSpeak publishes are working, it could be that the ThingSpeak library is using HTTP? Two things there: 1) it's stateless and reconnects as needed, so timeouts aren't a problem and 2) running multiple long client connections (e.g., two MQTT connections or one MQTT and HTTP) inside the same project on a microcontroller board like the ESP8266 is problematic. There's only one process available to the system, so it could be that calling ThingSpeak is disconnecting IO. I'm not familiar with their library, but both IO and ThingSpeak probably want full control of the WiFi hardware.

My advice for further troubleshooting is to: 1) shorten the delay to less than 5 minutes however you can and 2) either use 1 networking library at a time to see how each works on its own or move all the connection code into loop() and reconnect IO every time before you publish.


- adam b.


Hi and thanks for the reply.

to start with I thought that I could circumvent this connection dropout by altering the line in Adafruit_MQTT.h from:

Code: Select all | TOGGLE FULL SIZE
// Adjust as necessary, in seconds.  Default to 5 minutes.
#define MQTT_CONN_KEEPALIVE 300


to something like:

Code: Select all | TOGGLE FULL SIZE
#define MQTT_CONN_KEEPALIVE 950


but this made no difference at all, still ThingSpeak gets readings every 15 mins but Adafruit only every 30 mins.

I am guessing that 300s is the absolute max.

I appreciate your help with the suggestion of shortening the loop but this will not be feasible for my finished project where all of the units are remotely located and running from battery power and in deep sleep before been woken once per hour via RTC to send out the data, so pinging every few seconds or so defeats my low power ethic.

I may need to find an alternative to using MQTT because of these restrictions.

I am going to more testing before I need to move on & post back any relevant findings.

Kind regards..,

Kirk

1CM69
 
Posts: 4
Joined: Mon Jul 22, 2019 9:10 am

Re: Not All Data Is Being Logged

by abachman on Mon Jul 22, 2019 5:34 pm

Good find! In this case, though the MQTT_CONN_KEEPALIVE variable controls how often the client (your device) sends a ping, but won't have any effect on the server. It's accessible in the library in case you're working with a broker that has a longer timeout.

With MQTT, your device opens a TCP connection to the server. As long as there's data moving back and forth (publishing, subscribing, or PINGing), the server knows that the connection is alive. When your device isn't doing anything, though, the server has no way of knowing whether there's actually a device still connected. There are probably 10+ spots in between your device and IO across the Internet where the connection could be silently dropped and neither the server (Adafruit IO) nor the client (your device) will know that the connection is gone until they try to send something. In the server's case, it just assumes that any device it hasn't heard from in 5 minutes went away and it closes the connection silently on its end. The client's PING every 4min 59sec resets that countdown.

... my finished project where all of the units are remotely located and running from battery power and in deep sleep before been woken once per hour via RTC to send out the data, so pinging every few seconds or so defeats my low power ethic.


Ooooo, that's a totally different situation. If you're deep sleeping devices, it doesn't matter what sort of connection you use since it will always have to reopen connections from scratch when the device wakes up. If you're using really long delays to simulate a sleep/wake cycle, you could move the IO object creation and .connect() call into a function that's called from the top of the loop() function every time it comes around.

MQTT is good in cases where you either want devices to be awake all the time and available to receive a message at any moment or you're very aggressively counting bytes of network traffic (on the order of 100 bytes or so) and able to ping often enough to keep the connection alive.


- adam b.

abachman
 
Posts: 338
Joined: Mon Feb 01, 2010 12:48 pm

Please be positive and constructive with your questions and comments.