Data is truncated from adafruit IO Moderators: adafruit_support_bill, adafruit

[msdupriest]

Hi there,

I'm working on a project that lights up a strand of neopixels in response to tweets sent from Twitter.
I have an app on IFTTT.com that sends specified tweets to Adafruit IO,

And then my microcontroller (it's a Huzzah Wifi, one of these https://www.adafruit.com/product/2821) reads the data from Adafruit IO and based on key words in the tweet, it lights up an animation on the neopixels.

The problem is that if I send a tweet that's over 52 characters, it'll either not be received at all or when it is received, it's truncated.

Is there a length to how long data can be? I can see in the monitor that the entire tweet is being received and stored in Adafruit IO... but my device either doesn't receive all the data or truncates it if it does.

Here are some examples:

I sent this tweet.

tweeterror3.PNG (17.03 KiB) Viewed 132 times

You can see it was received by the Adafruit IO

tweeterror1.PNG (15.71 KiB) Viewed 132 times

My monitor shows it's there. But as you can see in the image, my device disconnected when it got this tweet.

tweeterror2.PNG (18.54 KiB) Viewed 132 times

This is what the code is supposed to do:

Code: Select all | TOGGLE FULL SIZE

void handleMessage(AdafruitIO_Data *data) {

  Serial.println("A tweet came in!");
 
  Serial.println(data->value());
  tweettext = String(data->value());
  tweettext.toLowerCase();
  received_tweet = true;
   
}

Nothing actually gets written to the serial monitor when the tweet is received. If I send another, shorter tweet afterward, it is correctly written to the serial monitor - so the program is still looping, it didn't exit - what happened? Is there a max length for the data -> value() function? Can I do anything to read an entire tweet and not just the first 50 characters?

Thank you
Dawn DuPriest

[brubell]

Hi Dawn,

Which library are you using with the Huzzah ESP8266?

Nothing actually gets written to the serial monitor when the tweet is received.

I'm not entirely understanding the workflow of what happens - could you clarify this for me:

When the tweet is received by the device, does the device show as disconnected in on io.adafruit.com/monitor? If it disconnects from Adafruit IO, nothing should be written to the serial monitor since the device is no longer communicating with Adafruit IO...

[msdupriest]

Hi,
I use the library recommended in the tutorial: the esp8266 by ESP8266 community.

The io monitor does show that my device gets disconnected. So it must get disconnected before my handler function is called and that's why nothing is written to the serial monitor.

I've seen a couple of cases where the serial monitor wrote the first 50 characters of the tweet and didn't parse the rest of it, but I couldn't reproduce that condition when I was writing the forum post. If a tweet was longer than 50-ish characters it just disconnected and ignored the tweet. It would handle tweets shorter than 50-ish characters just fine, and seemed to reconnect on its own to the adafruit IO to receive the tweet.

Is there a limit on the length of a string data value allowed to be received in a feed? Or are there certain characters that are disallowed and that's what I'm seeing?
Thank you

[brubell]

Is there a limit on the length of a string data value allowed to be received in a feed? Or are there certain characters that are disallowed and that's what I'm seeing?

Yes but you're not close to hitting the limit. By default you can push values up to 1KB in length, but with feed history off, you can publish values up to 100KB in length.

Are you sending alphanumeric characters only in these tweets, or characters like emojis?

[msdupriest]

I realized I was referring to the boards file, not the library. I'm using the Adafruit IO Arduino library version 4.0.2.

The program works exactly as it should as long as I only care about 52 characters or less in the data value coming from Adafruit IO. If I send anything longer than 52 characters, it behaves in an erratic way. I was able to reproduce the error where it truncated tweets.

Here is the io monitor output.

tweeterror8.PNG (47.84 KiB) Viewed 110 times

You can see that at 12:39pm I booted up the Huzzah wifi and it connected to Adafruit IO.
At 12:40pm I sent a really long tweet, 159 characters, and my device disconnected.
At 12:41 I sent a very short tweet but the device wouldn't read it anymore (disconnected) so I rebooted at 12:50.
I sent another long tweet (78 characters) at 12:50 and again it disconnected.
I sent a shorter tweet at 12:51 but the device was already disconnected so it wasn't recognized. If I reboot again, that last, short tweet will be read and the device will respond.

Here's what that looked like in the serial monitor. I'm also using the music maker featherwing so there's some serial output from checking the SD card is working. This part works fine.

tweeterror8.PNG (47.84 KiB) Viewed 110 times

My device never wrote the 12:40 tweet to the serial monitor - disconnected before the handler could run??
After I rebooted the Huzzah, it recognized the short tweet (because it was the most recent in the queue?), wrote it to the serial monitor and responded.
I sent the 78 character tweet next, and the handler got only the first 52 characters of it and wrote that to the serial monitor. My neopixel animation ran but had a power glitch or something at the end - the animation froze. I sent the last short tweet at 12:51 but the device was not responsive - and I had to reboot. After the reboot the device recognized the short tweet and responded appropriately.

I get consistent good results as long as I limit the tweets to 52 characters or less. More than that and it's very unpredictable. :(

If you need more of the code let me know.

[brubell]

Thank you for the detailed writeup. It's difficult for me to tell if the issue is on IFTTT's side, Adafruit IO's side or the library on the Huzzah.

Could you try copying the contents of one of your long tweets? Then on Adafruit IO, navigate to the sweater2020-feed's feed page and try manually adding the tweets string. does the huzzah still receive it or does it disconnect?

[msdupriest]

Thanks for prompting me to check this. I got the same behavior.

tweeterror9.PNG (30.35 KiB) Viewed 92 times

I pasted the entire text of a long tweet twice. The first time it was totally ignored, second time the monitor said "disconnected". I didn't reboot this time. I manually entered a long data value without the Twitter mentions, and the device still ignored it. Then I entered a short data value and the device received it and responded.

I tested manually adding data values into the feed of varying lengths and this is what I got:
If I enter data values of 52 characters or less, the Huzzah receives everything, writes everything to the serial monitor and behaves as expected.
If I enter data values of 52-84 characters, the Huzzah only writes the first 52 characters to the serial monitor but otherwise behaves as expected and continues to get communication from Adafruit IO.
If I enter data values of 85 characters and more, the Huzzah writes the first 52 characters to the serial monitor, but then won't receive any other communication from Adafruit IO and I have to reboot it.

Thanks for all your help with this! It isn't urgent, it just makes the project less usable as a twitter-enabled device.

So don't respond right away, enjoy the holiday with your family and get to this whenever you're back on duty :)

Cheers,
Dawn

[brubell]

Ok - we're getting closer. This looks like an issue with Adafruit IO Arduino or the Adafruit MQTT library, not with IFTTT or Adafruit IO.

Could you try navigating to your computer's installation of the Adafruit IO MQTT library for Arduino.

Then, in Adafruit_MQTT.h, try increasing the length of the following line:
https://github.com/adafruit/Adafruit_MQ ... QTT.h#L127