Managing Bursty Data Sources Moderators: adafruit_support_bill, adafruit

[mookiedog]

Forgive me, I am a total noob regarding AdafruitIO.

I have a project going that monitors the A/C current draw on my home's well and septic pumps. The pumps run irregularly, but typically only 3 to 10 times per day for maybe 3 minutes at a time. While a pump is running, my system will be logging current 16-bit resolution consumption data at 10 Hz. Conceptually, I understand how I could use AdafruitIO to display a pump operating state in terms of the pump either being 'on' or 'off'. There would be some value in that, and it would represent a very small amount of data. But what I would really like to see though is a graph of the current log during the most recent pump run event so that I can watch for anomalies like the length of the motor start current surge. Doing some rough math, a 3 minute pump run would generate about 1800 current readings, or 3600 bytes of data.

I don't need to see the detailed current log graph get updated in real time. It would be fine to push the data out when the run ends, and generate the graph after the fact. Is this something that AdafruitIO would be capable of doing?

[brubell]

But what I would really like to see though is a graph of the current log during the most recent pump run event so that I can watch for anomalies like the length of the motor start current surge. Doing some rough math, a 3 minute pump run would generate about 1800 current readings, or 3600 bytes of data.

Adding a Line Chart block on your Adafruit IO dashboard will allow you to visualize/plot up to five feeds. The block provides time-scales ranging from a real-time reading, up until 30 days. You could build a dashboard with a Line Chart visualizing data in real-time, and another line chart displaying a longer-term visualization for analysis.

You're also able to download feed information as a .csv file for parsing/visualizing, too.

If you have any other questions, don't hesitate to ask :)

[abachman]

Hi mookiedog,


IO's free limit of 30 data points per minute means you wouldn't be able to post 600 individual values, but we allow 1024 bytes per data point, so you could easily fit 1800 1-byte values in 3 minutes. We won't chart it, but you can at least store it online and retrieve it from anywhere else.

One tricky piece is that IO stores all data as valid UTF-8 strings, so shipping raw byte[] data (which most MQTT libraries permit) is dangerous, because it can generate invalid strings. A safer bet is to either store the numerical values as space-separated numbers or store the raw bytes as hex encoded strings. E.g., 0b11010011 becomes 0xd3 or sent to IO, "d3". Bottom line, come up with whatever string-compatible packing format you like and IO will store it.

For example, given a sensor running at 10hz, pack 30 values and send every 3 seconds. At the receiving end (either an MQTT subscriber in close-to-realtime or after downloading the feed data at your leisure), unpack each value back into 10 bytes per second and put your own timestamps on them.

I did similar with an art project that posted an 8x6 grid of motion data every 2 seconds: https://github.com/micais2019/interacti ... py#L70-L79. We also published two feeds of sound level data recorded at much greater than 10Hz and downsampled (via the LTTB algorithm) to 10Hz and published every three seconds: https://github.com/micais2019/interacti ... #L152-L168. Two feeds because we had two stations recording sound and publishing while subscribed to each other, unpacking the data and displaying it via Neopixel.

In each of those cases I used basic Python string formatting to concatenate int values into a space-separated string.

Code: Select all | TOGGLE FULL SIZE

from random import random
li = [int(random() * 16) for n in range(30)]
output = ' '.join("%i" % v for v in li)
print(output)


more or less. On the other end, we unpacked with split() + int():

Code: Select all | TOGGLE FULL SIZE

[int(v) for v in output.split(' ')]

.

It might be helpful to say it's less an IO / IoT problem and more a "creatively storing many values in a string" and "retrieving stored values from a string" problem, or, "running your own protocol on top of MQTT."


- adam b.

[mookiedog]

I could definitely send the data in 1K chunks.

I also understand that the line chart block at Adafruit.IO would not be able to deal with the chunked data, which is totally fair because the line graph dashboard processing block wouldn't know how the format of the items in the chunk, much less how they were packed.

It would seem that there is no support for users to create custom dashboard data processing block widgets on Adafruit.IO. It looks like there are a set of widgets we can choose from and configure, but not create custom ones which might be used to un-chunk data and display it. Correct?