0

Query stored data on adafruit.io by time
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Query stored data on adafruit.io by time

by cweiss on Fri May 10, 2019 2:19 pm

Hi there!

I am looking to query the last 24 hours of data from a feed. Im using arduino language and can successfully publish and subscribe via MQTT and adafruitmqtt library.

My goal is to retrieve the last x data values within the 24 hours from my feed via mqtt, but I'm not sure if this is possible. The data points are json strings that are time stamped. Once I get them back into my program, I will split the json and sum values (an interesting problem for later).

I see in the documentation that there is "Get activity by type" and query parameters such as start and end time. I don't understand how to implement it though.

Currently, I am subscribed and retrieve the last value each time and save it to download:

Code: Select all | TOGGLE FULL SIZE
//from mqtt_2_subs_esp8266 example:

  Adafruit_MQTT_Subscribe *subscription;
  while ((subscription = mqtt.readSubscription(5000))) {
    // Check if its the onoff button feed
    if (subscription == &onoffbutton) {

    download = (char *)onoffbutton.lastread;


If I could get a code snippet or guidance that makes download have the last 24 hours of values instead of lastread that would be perfect.

Thanks!

cweiss
 
Posts: 5
Joined: Sat Oct 20, 2018 4:01 pm

Re: Query stored data on adafruit.io by time

by abachman on Fri May 10, 2019 4:24 pm

Hi cweiss!


The API endpoint you're looking for is called "Get Feed Data" in our documentation. The "Activity" records are things like, "you created a feed" or "you removed a block from a dashboard". Not particularly interesting in terms of collected data.

I had to write a script last week to do pretty much exactly the same thing you're trying to do, I've posted it on GitHub here: https://gist.github.com/abachman/12df0b34503edd5692be22f6b9695539. It works with Python 3 and only uses the standard library. Near the top of the file are some configuration values you'll need to set up to make it work on your computer with your Adafruit IO account. Everything that's ALL_CAPITAL_LETTERS is intended to be modified before you run the sketch.

Because Adafruit IO only returns 1000 records at a time, we have a pagination system in place for letting HTTP clients know that there's more data available when you make a request that has more than 1000 data points in it. For example, 24 hours of data on a feed that you publish to once per minute would be about 1440 values, or two HTTP requests. If your initial request has start_time and end_time query parameters, then the pagination API will include them in subsequent requests automatically.

The script I linked to handles that automatically to make sure you get all your data.

Also note: the script includes a `time.sleep(1)` line in between requests. make sure that stays in or you will probably end up throttled by IO. We have a higher cap on HTTP GET requests, but we still have a cap.


- Adam

abachman
 
Posts: 287
Joined: Mon Feb 01, 2010 12:48 pm

Re: Query stored data on adafruit.io by time

by cweiss on Mon May 13, 2019 10:29 pm

Thanks for the reply Adam!

Ok, this will work I think. For now I don't actually need this, I can do some querying locally. I will certainly get this implemented down the road and will revisit this forum.

Cheers,

Corey

cweiss
 
Posts: 5
Joined: Sat Oct 20, 2018 4:01 pm

Please be positive and constructive with your questions and comments.