0

REST API get data between dates
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

REST API get data between dates

by koaiwi on Wed Jan 03, 2018 5:05 am

Hi

I've been using Xively (aka was pachube) for a few years now. Xively have decided to retire it now with 3 weeks notice :(

The API for adafruit is similar with the exception that retrieving a whole load of values has a very high data overhead. Generally I just what the data value and the date created. I don't want all the other stuff in the json response.

Xively did this by allowing a CSV format to be returned with just minimal data.

This is very gentle on the arduino based devices with limited memory.

I understand we can limit the number of data points returned but this will be laborious to iterate through the data and process it.

Could you please consider either:
1. allowing csv format response with just the value and time stamp OR
2. another RESP variable that specifies just which data element needs to be returned in the json response i.e: "value" and "created"

Unless there is another way to achieve the above?

Thanks.

koaiwi
 
Posts: 24
Joined: Thu Dec 28, 2017 10:11 am

Re: REST API get data between dates

by abachman on Thu Jan 04, 2018 6:41 pm

Posted an update for this question over at https://forums.adafruit.com/viewtopic.php?f=56&t=128849&p=641758#p641758.


- adam

abachman
 
Posts: 228
Joined: Mon Feb 01, 2010 12:48 pm

Re: REST API get data between dates

by koaiwi on Fri Jan 05, 2018 2:31 am

Thanks Adam.Brilliant.

Hidden away undocumented is there a way of defining how the data is retrieved. Xively had a way of defining discrete data or extrapolated data. Discrete data returned each saved data point. Extrapolated allowed you to bring back an average of the data between defined data intervals. For example the average of every 1 minute or every 5 mins or every hour. So if I wanted data between 12:00 and 2:00 in 5 min intervals I would get back 24 data points averaged for each data interval.

Thanks.

koaiwi
 
Posts: 24
Joined: Thu Dec 28, 2017 10:11 am

Re: REST API get data between dates

by abachman on Mon Jan 08, 2018 1:59 pm

Eh, not entirely undocumented, but we have the /data/chart API endpoint that is used by io.adafruit.com dashboards to render charts that cover > 480 data points.

You can find the raw docs here: https://io.adafruit.com/api/docs/#!/Data/chartData

I've posted in this forum a few times with details about our aggregation methods, so I'll leave details out here. If you search "aggregate" in this forum you should get a list of related posts (for example).

if I wanted data between 12:00 and 2:00 in 5 min intervals I would get back 24 data points averaged for each data interval


short answer: we got that.

Code: Select all | TOGGLE FULL SIZE
# my office temperature between 12:00 and 14:00 UTC on new year's day. brrrrrr
$ curl -H "X-AIO-Key: $IO_KEY" 'https://io.adafruit.com/api/v2/abachman/feeds/temperature/data/chart?start_time=2018-01-01T12:00Z&end_time=2018-01-01T14:00Z&resolution=5' | pretty-json
{
  "feed": {
    "id": 606108,
    "name": "Temperature",
    "key": "temperature"
  },
  "parameters": {
    "start_time": "2018-01-01T12:00:00Z",
    "end_time": "2018-01-01T14:00:00Z",
    "resolution": 5,
    "hours": 2,
    "field": "avg"
  },
  "columns": [
    "date",
    "value"
  ],
  "storage": "2",
  "data": [
    [
      "2018-01-01T12:00:00Z",
      "54.545"
    ],
    [
      "2018-01-01T12:05:00Z",
      "54.716"
    ],
    [
      "2018-01-01T12:10:00Z",
      "54.89"
    ],
    [
      "2018-01-01T12:15:00Z",
      "55.04"
    ],
    [
      "2018-01-01T12:20:00Z",
      "55.112"
    ],
    [
      "2018-01-01T12:25:00Z",
      "55.22"
    ],
    [
      "2018-01-01T12:30:00Z",
      "55.22"
    ],
    [
      "2018-01-01T12:35:00Z",
      "55.22"
    ],
    [
      "2018-01-01T12:40:00Z",
      "55.13"
    ],
    [
      "2018-01-01T12:45:00Z",
      "55.04"
    ],
    [
      "2018-01-01T12:50:00Z",
      "55.04"
    ],
    [
      "2018-01-01T12:55:00Z",
      "54.968"
    ],
    [
      "2018-01-01T13:00:00Z",
      "54.86"
    ],
    [
      "2018-01-01T13:05:00Z",
      "54.86"
    ],
    [
      "2018-01-01T13:10:00Z",
      "54.752"
    ],
    [
      "2018-01-01T13:15:00Z",
      "54.635"
    ],
    [
      "2018-01-01T13:20:00Z",
      "54.572"
    ],
    [
      "2018-01-01T13:25:00Z",
      "54.5"
    ],
    [
      "2018-01-01T13:30:00Z",
      "54.32"
    ],
    [
      "2018-01-01T13:35:00Z",
      "54.284"
    ],
    [
      "2018-01-01T13:40:00Z",
      "54.14"
    ],
    [
      "2018-01-01T13:45:00Z",
      "54.032"
    ],
    [
      "2018-01-01T13:50:00Z",
      "53.915"
    ],
    [
      "2018-01-01T13:55:00Z",
      "53.816"
    ],
    [
      "2018-01-01T14:00:00Z",
      "53.78"
    ]
  ]
}




- adam

abachman
 
Posts: 228
Joined: Mon Feb 01, 2010 12:48 pm

Re: REST API get data between dates

by koaiwi on Tue Jan 09, 2018 11:47 am

Once again Brilliant. Just what I need. I hadn't looked at chart option yet as it did not immediately strike me as an option for getting efficient aggregated data. But yes just what I need to feed my own charts.

Thanks.

ps. this is looking remarkably like Xively. Is this the pedigree of adafruit IO :) ?

koaiwi
 
Posts: 24
Joined: Thu Dec 28, 2017 10:11 am

Re: REST API get data between dates

by abachman on Tue Jan 09, 2018 12:09 pm

his is looking remarkably like Xively. Is this the pedigree of adafruit IO :) ?


Ha! No, no, just a case of very similar design requirements and architectural responses. Like most (all?) web applications, we're using a ton of open source code (Ruby on Rails, React, lots of node.js) but the application code is all custom :)


- adam

abachman
 
Posts: 228
Joined: Mon Feb 01, 2010 12:48 pm

Re: REST API get data between dates

by koaiwi on Tue Jan 09, 2018 12:22 pm

Good work. The old pachube was good. Sad to see it go but in adafruit I think you have it nailed. Better than thingspeak - my own personal opinion: ). I hope you can make it last. I don't mind paying for it.

koaiwi
 
Posts: 24
Joined: Thu Dec 28, 2017 10:11 am

Please be positive and constructive with your questions and comments.