Only 1000 data points? Moderators: adafruit_support_bill, adafruit

[kevinljxljx]

I am trying to use the data from adafruit.io with my project, but no matter what I try, adafruit.io only gives 1000 data points to my javascript program. ( I am using XMLHttpRequest). I tried to set limit to 2000 but without luck. ( I have 1200 or so data points under my feed). Is the maximum data points I can get is 1000? or am I missing something?

[ShelM]

Hi Kevin...

When you have a feed open, look on the righthand side of the page - you'll see several options which define how your feed works. The one you are interested in is "Feed History". The reason you're interested in this is that when History is ON, you get 1KB data storage, when OFF you get 100KB.

That said, It's not clear to me if 1KB actually *means* 1KB, or if it means 1K data points.

Here is the text from the Feed History menu settings box:

"Update Feed History Storage
Turning history off for your feed will mean only the last value received is stored. History is on for all new feeds by default.

While history is turned ON, feed data is limited to 1KB (1024 bytes) in size.

While history is turned OFF, feed data is limited to 100KB (102400 bytes) in size"

Bests,
...ShelM

[abachman]

hi kevin,


The API only returns a maximum of 1000 data points at a time, you'll have to "paginate" the data to get the whole feed.

When you do a data query, the results are always sorted newest-to-oldest and include x-pagination-* headers. For example,

Code: Select all | TOGGLE FULL SIZE

X-Pagination-Limit: 1000
X-Pagination-Total: 84548
X-Pagination-Start: 2019-02-11T22:52:18.103+0000
X-Pagination-End: 2019-02-12T16:03:00.694+0000
X-Pagination-Count: 1000

Limit is either the requested limit or 1000, whichever is less; Total is the total number of data points in the feed--note, this value may be up to 5 minutes behind real time; Start is the timestamp on the oldest value; End is the timestamp on the newest value; and Count is the number of data points in the current request. Whenever Limit and Count are both 1000 and Total is more than 1000, that's evidence that more data is available. You can get the next 1000 data points by using the X-Pagination-Start value OR the created_at value of the oldest data point in the API response as the end_time parameter in your next request to the data API.

I always end up thinking about it visually. On a timeline the idea kind of looks like this:

api-pagination.png (29.28 KiB) Viewed 134 times

NOTE: long running, frequently updated feeds could have more than a hundred "pages" of data. If you make requests without a delay in between, you could hit a rate limited. Watch for 429 HTTP error responses.

Regarding the data storage and feed history, storage size in this instance is the per-data point value size limit. With history on, meaning we preserve every data point, each data point value can be at most 1KB. With history off, meaning we only preserve the most recent data point, each value can be at most 100KB.


- adam b.

[kevinljxljx]

Hi adam,
Thanks a lot! This confirms that I can only get 1000 points each time. However, the API doesn't return X-Pagination-Total value. I guess the workaround is to set the limit to 1000, and if the return has 1000 data points, then run the query again. When the return is less than 1000 data points, then it means there is nothing after the last query.

[abachman]

Ah! Yes, the X-Pagination-Total is a new header field. It's present on all data API responses, but wasn't added to our Access-Control-Expose-Headers header, which means if you're running code in a modern browser from a different origin than io.adafruit.com, the browser will prevent your code from programmatically accessing the field. Cross origin request security We've fixed that oversight, so you should see X-Pagination-Total now.

The limit + available check you describe also works fine, though, and is how I've written pagination code for IO in the past before we added the Total field. Only weird edge case is if you have exactly 1000 data points, but then you make the extra request and get an empty list, so it's no biggie.

Just remember, the Total value can be up to 5 minutes behind real-time, so if you're querying a quickly updating feed, you may get more values than you expect rather than exactly as many.


- adam b.