0

Only 1000 data points?
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Only 1000 data points?

by kevinljxljx on Tue Feb 12, 2019 12:22 am

I am trying to use the data from adafruit.io with my project, but no matter what I try, adafruit.io only gives 1000 data points to my javascript program. ( I am using XMLHttpRequest). I tried to set limit to 2000 but without luck. ( I have 1200 or so data points under my feed). Is the maximum data points I can get is 1000? or am I missing something?

kevinljxljx
 
Posts: 10
Joined: Mon Feb 04, 2013 1:44 am

Re: Only 1000 data points?

by ShelM on Tue Feb 12, 2019 2:24 am

Hi Kevin...

When you have a feed open, look on the righthand side of the page - you'll see several options which define how your feed works. The one you are interested in is "Feed History". The reason you're interested in this is that when History is ON, you get 1KB data storage, when OFF you get 100KB.

That said, It's not clear to me if 1KB actually *means* 1KB, or if it means 1K data points.

Here is the text from the Feed History menu settings box:

"Update Feed History Storage
Turning history off for your feed will mean only the last value received is stored. History is on for all new feeds by default.

While history is turned ON, feed data is limited to 1KB (1024 bytes) in size.

While history is turned OFF, feed data is limited to 100KB (102400 bytes) in size"

Bests,
...ShelM

ShelM
 
Posts: 58
Joined: Sun Mar 11, 2018 1:45 am

Re: Only 1000 data points?

by abachman on Tue Feb 12, 2019 12:11 pm

hi kevin,


The API only returns a maximum of 1000 data points at a time, you'll have to "paginate" the data to get the whole feed.

When you do a data query, the results are always sorted newest-to-oldest and include x-pagination-* headers. For example,

Code: Select all | TOGGLE FULL SIZE
X-Pagination-Limit: 1000
X-Pagination-Total: 84548
X-Pagination-Start: 2019-02-11T22:52:18.103+0000
X-Pagination-End: 2019-02-12T16:03:00.694+0000
X-Pagination-Count: 1000


Limit is either the requested limit or 1000, whichever is less; Total is the total number of data points in the feed--note, this value may be up to 5 minutes behind real time; Start is the timestamp on the oldest value; End is the timestamp on the newest value; and Count is the number of data points in the current request. Whenever Limit and Count are both 1000 and Total is more than 1000, that's evidence that more data is available. You can get the next 1000 data points by using the X-Pagination-Start value OR the created_at value of the oldest data point in the API response as the end_time parameter in your next request to the data API.

I always end up thinking about it visually. On a timeline the idea kind of looks like this:
api-pagination.png
api-pagination.png (29.28 KiB) Viewed 223 times


NOTE: long running, frequently updated feeds could have more than a hundred "pages" of data. If you make requests without a delay in between, you could hit a rate limited. Watch for 429 HTTP error responses.

Regarding the data storage and feed history, storage size in this instance is the per-data point value size limit. With history on, meaning we preserve every data point, each data point value can be at most 1KB. With history off, meaning we only preserve the most recent data point, each value can be at most 100KB.


- adam b.

abachman
 
Posts: 327
Joined: Mon Feb 01, 2010 12:48 pm

Re: Only 1000 data points?

by kevinljxljx on Wed Feb 13, 2019 1:47 pm

Hi adam,
Thanks a lot! This confirms that I can only get 1000 points each time. However, the API doesn't return X-Pagination-Total value. I guess the workaround is to set the limit to 1000, and if the return has 1000 data points, then run the query again. When the return is less than 1000 data points, then it means there is nothing after the last query.

kevinljxljx
 
Posts: 10
Joined: Mon Feb 04, 2013 1:44 am

Re: Only 1000 data points?

by abachman on Thu Feb 14, 2019 1:50 pm

Ah! Yes, the X-Pagination-Total is a new header field. It's present on all data API responses, but wasn't added to our Access-Control-Expose-Headers header, which means if you're running code in a modern browser from a different origin than io.adafruit.com, the browser will prevent your code from programmatically accessing the field. Cross origin request security We've fixed that oversight, so you should see X-Pagination-Total now.

The limit + available check you describe also works fine, though, and is how I've written pagination code for IO in the past before we added the Total field. Only weird edge case is if you have exactly 1000 data points, but then you make the extra request and get an empty list, so it's no biggie.

Just remember, the Total value can be up to 5 minutes behind real-time, so if you're querying a quickly updating feed, you may get more values than you expect rather than exactly as many.


- adam b.

abachman
 
Posts: 327
Joined: Mon Feb 01, 2010 12:48 pm

Please be positive and constructive with your questions and comments.