0

Occasional 404 error and lost data...
Moderators: adafruit_support_bill, adafruit

Forum rules
If you're posting code, please make sure your code does not include your Adafruit IO Active Key or WiFi network credentials.
Please be positive and constructive with your questions and comments.

Re: Occasional 404 error and lost data...

by brubell on Thu Aug 13, 2020 12:16 pm

Ok - this is the expected scenario for a feed that hasn't been updated in 2 months (60+ days).

The time to live for the data points are 60 days on an io+ account. The data rolls off as it expires beyond that 60 day threshold. The '7200' value is the last_value saved, and we store that differently for convenience and performance reasons, and do not roll off that value. The historical data does expire though.

Free accounts are provided 30 days of storage, and io+ accounts are 60 days of storage.

brubell
 
Posts: 1193
Joined: Fri Jul 17, 2015 10:33 pm

Re: Occasional 404 error and lost data...

by cstanke on Thu Aug 13, 2020 12:37 pm

brubell wrote:The '7200' value is the last_value saved, and we store that differently for convenience and performance reasons, and do not roll off that value


The underlying issue here is that the "last_value saved” does roll off since the result of reading the feed after 60 days of no updates is a 404 error—as if the feed no longer exists. The result is an incongruity where the feed page shows a last value but that value is not accessible from the feed itself.

Is there no way to maintain a persistent value in a feed that is rarely updated (beyond caching it on the device and handling the 404 error state)? Seems clunky to roll off everything such that feed effectively becomes non-existent after 60 days of no changes.

cstanke
 
Posts: 71
Joined: Thu Jul 10, 2014 8:18 pm

Re: Occasional 404 error and lost data...

by jwcooper on Thu Aug 13, 2020 2:28 pm

I think in this case, the bug may be that we are using that last_value for that table, and probably shouldn't be, as it's not really indicative of the data records stored within that feed once all of the data rolls off.

The feed itself doesn't 404, does it? I assume calling
Code: Select all | TOGGLE FULL SIZE
/api/v2/{username}/feeds/{feed_key}
would return all of the metadata for that feed.

I would expect calling
Code: Select all | TOGGLE FULL SIZE
/api/v2/{username}/feeds/{feed_key}/data
something like that to return a 404, as there isn't any actual data records. Only the metadata of the last_value stored within that feed.

This is all sort of technical behind the scenes information, but I think it really comes down to is that there is a bug in that table showing data, when there really isn't any data. The value of seeing the last_value may be lost if it's confusing that there isn't actually any data records held within that feed.

We'll file an issue internally and sort out if we should return the last_value always (even if there is no data), or if we should update that UI to show 'no data' once the data records have expired.

jwcooper
 
Posts: 743
Joined: Tue May 01, 2012 9:08 pm

Re: Occasional 404 error and lost data...

by cstanke on Thu Aug 13, 2020 3:22 pm

jwcooper wrote:The feed itself doesn't 404, does it? I assume calling
Code: Select all | TOGGLE FULL SIZE
/api/v2/{username}/feeds/{feed_key}
would return all of the metadata for that feed.

Correct, using curl and the API url, the feed itself does indeed return metadata. Until you mentioned that, I'm not sure I would have taken the time to try it because I've only seen this issue using the CircuitPython http client object. For instance (pseudo-code):
Code: Select all | TOGGLE FULL SIZE
from adafruit_io.adafruit_io import IO_HTTP
io = IO_HTTP(ADAFRUIT_IO_USER, ADAFRUIT_IO_KEY, wifi)
apod_delay_feed = io.get_feed('pyportal-01.apod-delay')
io.receive_data(apod_delay_feed['key'])

The io.receive_data call returns the 404 error as shown in the PyPortal image in one of my previous posts (see below).
Image
Granted, catching the AdafruitIO_RequestError (which I wasn't doing) could have helped here. But, a 404 on a request for data, from a valid feed, with a known last value kinda seems weird, no?

jwcooper wrote:We'll file an issue internally and sort out if we should return the last_value always (even if there is no data), or if we should update that UI to show 'no data' once the data records have expired.

I expect what you ultimately do will warrant internal dev discussions, but my 2¢ is that if "last_value" is available but historical data has been purged, the CircuitPython case (at least) shouldn't return a 404 error, it should just offer up the last known value.

cstanke
 
Posts: 71
Joined: Thu Jul 10, 2014 8:18 pm

Please be positive and constructive with your questions and comments.