0

Operational Questions: Throttling and Monitor/API errors
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Operational Questions: Throttling and Monitor/API errors

by koaiwi on Sat Jan 13, 2018 2:14 am

Hi. I have the following questions:

1. On the monitor what does "4 actions requested, 3 remaining" mean, what causes it and what is the impact on data.
2. When requesting data I am getting from time to time "Retry Later" response. What is causing this?
3. Throttling. How is this calculated? i.e. 60/min. Is it actual feed data points or an API call? What is the impact to the data when the limit is exceeded? Is it lost or queued for a later data commit?

Thanks.

koaiwi
 
Posts: 14
Joined: Thu Dec 28, 2017 10:11 am

Re: Operational Questions: Throttling and Monitor/API errors

by abachman on Mon Jan 15, 2018 12:56 pm

1. On the monitor what does "4 actions requested, 3 remaining" mean, what causes it and what is the impact on data.


That means you've sent a request that includes multiple data points and publishing all of them at once will exceed the data rate, so none were published. We've chosen this strategy to avoid partially written data.

2. When requesting data I am getting from time to time "Retry Later" response. What is causing this?


That means you hit our HTTP API request limit. This is a more basic denial of service throttling system that limits overall HTTP API requests to 60 or fewer per minute. There is no API available for tracking that limit. If you're doing a lot of repetitive GET requests to ping feeds, you might have better luck using our MQTT interface/url] and subscribing as needed.

[url]3. Throttling. How is this calculated? i.e. 60/min. Is it actual feed data points or an API call? What is the impact to the data when the limit is exceeded? Is it lost or queued for a later data commit?


We use a rolling-window throttling algorithm to track data publishing. I've posted a detailed answer in the past, so I'll give you a link instead of copy-pasting that: https://forums.adafruit.com/viewtopic.php?f=56&t=125088&p=624726&hilit=throttle+rolling#p624726. We are tracking attempts to write data to feeds. So things that count against that limit are: publishing valid data to a feed, publishing multiple data points to a feed (via bulk API), publishing to multiple feeds in a group, and publishing invalid data (e.g., blank value or data is too big) even if it's not stored.

Data sent over the limit is dropped/ignored and would have to be resent if you need it to be stored. Throttle errors when publishing should result in no data being stored, whether the request was to store one data point or many.


- adam

abachman
 
Posts: 131
Joined: Mon Feb 01, 2010 12:48 pm

Re: Operational Questions: Throttling and Monitor/API errors

by koaiwi on Tue Jan 16, 2018 10:56 am

Hi

Thanks. The linked post is clear.

But still not quite getting error of 4 actions requested and 3 remaining. In the context of submitting several posts sequentially each post having several different feed data points for a group.

For instance is the 4 relating to the 4 sequential posts (actions) it is looking at and the 3 meaning that 3 of the 4 posts have been abandoned because if they were processed then the total feed data points ( lets say an aggregate of 12 which is made up of 4 feed data points for each of the 3 posts) would have blown the 60/min throttle.

So it is up to me to analyse the post response looking for a phrase that indicates that the throttle limit is exceeded and then repost the data later? What should I look for?

I see the api call now that will return the current rate. I could call that api and analyse the response prior to the post and only post when I had enough free rate?

Thanks.

koaiwi
 
Posts: 14
Joined: Thu Dec 28, 2017 10:11 am

Re: Operational Questions: Throttling and Monitor/API errors

by abachman on Tue Jan 16, 2018 11:13 am

But still not quite getting error of 4 actions requested and 3 remaining. In the context of submitting several posts sequentially each post having several different feed data points for a group.


The "4 requested" refers to countable data actions (create data, delete, or modify data point) and the "3 remaining" refers to the available quota. So, in a multi-feed group publish context, the system received a request that include 4 actions but only had slots available for 3 of them.

I see the api call now that will return the current rate. I could call that api and analyse the response prior to the post and only post when I had enough free rate?


That strategy seems like it would work as long as you are otherwise in control of all the inputs to your account.

It's also worth noting that the current free limit (60/min) is 2x the advertised rate (30/min). We're going to be adjusting it in the near future, so any setups that are riding the line now will be over later.


- adam

abachman
 
Posts: 131
Joined: Mon Feb 01, 2010 12:48 pm

Re: Operational Questions: Throttling and Monitor/API errors

by koaiwi on Tue Jan 16, 2018 4:21 pm

koaiwi wrote:The "4 requested" refers to countable data actions (create data, delete, or modify data point) and the "3 remaining" refers to the available quota. So, in a multi-feed group publish context, the system received a request that include 4 actions but only had slots available for 3 of them.


So still not quite clear. If this was just a single post with 4 data points and the error on the console referred to this post and said 4 received and 3 remaining. Does that mean the whole post was rejected i.e. all 4 points or just the 3 out of the 4. I think it is the whole post?

So in this situation what should I look for in the response from the post that will tell me the there was an issue.

koaiwi wrote:It's also worth noting that the current free limit (60/min) is 2x the advertised rate (30/min). We're going to be adjusting it in the near future, so any setups that are riding the line now will be over later.


Are you able to indicate what the near future may be and what the rate may get to. It may inform me whether to do something now about my use of the system or just wait.

The way I think you say it works is a rolling 60 secs which means that if all my devices were silent for 5 mins and then they all burst into life and fired 120 data points in 60 secs then I would blow the limit. I think this is where it differes from Xively which had a larger windo for working out the limits.

koaiwi
 
Posts: 14
Joined: Thu Dec 28, 2017 10:11 am

Please be positive and constructive with your questions and comments.