0

Operational Questions: Throttling and Monitor/API errors
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Operational Questions: Throttling and Monitor/API errors

by koaiwi on Sat Jan 13, 2018 2:14 am

Hi. I have the following questions:

1. On the monitor what does "4 actions requested, 3 remaining" mean, what causes it and what is the impact on data.
2. When requesting data I am getting from time to time "Retry Later" response. What is causing this?
3. Throttling. How is this calculated? i.e. 60/min. Is it actual feed data points or an API call? What is the impact to the data when the limit is exceeded? Is it lost or queued for a later data commit?

Thanks.

koaiwi
 
Posts: 24
Joined: Thu Dec 28, 2017 10:11 am

Re: Operational Questions: Throttling and Monitor/API errors

by abachman on Mon Jan 15, 2018 12:56 pm

1. On the monitor what does "4 actions requested, 3 remaining" mean, what causes it and what is the impact on data.


That means you've sent a request that includes multiple data points and publishing all of them at once will exceed the data rate, so none were published. We've chosen this strategy to avoid partially written data.

2. When requesting data I am getting from time to time "Retry Later" response. What is causing this?


That means you hit our HTTP API request limit. This is a more basic denial of service throttling system that limits overall HTTP API requests to 60 or fewer per minute. There is no API available for tracking that limit. If you're doing a lot of repetitive GET requests to ping feeds, you might have better luck using our MQTT interface/url] and subscribing as needed.

[url]3. Throttling. How is this calculated? i.e. 60/min. Is it actual feed data points or an API call? What is the impact to the data when the limit is exceeded? Is it lost or queued for a later data commit?


We use a rolling-window throttling algorithm to track data publishing. I've posted a detailed answer in the past, so I'll give you a link instead of copy-pasting that: https://forums.adafruit.com/viewtopic.php?f=56&t=125088&p=624726&hilit=throttle+rolling#p624726. We are tracking attempts to write data to feeds. So things that count against that limit are: publishing valid data to a feed, publishing multiple data points to a feed (via bulk API), publishing to multiple feeds in a group, and publishing invalid data (e.g., blank value or data is too big) even if it's not stored.

Data sent over the limit is dropped/ignored and would have to be resent if you need it to be stored. Throttle errors when publishing should result in no data being stored, whether the request was to store one data point or many.


- adam

abachman
 
Posts: 185
Joined: Mon Feb 01, 2010 12:48 pm

Re: Operational Questions: Throttling and Monitor/API errors

by koaiwi on Tue Jan 16, 2018 10:56 am

Hi

Thanks. The linked post is clear.

But still not quite getting error of 4 actions requested and 3 remaining. In the context of submitting several posts sequentially each post having several different feed data points for a group.

For instance is the 4 relating to the 4 sequential posts (actions) it is looking at and the 3 meaning that 3 of the 4 posts have been abandoned because if they were processed then the total feed data points ( lets say an aggregate of 12 which is made up of 4 feed data points for each of the 3 posts) would have blown the 60/min throttle.

So it is up to me to analyse the post response looking for a phrase that indicates that the throttle limit is exceeded and then repost the data later? What should I look for?

I see the api call now that will return the current rate. I could call that api and analyse the response prior to the post and only post when I had enough free rate?

Thanks.

koaiwi
 
Posts: 24
Joined: Thu Dec 28, 2017 10:11 am

Re: Operational Questions: Throttling and Monitor/API errors

by abachman on Tue Jan 16, 2018 11:13 am

But still not quite getting error of 4 actions requested and 3 remaining. In the context of submitting several posts sequentially each post having several different feed data points for a group.


The "4 requested" refers to countable data actions (create data, delete, or modify data point) and the "3 remaining" refers to the available quota. So, in a multi-feed group publish context, the system received a request that include 4 actions but only had slots available for 3 of them.

I see the api call now that will return the current rate. I could call that api and analyse the response prior to the post and only post when I had enough free rate?


That strategy seems like it would work as long as you are otherwise in control of all the inputs to your account.

It's also worth noting that the current free limit (60/min) is 2x the advertised rate (30/min). We're going to be adjusting it in the near future, so any setups that are riding the line now will be over later.


- adam

abachman
 
Posts: 185
Joined: Mon Feb 01, 2010 12:48 pm

Re: Operational Questions: Throttling and Monitor/API errors

by koaiwi on Tue Jan 16, 2018 4:21 pm

koaiwi wrote:The "4 requested" refers to countable data actions (create data, delete, or modify data point) and the "3 remaining" refers to the available quota. So, in a multi-feed group publish context, the system received a request that include 4 actions but only had slots available for 3 of them.


So still not quite clear. If this was just a single post with 4 data points and the error on the console referred to this post and said 4 received and 3 remaining. Does that mean the whole post was rejected i.e. all 4 points or just the 3 out of the 4. I think it is the whole post?

So in this situation what should I look for in the response from the post that will tell me the there was an issue.

koaiwi wrote:It's also worth noting that the current free limit (60/min) is 2x the advertised rate (30/min). We're going to be adjusting it in the near future, so any setups that are riding the line now will be over later.


Are you able to indicate what the near future may be and what the rate may get to. It may inform me whether to do something now about my use of the system or just wait.

The way I think you say it works is a rolling 60 secs which means that if all my devices were silent for 5 mins and then they all burst into life and fired 120 data points in 60 secs then I would blow the limit. I think this is where it differes from Xively which had a larger windo for working out the limits.

koaiwi
 
Posts: 24
Joined: Thu Dec 28, 2017 10:11 am

Re: Operational Questions: Throttling and Monitor/API errors

by koaiwi on Fri Jan 19, 2018 7:40 am

Any thoughts on the above Adam?

Thanks.

koaiwi
 
Posts: 24
Joined: Thu Dec 28, 2017 10:11 am

Re: Operational Questions: Throttling and Monitor/API errors

by abachman on Fri Jan 19, 2018 3:52 pm

I mentioned the behavior on throttle errors in an earlier message: "Throttle errors when publishing should result in no data being stored, whether the request was to store one data point or many."

Publish requests are "atomic" with respect to throttling in that the whole request either succeeds (all data written) or fails (no data written). So a group publish with 4 feeds shouldn't store anything if there's only room under the limit for 3. NOTE: failed publish events still count against the data rate limit.

Are you able to indicate what the near future may be and what the rate may get to. It may inform me whether to do something now about my use of the system or just wait./quote]

Like I said, the Free plan right now, as advertised, is 30 points-per-second.
Screen Shot 2018-01-19 at 2.42.27 PM.png
Screen Shot 2018-01-19 at 2.42.27 PM.png (57.67 KiB) Viewed 226 times
As a courtesy, we left it at it's original level when we launched IO Plus but will be reducing it incrementally starting today. We're going to start by dropping it by 10 from 60 to 50.

The way I think you say it works is a rolling 60 secs which means that if all my devices were silent for 5 mins and then they all burst into life and fired 120 data points in 60 secs then I would blow the limit. I think this is where it differes from Xively which had a larger windo for working out the limits.


Your understanding is correct, we only allow bursting of up to MAX_PER_MINUTE data points. We used a "token bucket" style algorithm for throttling in the past, but that resulted in spikes in traffic as aggressive devices on auto-pilot hit us with 300 points every 5 minutes. An ESP8266 can push a surprising amount of data if it's running without delays and we're trying to process and store all of it. A key difference between "token bucket" and "rolling window" is that rolling window means once an account is throttled, it continues to be throttled until the data rate slows.


- adam

abachman
 
Posts: 185
Joined: Mon Feb 01, 2010 12:48 pm

Re: Operational Questions: Throttling and Monitor/API errors

by koaiwi on Sat Jan 20, 2018 7:15 am

Hi

That's clear now thanks.

"As a courtesy, we left it at it's original level when we launched IO Plus but will be reducing it incrementally starting today. We're going to start by dropping it by 10 from 60 to 50."

But what is worrying is what you said about starting to drop the throttle threshold. Are you saying that having just signed up to a threshold of 60 for $10/month that now my threshold is reduced to 50 shots per minute for $10? Am I resign that right? Or are your referring to the free option?

koaiwi
 
Posts: 24
Joined: Thu Dec 28, 2017 10:11 am

Re: Operational Questions: Throttling and Monitor/API errors

by abachman on Sat Jan 20, 2018 8:19 am

Oh! Just the free plan. Plus is not changing at all. Sorry for the confusion.


- adam

abachman
 
Posts: 185
Joined: Mon Feb 01, 2010 12:48 pm

Re: Operational Questions: Throttling and Monitor/API errors

by koaiwi on Sat Jan 20, 2018 8:52 am

Thought so. No problem. It's just that the post had a mixed message the way it was sequenced that may have been confusing if left unclarified :)

Thanks.

BTW I'm not getting any more retry messages in my gets. But I will slow them down a bit when I can get back to the code. With Blynk you have to be bit careful and crafty with delay(). It doesn't like it much.

koaiwi
 
Posts: 24
Joined: Thu Dec 28, 2017 10:11 am

Please be positive and constructive with your questions and comments.