0

Update rate and throttling
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Update rate and throttling

by IAM3358 on Fri Mar 01, 2019 10:20 am

I have software which publishes 4 values every 10 seconds, i.e. 24 per minute which should be acceptable on the free account. However my update rate keeps getting reported incorrectly. At the moment the side bar is showing an update rate of 96 /min, four times what it should be. Hence the Monitor is reporting "data rate limit reached, 44 seconds until throttle reset".
Is this a bug or am I doing something wrong in the MQTT, it's my first go at this.

A related issue: I publish 4 values to 4 feeds, if I grouped the feeds and published once to the group, would this lower my update-rate by 4? Or is it per feed item?

IAM3358
 
Posts: 7
Joined: Thu Jan 24, 2019 11:19 am

Re: Update rate and throttling

by abachman on Fri Mar 01, 2019 11:06 am

IAM3358,


The first thing to check would be your [url]io.adafruit.com/monitor[/url] page when you're signed in, which should show you all data as it arrives. If the messages come quickly when you start up your sketch/program/device, then there may be an issue with your timing code.

If you're publishing from hardware, something to try with that is adding a small indication on the device when a publish event happens. Blinking the onboard LED is handy for this if you're using a Feather or similar board.

Feel free to share your sketch here if you'd like to have anyone else take a look at it.

A related issue: I publish 4 values to 4 feeds, if I grouped the feeds and published once to the group, would this lower my update-rate by 4? Or is it per feed item?


The rate limit is per data point create / update / delete. Group publishing is handy for keeping data organized, but the rate limit is still per feed.

If you're want to handle charting yourself, you could pack multiple values into a single data point using CSV or JSON. The rate limit will still be the same, but IO will store up to 1024 bytes per data point.


- Adam B>

abachman
 
Posts: 276
Joined: Mon Feb 01, 2010 12:48 pm

Re: Update rate and throttling

by IAM3358 on Fri Mar 01, 2019 11:23 am

I have an embedded board producing the data, it's my own code. I have a debug output and can see each publish being done, 4 every 10 seconds.
The Dashboard plots of the data look correct from what I can see.

Are the gps long, lat, elev values associated with a Value counted as additional data too? As in :
{
"value": 22.587,
"lat": 38.1123,
"lon": -91.2325,
"ele": 112
}

If so then I'm publishing 7 items every 10 seconds, 42 per minute. That might be it.
However the io.adafruit.com/monitor is showing 88 / min

IAM3358
 
Posts: 7
Joined: Thu Jan 24, 2019 11:19 am

Re: Update rate and throttling

by IAM3358 on Fri Mar 01, 2019 11:28 am

If I have 9 digit lat / long (excluding the decimal place), does that get accepted and used? The Log is only showing 4 decimal places and no "ele".

IAM3358
 
Posts: 7
Joined: Thu Jan 24, 2019 11:19 am

Re: Update rate and throttling

by abachman on Fri Mar 01, 2019 11:38 am

If you're publishing to a group endpoint, HTTP or MQTT, then the payload you describe would be interpreted as 4 different feeds. In that case, though, your feed listing page would show feeds with the names "value", "lat", "lon", and "ele".

Does the data rate only go up when your code is running?

What library are you using to send data to IO?


- ab

abachman
 
Posts: 276
Joined: Mon Feb 01, 2010 12:48 pm

Re: Update rate and throttling

by IAM3358 on Fri Mar 01, 2019 11:57 am

I've not created any groups so far. I'm using MQTT lib from Quectel.
The Value appears in the feeds, but not lat, lon, ele.
I can plot a Map point of the value however. The LiveData shows "2019/03/01 3:44pm voc 57 51.0003, 1.0001"

The data rate only increases when my code is running.

OK, I've cut my publish rate to every 15 seconds, 4 feeds, but one feed has lat/long in the json as above.
The "Current Rate" is now listed as 16/min which looks like 4 uploads of 4 data items. This must be not counting the lat/long. and I've trimmed the lat / long to 4 dec places.

IAM3358
 
Posts: 7
Joined: Thu Jan 24, 2019 11:19 am

Re: Update rate and throttling

by IAM3358 on Fri Mar 01, 2019 1:53 pm

I have a feeling the Current Usage value doesn't agree with the throttling mechanism, the current usage ignores the lat/long meta data, and the throttling doesn't.
That's my guess from what I can see.
Currently running OK with 5 posts of 4 data items per min (current usage says 20/min), but one feed item has lat and long, so I guess the throttle mechanism is seeing 5*6 items=30 so just OK.

I think I'll upgrade, but its useful to get a better idea of how the things work in AdafruitIO.
Thanks for your help AB.

IAM3358
 
Posts: 7
Joined: Thu Jan 24, 2019 11:19 am

Re: Update rate and throttling

by abachman on Fri Mar 01, 2019 4:58 pm

I have a feeling the Current Usage value doesn't agree with the throttling mechanism


The current rate count may be off by a small amount, depending on at which second the request was made, but it shouldn't be a factor of 4.. Our rate limiting algorithm uses a strict rolling window function to count each event with a 1 minute expiration, so if you PUBLISH and then check immediately, you'll see a rate of 1, but if you publish every 2 seconds and check after a minute, you could see either 29 or 30.

Also, location data is not counted against the data rate limit. The data point value and (optional) location metadata are considered a single thing for the purpose of rate limiting.

If the data rate spikes while your code is running, then your code is sending that amount of discrete PUBLISH events. Is it possible your MQTT library is splitting packets somehow or otherwise sending PUBLISH messages more frequently that once per line-of-code-where-data-is-sent? Are you just sending a single PUBLISH to the appropriate feed topic {username}/feeds/{feed_key}/json, or are you also sending other MQTT commands?

It would be super helpful to get the relevant code for your project or a link to the source for the library you're using, if it can be shared. If not, you might have better luck debugging the issue by directing traffic to an MQTT broker you control and watching everything the broker sees.


- adam

abachman
 
Posts: 276
Joined: Mon Feb 01, 2010 12:48 pm

Please be positive and constructive with your questions and comments.