I am using the HUZZAH 32 esp32 feather.
I want to upload 50 numbers per second in adafruit io.
Because there is a limit of one data packet per second I tried to create a vector or an array of 50 numbers every second and then to upload that in adafruit io.
Unfortunately I can't use array or vector with the save() command.
I also tried to create a string with the 50 numbers and then upload that, this seems to work but only for around 20 numbers per string.
Is there any way of uploading a packet each time I use save(), or is there any other command for this matter?
Upload an array or a vector to adafruit io (Arduino)
Moderators: adafruit_support_bill, adafruit
Forum rules
If you're posting code, please make sure your code does not include your Adafruit IO Active Key or WiFi network credentials.
If you're posting code, please make sure your code does not include your Adafruit IO Active Key or WiFi network credentials.
- billos
- Posts: 20
- Joined: Thu Nov 16, 2017 9:16 am
- btreichel
- Posts: 79
- Joined: Tue Jul 25, 2017 3:00 pm
Re: Upload an array or a vector to adafruit io (Arduino)
You can use 'groups' to upload multiple data points. Not sure what the limit is on the number of items per group. Each item in a group does count as a feed.
"AdafruitIO_Group *group = io.group("roaster_data");"
and
" group->set("inlet-temp", inlet_temp);
group->set("bean-temp", bean_temp);
group->set("outlet-temp", outlet_temp);
group->set("ROR", ror);
group->set("ROR-10x", ror_10);
group->set("fan-data", fan_data);
group->set("gas-data", gas_data);
group->save();"
"AdafruitIO_Group *group = io.group("roaster_data");"
and
" group->set("inlet-temp", inlet_temp);
group->set("bean-temp", bean_temp);
group->set("outlet-temp", outlet_temp);
group->set("ROR", ror);
group->set("ROR-10x", ror_10);
group->set("fan-data", fan_data);
group->set("gas-data", gas_data);
group->save();"
- billos
- Posts: 20
- Joined: Thu Nov 16, 2017 9:16 am
Re: Upload an array or a vector to adafruit io (Arduino)
First of all Thanks!!
But I don't know if this could help me because ,do you know if I can upload 60 "groups" per minute or if I can upload 60 items per minute ?
For example if I have a group of 10 items, and the limit is 60 items per minute then I could apparently upload only 6 groups (6*10 = 60 items per minute).
But I don't know if this could help me because ,do you know if I can upload 60 "groups" per minute or if I can upload 60 items per minute ?
For example if I have a group of 10 items, and the limit is 60 items per minute then I could apparently upload only 6 groups (6*10 = 60 items per minute).
- btreichel
- Posts: 79
- Joined: Tue Jul 25, 2017 3:00 pm
Re: Upload an array or a vector to adafruit io (Arduino)
A group counts as one upload, once there, each item in the group counts as a feed. So, if a group allows 50 data points, that's one upload of the 50 feeds, if your account allows that many feeds.
- billos
- Posts: 20
- Joined: Thu Nov 16, 2017 9:16 am
Re: Upload an array or a vector to adafruit io (Arduino)
Nice, I will try then!
Thanks !!!
Thanks !!!
- btreichel
- Posts: 79
- Joined: Tue Jul 25, 2017 3:00 pm
Re: Upload an array or a vector to adafruit io (Arduino)
Let me know how it works
- abachman
- Posts: 352
- Joined: Mon Feb 01, 2010 12:48 pm
Re: Upload an array or a vector to adafruit io (Arduino)
Hey billos!
With the group publishing, each data point in the group counts as a separate data point against the total allowed data rate. So, a group message with 5 feeds would be 5 data points, all at once. If your limit is 30 points per second, you could publish to that group 6 times in a given 60 second period.
Regarding strategies for publishing large amounts of data, quickly, if you're not counting on IO displaying it anywhere you could turn off feed history and use just about any encoding scheme since that would let you publish up to 100KB at a time. You'd still need to accumulate and throttle posts on your own though, to avoid hitting rate limits. The Free account, as advertised, only allows 30 publish events per-minute, so no matter how you package the data you can't publish once a second.
We may eventually have a way of balancing feed lifetime against a higher data rate (e.g., 30 day expiration @ 30 points per minute versus 1 day expiration @ 900 points per minute), but right now you're constrained by our need to balance total data storage versus momentary data rate.
Bottom line, if you want or need charts / dashboards, data storage, and 50 values per second, IO is probably not a good choice :(
I'm curious to hear what you come up with and what kind of project you're working on, if you're willing to share.
- Adam
With the group publishing, each data point in the group counts as a separate data point against the total allowed data rate. So, a group message with 5 feeds would be 5 data points, all at once. If your limit is 30 points per second, you could publish to that group 6 times in a given 60 second period.
Regarding strategies for publishing large amounts of data, quickly, if you're not counting on IO displaying it anywhere you could turn off feed history and use just about any encoding scheme since that would let you publish up to 100KB at a time. You'd still need to accumulate and throttle posts on your own though, to avoid hitting rate limits. The Free account, as advertised, only allows 30 publish events per-minute, so no matter how you package the data you can't publish once a second.
We may eventually have a way of balancing feed lifetime against a higher data rate (e.g., 30 day expiration @ 30 points per minute versus 1 day expiration @ 900 points per minute), but right now you're constrained by our need to balance total data storage versus momentary data rate.
Bottom line, if you want or need charts / dashboards, data storage, and 50 values per second, IO is probably not a good choice :(
I'm curious to hear what you come up with and what kind of project you're working on, if you're willing to share.
- Adam
- btreichel
- Posts: 79
- Joined: Tue Jul 25, 2017 3:00 pm
Re: Upload an array or a vector to adafruit io (Arduino)
Adam,
Thats not the way that it has been working. I've been using groups to load 5 feeds just under the data rate. I'd tell you to look at my feed data, but I just purged it all trying to get rid of the two ghost feeds that are being counted against me. P.S. people in Discord have seen a public dashboard of this function.
In general, I presume thats the way that you throttle the broker, not a MQTT limit? I presume that if I set up my own broker I could set the limits where I want. P.S. not a complaint against the limits since I just using this for learning and experience.
Thats not the way that it has been working. I've been using groups to load 5 feeds just under the data rate. I'd tell you to look at my feed data, but I just purged it all trying to get rid of the two ghost feeds that are being counted against me. P.S. people in Discord have seen a public dashboard of this function.
In general, I presume thats the way that you throttle the broker, not a MQTT limit? I presume that if I set up my own broker I could set the limits where I want. P.S. not a complaint against the limits since I just using this for learning and experience.
- billos
- Posts: 20
- Joined: Thu Nov 16, 2017 9:16 am
Re: Upload an array or a vector to adafruit io (Arduino)
Hey Adam,
I am working on my Dipl0ma Thesis, and I'm trying to make a fall detector. So I want to upload data from an accelerometer at about 50Hz. Right now I am only concern about collecting data so I don't mind if I have 50 publishes per second (That I know I can't in adafruitIO) or if I have 1 publish with 100 values (2*50 --> 50Hz) every 2 seconds. Do you know how can I create a variable of 100 values in arduino and in the same time be able to upload it to adafrutIO ?
Because I tried with string but after putting over 20 values something went wrong and my Huzzah32 crashed. I don't know what is the problem or if my memory is full but I supposedly have about 520Kb of SRAM.
Thanks in advance!!!
I am working on my Dipl0ma Thesis, and I'm trying to make a fall detector. So I want to upload data from an accelerometer at about 50Hz. Right now I am only concern about collecting data so I don't mind if I have 50 publishes per second (That I know I can't in adafruitIO) or if I have 1 publish with 100 values (2*50 --> 50Hz) every 2 seconds. Do you know how can I create a variable of 100 values in arduino and in the same time be able to upload it to adafrutIO ?
Because I tried with string but after putting over 20 values something went wrong and my Huzzah32 crashed. I don't know what is the problem or if my memory is full but I supposedly have about 520Kb of SRAM.
Thanks in advance!!!
- abachman
- Posts: 352
- Joined: Mon Feb 01, 2010 12:48 pm
Re: Upload an array or a vector to adafruit io (Arduino)
Ah ha! You are correct.btreichel wrote:Adam,
Thats not the way that it has been working. I've been using groups to load 5 feeds just under the data rate. I'd tell you to look at my feed data, but I just purged it all trying to get rid of the two ghost feeds that are being counted against me. P.S. people in Discord have seen a public dashboard of this function.
Our HTTP API is correctly counting group and bulk (multiple feed data points in one HTTP request) publish events as separate points against the data rate, but the MQTT interface is not, currently. I'm deploying a fix for that this morning and will probably write up a blog post about it too. We spend a lot of time trying to get the throttling system correct, fair, and predictable since that's about the only tool we have for protecting the availability of the system.
And I've taken care of the ghost feeds. They shouldn't be a problem again, but if they are please let us know.
Yes, totally. Our limits are determined by a combination of: 1) the cost of data storage and 2) the cost of servers to support a given data rate across all accounts. 1 and 2 combine to give us a rough estimate of how much each free IO account could cost if it uses the maximum available resources. We picked a reasonable cost and then set our limits. None of the limits we set are inherent to MQTT or any other communication protocol. Our MQTT broker receives every message that's sent, regardless of whether it forwards it to subscribers or stores it permanently, it just counts them up and then drops / ignores extras if they're arriving too quickly.In general, I presume thats the way that you throttle the broker, not a MQTT limit? I presume that if I set up my own broker I could set the limits where I want.
If you wanted to set up your own MQTT broker you would only be limited by your hardware and network. I'm willing to bet you could push a lot more data through an entirely local MQTT broker, even on something like a Raspberry Pi. If you're comfortable setting up and operating a tool like Home Assistant, that would be a good way to get a lot more flexibility from your devices.
There are also IoT services like Amazon's that will let you scale up to pretty huge data rates (9000 publish events per second, per account and 100 publish events per second per client) but in most cases they charge per message. In Amazon's case, they charge $0.000005 per message ($5 per million) so you'd pay about $5 per month for the same data rate we're providing for free, not including storage.
It's all tradeoffs :D We're trying to favor stable, simple, and clear over expert, technical, and infinitely scalable. Where we get it wrong, we do our best to make corrections.
- Adam
- btreichel
- Posts: 79
- Joined: Tue Jul 25, 2017 3:00 pm
Re: Upload an array or a vector to adafruit io (Arduino)
Well, I'm happy and sad at the same time. However, I did get to demonstrate, and experiment with functionality that I need for a project that I'm working on. I'm toying with the idea of starting my own company since I committed the corporate sin of getting within a decade of retirement age, so they 'retired me'; of course before I got the benefits. Funding of this project is of course a function of how well my consulting business goes; one funds the other.
My application involves 6 to 7 feeds every second while the coffee roaster is running, with about 15 minutes a batch. This is an IoT add on for community purposes of watching someone else roast, and/or logging and re-using roast profiles. So there would need to be more of a database backend. The main idea behind it is a community of (hobby) roasters that can roast, and discuss roasting, based upon a common data set and common roasting platform.
While I have and old desktop (was great and powerful 8 years ago) and a 100+ mB fiber connection, the idea of hosting on a PI is interesting from the aspect of having a portable MQTT host to go with the demo roaster (if it gets built). Currently its just a 3- printed demo; which would be a very bad idea to incorporate a natural gas burner into.
P.S. Ghost feeds fixed, and you can see how my skills have gotten old and out-of-date as the beard has gone grey.
My application involves 6 to 7 feeds every second while the coffee roaster is running, with about 15 minutes a batch. This is an IoT add on for community purposes of watching someone else roast, and/or logging and re-using roast profiles. So there would need to be more of a database backend. The main idea behind it is a community of (hobby) roasters that can roast, and discuss roasting, based upon a common data set and common roasting platform.
While I have and old desktop (was great and powerful 8 years ago) and a 100+ mB fiber connection, the idea of hosting on a PI is interesting from the aspect of having a portable MQTT host to go with the demo roaster (if it gets built). Currently its just a 3- printed demo; which would be a very bad idea to incorporate a natural gas burner into.
P.S. Ghost feeds fixed, and you can see how my skills have gotten old and out-of-date as the beard has gone grey.
Forum rules
If you're posting code, please make sure your code does not include your Adafruit IO Active Key or WiFi network credentials.
If you're posting code, please make sure your code does not include your Adafruit IO Active Key or WiFi network credentials.