Upload an array or a vector to adafruit io (Arduino)

Moderators: adafruit_support_bill, adafruit

Forum rules
If you're posting code, please make sure your code does not include your Adafruit IO Active Key or WiFi network credentials.
Locked
User avatar
billos
 
Posts: 20
Joined: Thu Nov 16, 2017 9:16 am

Upload an array or a vector to adafruit io (Arduino)

Post by billos »

I am using the HUZZAH 32 esp32 feather.

I want to upload 50 numbers per second in adafruit io.
Because there is a limit of one data packet per second I tried to create a vector or an array of 50 numbers every second and then to upload that in adafruit io.
Unfortunately I can't use array or vector with the save() command.
I also tried to create a string with the 50 numbers and then upload that, this seems to work but only for around 20 numbers per string.

Is there any way of uploading a packet each time I use save(), or is there any other command for this matter?

User avatar
btreichel
 
Posts: 79
Joined: Tue Jul 25, 2017 3:00 pm

Re: Upload an array or a vector to adafruit io (Arduino)

Post by btreichel »

You can use 'groups' to upload multiple data points. Not sure what the limit is on the number of items per group. Each item in a group does count as a feed.

"AdafruitIO_Group *group = io.group("roaster_data");"

and

" group->set("inlet-temp", inlet_temp);
group->set("bean-temp", bean_temp);
group->set("outlet-temp", outlet_temp);
group->set("ROR", ror);
group->set("ROR-10x", ror_10);
group->set("fan-data", fan_data);
group->set("gas-data", gas_data);
group->save();"

User avatar
billos
 
Posts: 20
Joined: Thu Nov 16, 2017 9:16 am

Re: Upload an array or a vector to adafruit io (Arduino)

Post by billos »

First of all Thanks!!
But I don't know if this could help me because ,do you know if I can upload 60 "groups" per minute or if I can upload 60 items per minute ?
For example if I have a group of 10 items, and the limit is 60 items per minute then I could apparently upload only 6 groups (6*10 = 60 items per minute).

User avatar
btreichel
 
Posts: 79
Joined: Tue Jul 25, 2017 3:00 pm

Re: Upload an array or a vector to adafruit io (Arduino)

Post by btreichel »

A group counts as one upload, once there, each item in the group counts as a feed. So, if a group allows 50 data points, that's one upload of the 50 feeds, if your account allows that many feeds.

User avatar
billos
 
Posts: 20
Joined: Thu Nov 16, 2017 9:16 am

Re: Upload an array or a vector to adafruit io (Arduino)

Post by billos »

Nice, I will try then!
Thanks !!!

User avatar
btreichel
 
Posts: 79
Joined: Tue Jul 25, 2017 3:00 pm

Re: Upload an array or a vector to adafruit io (Arduino)

Post by btreichel »

Let me know how it works

User avatar
abachman
 
Posts: 352
Joined: Mon Feb 01, 2010 12:48 pm

Re: Upload an array or a vector to adafruit io (Arduino)

Post by abachman »

Hey billos!


With the group publishing, each data point in the group counts as a separate data point against the total allowed data rate. So, a group message with 5 feeds would be 5 data points, all at once. If your limit is 30 points per second, you could publish to that group 6 times in a given 60 second period.

Regarding strategies for publishing large amounts of data, quickly, if you're not counting on IO displaying it anywhere you could turn off feed history and use just about any encoding scheme since that would let you publish up to 100KB at a time. You'd still need to accumulate and throttle posts on your own though, to avoid hitting rate limits. The Free account, as advertised, only allows 30 publish events per-minute, so no matter how you package the data you can't publish once a second.

We may eventually have a way of balancing feed lifetime against a higher data rate (e.g., 30 day expiration @ 30 points per minute versus 1 day expiration @ 900 points per minute), but right now you're constrained by our need to balance total data storage versus momentary data rate.

Bottom line, if you want or need charts / dashboards, data storage, and 50 values per second, IO is probably not a good choice :(

I'm curious to hear what you come up with and what kind of project you're working on, if you're willing to share.


- Adam

User avatar
btreichel
 
Posts: 79
Joined: Tue Jul 25, 2017 3:00 pm

Re: Upload an array or a vector to adafruit io (Arduino)

Post by btreichel »

Adam,

Thats not the way that it has been working. I've been using groups to load 5 feeds just under the data rate. I'd tell you to look at my feed data, but I just purged it all trying to get rid of the two ghost feeds that are being counted against me. P.S. people in Discord have seen a public dashboard of this function.

In general, I presume thats the way that you throttle the broker, not a MQTT limit? I presume that if I set up my own broker I could set the limits where I want. P.S. not a complaint against the limits since I just using this for learning and experience.

User avatar
billos
 
Posts: 20
Joined: Thu Nov 16, 2017 9:16 am

Re: Upload an array or a vector to adafruit io (Arduino)

Post by billos »

Hey Adam,

I am working on my Dipl0ma Thesis, and I'm trying to make a fall detector. So I want to upload data from an accelerometer at about 50Hz. Right now I am only concern about collecting data so I don't mind if I have 50 publishes per second (That I know I can't in adafruitIO) or if I have 1 publish with 100 values (2*50 --> 50Hz) every 2 seconds. Do you know how can I create a variable of 100 values in arduino and in the same time be able to upload it to adafrutIO ?
Because I tried with string but after putting over 20 values something went wrong and my Huzzah32 crashed. I don't know what is the problem or if my memory is full but I supposedly have about 520Kb of SRAM.

Thanks in advance!!!

User avatar
abachman
 
Posts: 352
Joined: Mon Feb 01, 2010 12:48 pm

Re: Upload an array or a vector to adafruit io (Arduino)

Post by abachman »

btreichel wrote:Adam,

Thats not the way that it has been working. I've been using groups to load 5 feeds just under the data rate. I'd tell you to look at my feed data, but I just purged it all trying to get rid of the two ghost feeds that are being counted against me. P.S. people in Discord have seen a public dashboard of this function.
Ah ha! You are correct.

Our HTTP API is correctly counting group and bulk (multiple feed data points in one HTTP request) publish events as separate points against the data rate, but the MQTT interface is not, currently. I'm deploying a fix for that this morning and will probably write up a blog post about it too. We spend a lot of time trying to get the throttling system correct, fair, and predictable since that's about the only tool we have for protecting the availability of the system.

And I've taken care of the ghost feeds. They shouldn't be a problem again, but if they are please let us know.
In general, I presume thats the way that you throttle the broker, not a MQTT limit? I presume that if I set up my own broker I could set the limits where I want.
Yes, totally. Our limits are determined by a combination of: 1) the cost of data storage and 2) the cost of servers to support a given data rate across all accounts. 1 and 2 combine to give us a rough estimate of how much each free IO account could cost if it uses the maximum available resources. We picked a reasonable cost and then set our limits. None of the limits we set are inherent to MQTT or any other communication protocol. Our MQTT broker receives every message that's sent, regardless of whether it forwards it to subscribers or stores it permanently, it just counts them up and then drops / ignores extras if they're arriving too quickly.

If you wanted to set up your own MQTT broker you would only be limited by your hardware and network. I'm willing to bet you could push a lot more data through an entirely local MQTT broker, even on something like a Raspberry Pi. If you're comfortable setting up and operating a tool like Home Assistant, that would be a good way to get a lot more flexibility from your devices.

There are also IoT services like Amazon's that will let you scale up to pretty huge data rates (9000 publish events per second, per account and 100 publish events per second per client) but in most cases they charge per message. In Amazon's case, they charge $0.000005 per message ($5 per million) so you'd pay about $5 per month for the same data rate we're providing for free, not including storage.

It's all tradeoffs :D We're trying to favor stable, simple, and clear over expert, technical, and infinitely scalable. Where we get it wrong, we do our best to make corrections.


- Adam

User avatar
btreichel
 
Posts: 79
Joined: Tue Jul 25, 2017 3:00 pm

Re: Upload an array or a vector to adafruit io (Arduino)

Post by btreichel »

Well, I'm happy and sad at the same time. However, I did get to demonstrate, and experiment with functionality that I need for a project that I'm working on. I'm toying with the idea of starting my own company since I committed the corporate sin of getting within a decade of retirement age, so they 'retired me'; of course before I got the benefits. Funding of this project is of course a function of how well my consulting business goes; one funds the other.

My application involves 6 to 7 feeds every second while the coffee roaster is running, with about 15 minutes a batch. This is an IoT add on for community purposes of watching someone else roast, and/or logging and re-using roast profiles. So there would need to be more of a database backend. The main idea behind it is a community of (hobby) roasters that can roast, and discuss roasting, based upon a common data set and common roasting platform.

While I have and old desktop (was great and powerful 8 years ago) and a 100+ mB fiber connection, the idea of hosting on a PI is interesting from the aspect of having a portable MQTT host to go with the demo roaster (if it gets built). Currently its just a 3- printed demo; which would be a very bad idea to incorporate a natural gas burner into.

P.S. Ghost feeds fixed, and you can see how my skills have gotten old and out-of-date as the beard has gone grey.

Locked
Forum rules
If you're posting code, please make sure your code does not include your Adafruit IO Active Key or WiFi network credentials.

Return to “Internet of Things: Adafruit IO and Wippersnapper”