0

data export
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

data export

by AchimM on Mon Jul 01, 2019 11:11 am

Hi,

I have the problem that the data in the .csv file I download is not the same as I have in the list of the feed.
This results in some strange effects and I do not know how I can get out my real data now.

Anyone an idea why this is Happening?

I attached a picture with the original ploted data on a feed and the data ploted on Excel.
It is not a plotting error from Excel. The data written in the .csv is really not matching the data in the list of the feed. I am using the plot to better illustrate the problem….
One more point to the picture: The left plot is only the data of the right half of the right plot due to the fact that on io adafruit there is no possibility to scale in between 2 and 7 days...

Best,
Achim
Attachments
plot of downloaded data.png
plot of downloaded data.png (369.43 KiB) Viewed 382 times

AchimM
 
Posts: 5
Joined: Mon Jul 01, 2019 10:40 am

Re: data export

by abachman on Mon Jul 01, 2019 2:44 pm

hi AchimM,


Charts in IO aggregate data points by averaging over time slices (1 minute, 5 minute, 10m, etc. based on chart length in hours) to avoid overwhelming the data API and our browser-based charting libraries. You can switch to "raw" chart mode, but you'll still be limited to 640 data points in the chart. If you need absolutely accurate charting you'll have to export the data first.

I have the problem that the data in the .csv file I download is not the same as I have in the list of the feed.


Do you mean the exported .csv has different values for the same timestamps than are shown on the feed page? Or do you mean in the charts?


- Adam B.

abachman
 
Posts: 327
Joined: Mon Feb 01, 2010 12:48 pm

Re: data export

by AchimM on Tue Jul 02, 2019 2:22 am

Dear Adam,

first of all thank you for the fast reply and sorry for the late replay from my side. But I am in Germany and we will have to handle a certain time shift on our Forum communication ;)

abachman wrote:Do you mean the exported .csv has different values for the same timestamps than are shown on the feed page? Or do you mean in the charts?


Yes this is exactely what I mean.
I attached a screen shot where you can see the difference in the exported data on .csv (left side) and on the data displayed at the feed (right side).
Do not worry about the difference of a factor 100. This is due to the fact that we use "," as decimal and excel did somehow not recognize that the "." is the decimal. I put in the Settings of excel but by importing it still did it wrong. But this is not an issue. I can divide the value by 100 and than it is fine.
The Problem is more that even after dividing by 100 my sensor resistance value is not the same as in the feed and also the date is not the same. Could this be due to the fact that my PC is in a different time zone?

I even opened the file with scilab and I have the same data like in the Excel sheet….. Just to be sure before I am posting you that the exported data is not fitting to the data on the feed

Thank you for your help!

Best,
Achim
Attachments
data on csv and feed.png
data on csv and feed.png (341.82 KiB) Viewed 349 times

AchimM
 
Posts: 5
Joined: Mon Jul 01, 2019 10:40 am

Re: data export

by abachman on Tue Jul 02, 2019 12:21 pm

first of all thank you for the fast reply and sorry for the late replay from my side. But I am in Germany and we will have to handle a certain time shift on our Forum communication ;)


ha, no worries, I'm in no rush.

Could this be due to the fact that my PC is in a different time zone?


Time is definitely a factor if you're comparing .csv to browser. Your browser will display time in your local timezone, but the .csv will always show the time in UTC. So, I'm east coast USA (EDT)--minus 4 hours UTC right now--I'd have to translate the .csv times by subtracting four hours.

Timezones range somewhere between annoying and dangerous for programmers, so we tend to do everything behind the scenes in UTC and then convert to local when displaying. The .csv export is considered "behind the scenes" since it's just a dump of the database. IO sends all dates in UTC, but your browser renders times in your own zone. So, if you're CEST, bump all the times in the csv by +2 hours.

If that still doesn't help you line up the data, I can take a look at the feed to see if something stranger is happening.


- adam b.

abachman
 
Posts: 327
Joined: Mon Feb 01, 2010 12:48 pm

Re: data export

by AchimM on Wed Jul 03, 2019 8:59 am

Dear Adam,

I really have to apologize to have wasted your time.
The difference in data is due to the fact that the time is UTC and I did not take care of it....

Thank you very much for your support!

Best,
Achim

AchimM
 
Posts: 5
Joined: Mon Jul 01, 2019 10:40 am

Re: data export

by abachman on Wed Jul 03, 2019 9:15 am

Not a waste at all! This thread has some good observations and research (mostly yours) that will hopefully help people in the future with similar problems.

That's why we (all of Adafruit, really) try to do support on the forums here, where it's public and searchable.

Thanks for checking out IO!


- adam b.

abachman
 
Posts: 327
Joined: Mon Feb 01, 2010 12:48 pm

Re: data export

by AchimM on Mon Jul 08, 2019 3:38 am

Dear Adam,

thank you very much!

I have a second issue/Question?

Is it possible to apply the filter to the Export or to delete data that has been exported?

I could not find on the Website/forum how to do it. When I apply the filter it is not applied to the csv Export function….

Best,
Achim

AchimM
 
Posts: 5
Joined: Mon Jul 01, 2019 10:40 am

Re: data export

by abachman on Mon Jul 08, 2019 12:07 pm

No, the filter is not applied to the export. The bulk export always covers the whole feed. You can use the HTTP get feed data API to list the raw feed data using date parameters to get within a range of dates (start_time and end_time). I have an example script I use for downloading data over on GitHub, here.

We also don't have a bulk delete feature either, unfortunately. Deletes are "write events" as far as IO is concerned, data updates too. So each of those events, in addition to data creation, affects your data rate usage. They're usable, but if you find yourself deleting a significant percentage of your feed's data, it may be better to just make a new feed.

abachman
 
Posts: 327
Joined: Mon Feb 01, 2010 12:48 pm

Re: data export

by AchimM on Tue Jul 09, 2019 1:12 am

Dear Adam,

thank you very much for your fast and constructive Feedback (again ;) ).

Yes it is OK like this. We wrote a matlab script to plot the data where I can just change the limits of the x-axis to select the range.

For Overview reasons it would have been easier to download a certain time span. Maybe later on I will use your source Code if it becomes too much…..

Thank you very much!

Best,
Achim

AchimM
 
Posts: 5
Joined: Mon Jul 01, 2019 10:40 am

Please be positive and constructive with your questions and comments.