Black Lives Matter - Action and Equality.
0

Row Limit Python data retrirval
Moderators: adafruit_support_bill, adafruit

Forum rules
If you're posting code, please make sure your code does not include your Adafruit IO Active Key or WiFi network credentials.
Please be positive and constructive with your questions and comments.

Row Limit Python data retrirval

by bayesp on Mon Apr 20, 2020 1:50 pm

HI

Is there a row limit when fetching data using the Adafruit io python library? If I pull back a feed to an array and then from th array to a dataframe, I only get 1000 rows?

Thanks
bayesp
 
Posts: 6
Joined: Tue Jan 15, 2013 8:22 am

Re: Row Limit Python data retrirval

by brubell on Tue Apr 21, 2020 9:40 am

Could you please post your python code file along with an example of the output? Thanks

brubell
 
Posts: 891
Joined: Fri Jul 17, 2015 10:33 pm

Re: Row Limit Python data retrirval

by bayesp on Tue Apr 21, 2020 11:50 am

Hi here is the code with my credentials redacted:
Code: Select all | TOGGLE FULL SIZE
!pip install adafruit-io

from Adafruit_IO import Client

aio = Client('xxxxxxxx','xxxxxxxx')

data = aio.data('enviro')
import pandas as pd

df= pd.DataFrame(data)

import altair as alt

df.sort_values(by=['created_epoch'], inplace=True, ascending=False)
df['created_at'] = df['created_at'].astype('datetime64[ns]')

source = df

alt.Chart(source).mark_line().encode(
    x='created_at:T',
    y='value:Q'
)

df.head()


df.head
created_epoch created_at updated_at value completed_at feed_id expiration position id lat lon ele
0 1587483858 2020-04-21 15:44:18 None 21.84 None 1356732 2020-05-21T15:44:18Z None 0EE0K1TPK75WG22VC91PCY8MS3 None None None
1 1587483847 2020-04-21 15:44:07 None 21.84 None 1356732 2020-05-21T15:44:07Z None 0EE0K1QAEBPZT8Z5CMP53ST4YD None None None
2 1587483836 2020-04-21 15:43:56 None 21.84 None 1356732 2020-05-21T15:43:56Z None 0EE0K1KZ1PW8TQKKEKBQBX8BDD None None None
3 1587483825 2020-04-21 15:43:45 None 21.84 None 1356732 2020-05-21T15:43:45Z None 0EE0K1GK62VQ788G03R5NVGRQ1 None None None
4 1587483814 2020-04-21 15:43:34 None 21.84 None 1356732 2020-05-21T15:43:34Z None 0EE0K1D7V1EEG7KBQ3MEY9F8B6 None None None

df.describe

created_epoch feed_id
count 1.000000e+03 1000.0
mean 1.587478e+09 1356732.0
std 3.183911e+03 0.0
min 1.587473e+09 1356732.0
25% 1.587476e+09 1356732.0
50% 1.587478e+09 1356732.0
75% 1.587481e+09 1356732.0
max 1.587484e+09 1356732.0
bayesp
 
Posts: 6
Joined: Tue Jan 15, 2013 8:22 am

Re: Row Limit Python data retrirval

by brubell on Wed Apr 29, 2020 12:33 pm

I'm not sure why you're hitting a row limit. You could try downloading the feed as a CSV or JSON, and parse through it:

downloaddata.png
downloaddata.png (16.66 KiB) Viewed 20 times

brubell
 
Posts: 891
Joined: Fri Jul 17, 2015 10:33 pm

Please be positive and constructive with your questions and comments.