0

Pagination link parsing
Moderators: adafruit_support_bill, adafruit

Forum rules
If you're posting code, please make sure your code does not include your Adafruit IO Active Key or WiFi network credentials.
Please be positive and constructive with your questions and comments.

Pagination link parsing

by LC_ on Sat Feb 06, 2021 2:28 pm

Hello all,

I have a question about how pagination works in the API, and I in general. I'm a newb to web APIs so I may be overthinking or overlooking something, so bear with me.

When getting feed data, the response includes a link header that currently looks like this:

<https://the/original/url>;rel="next", <https://the/next/url>

When parsed using the links method of requests.Response (I'm working in Python) you get this:
Code: Select all | TOGGLE FULL SIZE
{'next': {'url': 'https://the/original/url', 'rel': 'next'},
 'https://the/next/url': {'url': 'https://the/next/url'}}
which doesn't seem right. You can access the original url by
Code: Select all | TOGGLE FULL SIZE
response.links['next']['url']
but that's not the one you want, and the other part of the dictionary isn't really usable since the url you want is both key and value.

The documentation links to this example, which uses a regular expression to parse the url after rel="next"; but I don't think this regular expression is necessary.

If it instead looked like this:

<https://the/original/url>;rel="last", <https://the/next/url>;rel="next"
(note the rel="next" parameter is associated with the appropriate link)

then
Code: Select all | TOGGLE FULL SIZE
response.links['next']['url']
would return the url of the next page of data, which is what we need.

So I guess my question is, am I understanding all of this right?

The question may be academic, I suspect this API has been in production long enough that changing it would break more than it would fix. But as I said I'm a newb trying to learn and just taking the time to write up this question has taught me a lot. I just want to make sure I'm not completely missing something, and maybe it'll help the next newb struggling with the same issue.

Thanks!

LC_
 
Posts: 16
Joined: Sun Nov 01, 2020 3:16 pm

Re: Pagination link parsing

by jwcooper on Mon Feb 08, 2021 4:01 pm

Thanks for the very detailed forum post. It appears you've uncovered a bug in our Links implementation. I'd also prefer it to how you've defined it, and tested it in the python library as well.

As you've already made mention, seeing as how it's already in production this one could be tricky to fix. We may go ahead and do it anyways, as I see the value in updating it, as it's an existing bug.

I'll file an issue, and see if we can get it fixed in the near future.

Thanks!

jwcooper
 
Posts: 783
Joined: Tue May 01, 2012 9:08 pm

Re: Pagination link parsing

by LC_ on Mon Feb 08, 2021 4:13 pm

Excellent! I'm hoping to work on this issue in the Adafruit_IO_Python package and having this fixed would make things a bit cleaner.

What would be the best way for me to track the status? Would the issue you plan to submit be on github?

Thanks!

LC_
 
Posts: 16
Joined: Sun Nov 01, 2020 3:16 pm

Re: Pagination link parsing

by jwcooper on Mon Feb 08, 2021 4:29 pm

This would be a major change to our API, so when we're ready to roll it out, it will be quite obvious as we'll announce it in multiple places. As part of fixing this, we'll make sure we fix any changes that are made to the python api and our other client libraries. I've linked your issue to ensure we don't break existing functionality.

You could go ahead and fix the issue with our current implementation of the Link header (if you want to!), and we'll track it to make sure we fix anything we break in the future.

jwcooper
 
Posts: 783
Joined: Tue May 01, 2012 9:08 pm

Re: Pagination link parsing

by LC_ on Mon Feb 08, 2021 4:32 pm

Understood, thanks again!

LC_
 
Posts: 16
Joined: Sun Nov 01, 2020 3:16 pm

Re: Pagination link parsing

by LC_ on Fri Feb 12, 2021 10:59 am

As I work on this I'm coming across other aspects of the API that I don't quite understand, specifically with regard to the Get Feed Data request and pagination. They may be issues or they may just be things I don't understand. For example, when I use the request:
Code: Select all | TOGGLE FULL SIZE
/api/v2/{username}/feeds/{feed_key}/data
and include the the start_time query parameter, I'd expect to get the most recent limit records. Then, use the pagination link to get previous records up to the start_time.

Instead, the first response includes limit records with the oldest being start_time up to limit records after start time. I'm not sure what the pagination link should be after this because typically it would return the next limit older records, but now the records we need are newer than the last received. But if you use the pagination link, it actually returns the same records, minus the newest one, from the last response. If you use the start_time parameter, there doesn't seem to be a way, using pagination, to get anything newer than 1000 records after start_time.

To give an example
Code: Select all | TOGGLE FULL SIZE
https://io.adafruit.com/api/v2/LC_/feeds/temperature/data?limit=4&start_time=2021-02-06T02:00:00Z
returns 2 records from my temperature feed, created at 2021-02-06T02:00:15Z and 2021-02-06T02:00:05Z. Using the following pagination link
Code: Select all | TOGGLE FULL SIZE
https://io.adafruit.com/api/v2/LC_/feeds/temperature/data?before={a_long_string}&end_time=2021-02-12+14%3A49%3A50+UTC&limit=2&start_time=2021-02-06+02%3A00%3A00+UTC&limit=2&start_time=2021-02-06T02%3A00%3A00Z
I get one record in response, the same one from 2021-02-06T02:00:05Z that I already received. It just scales up if you increase the limit, but you always get the same thing as before, minus one.

Is this another bug or am I misunderstanding the intent of the start_time parameter?

I think I may end up having a few other questions like this. Is this the best forum to ask this type of question about the details of the API?

Thanks again!

LC_
 
Posts: 16
Joined: Sun Nov 01, 2020 3:16 pm

Re: Pagination link parsing

by LC_ on Thu Apr 15, 2021 10:09 pm

Bump, see post above. I haven't had much time to work on this lately but I'd like to make more progress. Anyone have any thoughts on the issues I raised in the last post?

Also, any word on a fix for the malformed pagination url?

Thanks!

LC_
 
Posts: 16
Joined: Sun Nov 01, 2020 3:16 pm

Please be positive and constructive with your questions and comments.