PyPortal adafruit_requests.py issue with CircuitPython 7.2.x

[jeremyamoore]

Have some code running on my PyPortal that makes requests to local web devices for data. Works fine with CircuitPython 7.1.1 and earlier. Trying with any of the newer 7.2.x gets the below error. I have updated to the latest libraries (as of 20220318) and also updated to the latest WiFi firmware (1.7.4\x00). It looks like if the content of the response is more than around 26 bytes, it get the error. Less than that it works.

PyPortal code that throws the error - this URL is to a test node.js HTTP server below, but fails on other URLs also:

Code: Select all

responseTest = requests.get( url="http://192.168.1.208:8080/27")
print(responseTest.text)

Error:

Code: Select all

Traceback (most recent call last):
  File "code.py", line 1, in <module>
  File "code_meteobridge.py", line 69, in <module>
  File "adafruit_requests.py", line 446, in text
  File "adafruit_requests.py", line 426, in content
  File "adafruit_requests.py", line 489, in iter_content
  File "adafruit_requests.py", line 352, in _readinto
  File "adafruit_requests.py", line 241, in _recv_into
  File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 175, in recv_into
ValueError: Can only read number of bytes between 0 and length of supplied buffer

node.js HTTP server to test with:

Code: Select all

const http = require('http');
const serverPort = 8080
const requestListener = function (req, res) {
	UrlPath = parseInt(req.url.split('/')[1])
	
console.log(UrlPath)
outstring = ''
if (typeof UrlPath == 'number' & UrlPath < 100000 & UrlPath > 0) {
	while (UrlPath)
	{
	 outstring += 'x';
	UrlPath--;}
	
	}
	else
	{outstring = 'Bad number'}
  res.writeHead(200);
  res.end(outstring);
}

const server = http.createServer(requestListener);
console.log('Starting web server on port ' + serverPort)
server.listen(serverPort);

[jeremyamoore]

Quick update... looks like I had some older libraries installed for esp32spi. When updating to the newer ones, I get the errors on the older CircuitPython 7.1.1 also.

Did some troubleshooting and it looks like it is getting a negative value to make that error show.

[danhalbert]

HI, sorry no one replied earlier. Are you still having this problem?

[jeremyamoore]

Yes, still have this issue. Just tested with 7.3.0-beta.2 and 20220507 libraries. Looks like in adafruit_esp32spi_socket recv_into, it gets to the last set of data in the buffer and passes in nbytes that is smaller than the buffer size (which should be OK). When it does the avail = self.available() the value of avail is more than nbytes. So the returned value is larger than nbytes. In adafruit_requests _readinto, self._remaining -= read becomes a negative number. The next call to socket recv_into gets called with a negative nbytes value and the message. If I rename the socket method recv_into to something else, it makes self._backwards_compatible in requests to be true and does the simplified read. So not sure what the root cause is but seems to be either the logic in socket recv_into or deeper into _the_interface.socket_available for the PyPortal.

[danhalbert]

Your observations about the negative number of bytes is very interesting, and may be part of the the key to this and other problems. I will pass this on. Thanks!

[danhalbert]

I reproduced your error using your node.js server, and when it does work it seems very slow, and sometimes seems to hang. Interestingly, using a similar Flask server did not cause the problem:

Code: Select all

from flask import Flask

app = Flask(__name__)

@app.route('/<length>')
def top(length):
    try:
        return 'x' * int(length)
    except ValueError:
        return "not an integer"

app.run(host="0.0.0.0", port=8000)

There is something different about the node.js server; it may be chunking, but I'm not sure. Using curl to talk to your node.js server works fine.

Tracking this problem in https://github.com/adafruit/Adafruit_Ci ... issues/109

[danhalbert]

I believe I have a fix for this here: https://github.com/adafruit/Adafruit_Ci ... I/pull/166. @jeremyamoore if you have time to test I would be grateful. I used your example node webserver, more or less.

[jeremyamoore]

Sorry for the delay - vacation/Covid/work took most of my time lately. I pulled the version from your commit and it does seem to work! Note that what I was really doing is pulling data from my weather station running meteobridge on a RaspberryPI via its /cgi-bin/livedata.cgi URL. The node.js was a way to test/troubleshoot as it was getting the same error. I am getting the weather data now without error and able to parse out what I need. Thanks!

I tried with the adafruit-circuitpython-bundle-7.x-mpy-20220614 libraries but got the same error. I am guessing the update has not been merged in yet?

Thanks again for working on this...

Jeremy A. Moore

[danhalbert]

The update has been merged, about 15 days ago, as version 4.2.2, but I found a second bug in the same code and release 4.2.3 about 9 days ago. However, if that is not working for you, I am interested in that. You could open an issue about how to reproduce the problem in the repo: https://github.com/adafruit/Adafruit_Ci ... _ESP32SPI/. Version 4.2.4 was released more recently, but is an unrelated change that is just a minor documentation fix.

Make sure you are using the correct version. CircuitPython will look first in the top level of CIRCUITPY for files, and then in lib/ . And it will choose .py files over .mpy if both are present. So make sure there is only one copy.

[jeremyamoore]

I think my issue is that the libraries are in the PyPortal's "frozen modules" list and shows up with help("modules") from the REPL so it is loading it from there rather than pulling the updated one from the lib folder. Is there a way to override those or when do updates get compiled into the CircuitPython firmware?

Like I said, it placed the code into a different file name which I imported instead and it works. Plus when it does not work with the frozen modules, it shows the old error "Can only read number of bytes between 0 and length of supplied buffer" rather than the one in the updated code.

[danhalbert]

Ah, I see. The search order for ligbraries is determined by sys.path:

Code: Select all

>>> sys.path
['', '/', '.frozen', '/lib']

So frozen libraries will be used before libraries in lib. But you can put replacement libraries in `/` (the top level of CIRCUITPY) instead of in `lib/`, and they will be used before the frozen libraries. So that's how to override the frozen libraries. Sorry I did not mention that!

[jeremyamoore]

Perfect, that did it! Moved the .mpy file to root and everything works now. Thanks! Will have to remember that trick.

[jeremyamoore]

Installed the updated 7.3.1 CircuitPython firmware on the PyPortal and removed the mpy library from the root and everything is working good.

[danhalbert]

Thanks! Glad to hear it's working OK.