Voting resources, early voting, and poll worker information - VOTE. ... Adafruit is open and shipping.
0

Matrix Portal MemoryError
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Matrix Portal MemoryError

by Agies on Wed Oct 14, 2020 5:40 pm

I tried to modify the Weather Display Matrix project code to use the OpenWeatherMap One Call API so I can get some forecast data to go along with the current conditions. The new API returns a fair amount of data but not so much that it should overwhelm the Matrix Portal. That being said I've found that I get about three refreshes of data before I get "MemoryError: memory allocation failed".

I went back and ran the original code for about an hour and found that the memory appears to be leaking there as well albeit slowly:
Free mem: 41904
Free mem: 36416
Free mem: 36368
Free mem: 36368
Free mem: 36368
Free mem: 36144


I'm assuming that there's some sort of cleanup that is not being done with the JSON data. Is there anything I can do to rein this in?

Agies
 
Posts: 13
Joined: Sat May 09, 2015 9:21 am

Re: Matrix Portal MemoryError

by tannewt on Wed Oct 14, 2020 5:49 pm

Sorry, for the trouble. Memory is tight on these devices.

Are you calling `gc.collect` before printing the mem_free stat? What is the backtrace of the MemoryAllocation error?

tannewt
 
Posts: 1837
Joined: Thu Oct 06, 2016 8:48 pm

Re: Matrix Portal MemoryError

by Agies on Thu Oct 15, 2020 10:12 am

I'll be honest I'm not sure where the Free mem statement is coming from. I can't find a call for it but based on the output I have it looks like it's coming from network.fetch_data as this comes before the API response:
Free mem: 25440
Retrieving data...Reply is OK!


This is my trace:
Retrieving data...Traceback (most recent call last):
File "code.py", line 101, in <module>
File "code.py", line 95, in <module>
File "adafruit_matrixportal/network.py", line 410, in fetch_data
File "adafruit_matrixportal/network.py", line 393, in fetch
File "adafruit_requests.py", line 291, in get
File "adafruit_requests.py", line 251, in request
File "adafruit_requests.py", line 235, in request
File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 99, in readline
File "adafruit_esp32spi/adafruit_esp32spi.py", line 736, in socket_read
File "adafruit_esp32spi/adafruit_esp32spi.py", line 323, in _send_command_get_response
File "adafruit_esp32spi/adafruit_esp32spi.py", line 306, in _wait_response_cmd
File "adafruit_esp32spi/adafruit_esp32spi.py", line 303, in _wait_response_cmd
MemoryError: memory allocation failed, allocating 4000 bytes


Line 95 is "value = network.fetch_data(DATA_SOURCE, json_path=(DATA_LOCATION,))" in this snippet:
Code: Select all | TOGGLE FULL SIZE
while True:
    # only query the online time once per hour (and on first run)
    if (not localtime_refresh) or (time.monotonic() - localtime_refresh) > 3600:
        try:
            print("Getting time from internet!")
            network.get_local_time()
            localtime_refresh = time.monotonic()
        except RuntimeError as e:
            print("Some error occured, retrying! -", e)
            continue

    # only query the weather every 10 minutes (and on first run)
    if (not weather_refresh) or (time.monotonic() - weather_refresh) > 600:
        try:
            value = network.fetch_data(DATA_SOURCE, json_path=(DATA_LOCATION,))
            print("Response is", value)
            gfx.display_weather(value)
            weather_refresh = time.monotonic()
        except RuntimeError as e:
            print("Some error occured, retrying! -", e)
            continue


I tried adding a gc.collect() before the network.fetch_data to no avail but maybe that was not enough? My coding skills are pretty rusty.

Agies
 
Posts: 13
Joined: Sat May 09, 2015 9:21 am

Re: Matrix Portal MemoryError

by tannewt on Thu Oct 15, 2020 1:51 pm

Thank you for the additional info! Requesting 4000 bytes at once can be pretty troublesome because memory tends to be fragmented.

Could you try updating all of the libraries? The latest 1.7.x requests has some bigger changes that try to reduce the size of the largest memory allocation. Thanks!

tannewt
 
Posts: 1837
Joined: Thu Oct 06, 2016 8:48 pm

Re: Matrix Portal MemoryError

by Agies on Thu Oct 15, 2020 3:06 pm

There's still something wrong with requests in 1.7.x

I get errors
Retrieving data...Socket missing recv_into. Using more memory to be compatible

and
Retrieving data...Some error occured, retrying! - Expected 01 but got 00


It does eventually work but it takes a long time.

Agies
 
Posts: 13
Joined: Sat May 09, 2015 9:21 am

Re: Matrix Portal MemoryError

by tannewt on Thu Oct 15, 2020 5:50 pm

Ya, that print is for something I need to do (add recv_into). It will slow it down but it should work.

I'm not sure what the error is from but I'm glad it does work eventually.

The 1.7.0 version makes the trade off of being slower in order to reduce the size of the largest allocation. There is definitely room to improve it.

tannewt
 
Posts: 1837
Joined: Thu Oct 06, 2016 8:48 pm

Re: Matrix Portal MemoryError

by Agies on Fri Oct 16, 2020 10:15 am

Looks like I'm experiencing similar issues to this thread https://forums.adafruit.com/viewtopic.php?f=60&t=170103

We're both getting
Retrieving data...Some error occured, retrying! - Failed to send 3 bytes (sent 0)

and
Retrieving data...Some error occured, retrying! - Expected 01 but got 00

The "Failed to send" error appears to only happen after the loop has gone through once. Eg I can request the time and it will come back without any errors the first time but if I have already gotten a value and try it again I get the error.

As far as I can tell the "Expected 01 but got 00" error only happens on startup, but it happens a lot:
Code: Select all | TOGGLE FULL SIZE
gfx loaded
Getting time from internet!
Connecting to AP [b]REDACTED[/b]
Getting time for timezone America/New_York
Socket missing recv_into. Using more memory to be compatible
Time request:  https://io.adafruit.com/api/v2/Agies/integrations/time/strftime?x-aio-key=[b]REDACTED[/b]
Time reply:  2020-10-16 08:50:28.423 290 5 -0400 EDT
Free mem:  31168
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Expected 01 but got 00
Free mem:  33424
Retrieving data...Some error occured, retrying! - Error response to command
Free mem:  33472
Connecting to AP [b]REDACTED[/b]
Retrieving data...Socket missing recv_into. Using more memory to be compatible
Reply is OK!

I'm not sure what's up with re-connecting to wifi there either.

Agies
 
Posts: 13
Joined: Sat May 09, 2015 9:21 am

Please be positive and constructive with your questions and comments.