MatrixPortal Memory Error

CircuitPython on hardware including Adafruit's boards, and CircuitPython libraries using Blinka on host computers.

Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.
User avatar
MakerMelissa
 
Posts: 304
Joined: Wed Jun 05, 2013 2:10 am

Re: MatrixPortal Memory Error

Post by MakerMelissa »

Ok, I now have a PR in to freeze the PortalBase library and make use of some of the other libraries that were already frozen in here: https://github.com/adafruit/circuitpython/pull/6043. It likely won't be in there until the next release of CircuitPython.

User avatar
Tortilla
 
Posts: 31
Joined: Sun Jan 09, 2022 5:03 am

Re: MatrixPortal Memory Error

Post by Tortilla »

MakerMelissa wrote:Ok, I now have a PR in to freeze the PortalBase library and make use of some of the other libraries that were already frozen in here: https://github.com/adafruit/circuitpython/pull/6043. It likely won't be in there until the next release of CircuitPython.
Thanks! I'm looking forward to trying it out.

User avatar
jliu70
 
Posts: 84
Joined: Wed Oct 21, 2020 9:27 pm

Re: MatrixPortal Memory Error

Post by jliu70 »

So just out of curiosity, would you show the contents of your CIRCUITPY lib folder?
Are all the libraries ".mpy" versions?

Thanks,
Jeff

User avatar
dastels
 
Posts: 10624
Joined: Tue Oct 20, 2015 3:22 pm

Re: MatrixPortal Memory Error

Post by dastels »

jliu70 wrote:So just out of curiosity, would you show the contents of your CIRCUITPY lib folder?
Are all the libraries ".mpy" versions?

Thanks,
Jeff
You really should read the entire thread.

Dave

User avatar
jliu70
 
Posts: 84
Joined: Wed Oct 21, 2020 9:27 pm

Re: MatrixPortal Memory Error

Post by jliu70 »

dastels wrote:
jliu70 wrote:So just out of curiosity, would you show the contents of your CIRCUITPY lib folder?
Are all the libraries ".mpy" versions?

Thanks,
Jeff
You really should read the entire thread.

Dave
I saw the post stating that all the libraries were ".mpy", but from the code snippet, it seemed to only be making a call to one external library. adafruit_datetime and perhaps terminal.

Code: Select all

"""One panel display using 64 X 32 RGB panel to scroll ESPN game information"""
import gc
import time
import json
from adafruit_datetime import datetime, timedelta
import board
from adafruit_matrixportal.matrixportal import MatrixPortal

Memory_free = gc.mem_free()
print("Total memory free right after importing matrixportal library is", Memory_free)
# import terminalio

I was just wondering if that was the case that he was only using the two libraries, of if there were additional libraries being used.

And I wasn't quite understanding if the issue was with the RAM or with the storage on the MatrixPortal.

Thanks,
Jeff

User avatar
MakerMelissa
 
Posts: 304
Joined: Wed Jun 05, 2013 2:10 am

Re: MatrixPortal Memory Error

Post by MakerMelissa »

Tortilla wrote:
MakerMelissa wrote:Ok, I now have a PR in to freeze the PortalBase library and make use of some of the other libraries that were already frozen in here: https://github.com/adafruit/circuitpython/pull/6043. It likely won't be in there until the next release of CircuitPython.
Thanks! I'm looking forward to trying it out.
It's now merged in. You can try out one of the automated builds if you want: https://adafruit-circuit-python.s3.amaz ... portal_m4/. You'll notice a slight rise in size for the ones with the libraries frozen in. You should be able to run it without installing the following libraries:
Adafruit_CircuitPython_PortalBase
Adafruit_CircuitPython_Requests
Adafruit_CircuitPython_ESP32SPI
Adafruit_CircuitPython_NeoPixel

User avatar
Tortilla
 
Posts: 31
Joined: Sun Jan 09, 2022 5:03 am

Re: MatrixPortal Memory Error

Post by Tortilla »

MakerMelissa wrote:
Tortilla wrote:
MakerMelissa wrote:Ok, I now have a PR in to freeze the PortalBase library and make use of some of the other libraries that were already frozen in here: https://github.com/adafruit/circuitpython/pull/6043. It likely won't be in there until the next release of CircuitPython.
Thanks! I'm looking forward to trying it out.
It's now merged in. You can try out one of the automated builds if you want: https://adafruit-circuit-python.s3.amaz ... portal_m4/. You'll notice a slight rise in size for the ones with the libraries frozen in. You should be able to run it without installing the following libraries:
Adafruit_CircuitPython_PortalBase
Adafruit_CircuitPython_Requests
Adafruit_CircuitPython_ESP32SPI
Adafruit_CircuitPython_NeoPixel
I'm definetly going to try it this weekend. Thanks again!

User avatar
jliu70
 
Posts: 84
Joined: Wed Oct 21, 2020 9:27 pm

Re: MatrixPortal Memory Error

Post by jliu70 »

Just out of curiosity, I gave it a try,
Looks like the memory error persists, but I don't know if it's because I'm not loading the "mpy" versions of the libraries.

Here's the REPL output:

Code: Select all


Adafruit CircuitPython 7.2.0-alpha.2-27-g57bbe7922 on 2022-02-18; Adafruit Matrix Portal M4 with samd51j19
>>> 
soft reboot

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Total memory free right before importing matrixportal library is 99216
Total memory free right after importing matrixportal library is 72576
Total memory free right before matrixportal call is 71952
Connecting to AP FiOS-ZG466
Retrieving data...Reply is OK!
Traceback (most recent call last):
  File "code.py", line 193, in <module>
  File "code.py", line 140, in fetch_sports_data
  File "adafruit_portalbase/__init__.py", line 412, in fetch
  File "adafruit_portalbase/network.py", line 560, in fetch_data
  File "adafruit_portalbase/network.py", line 591, in _parse_data
MemoryError: memory allocation failed, allocating 65 bytes (data is likely too large)

Code done running.

Adafruit CircuitPython 7.2.0-alpha.2-27-g57bbe7922 on 2022-02-18; Adafruit Matrix Portal M4 with samd51j19
>>> 
Here's the contents of the CIRCUITPY/lib folder:

Code: Select all

(circup) jliu@JEFFREYs-MBP ~/git/jliu-circuitpython-examples  % ls -l /Volumes/CIRCUITPY/lib
total 104
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_bitmap_font
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_bus_device
-rwxrwxrwx  1 jliu  staff  18705 Feb 18 16:53 adafruit_datetime.mpy
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_display_shapes
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_display_text
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_esp32spi
-rwxrwxrwx  1 jliu  staff    363 Feb 18 17:00 adafruit_fakerequests.mpy
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_imageload
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_io
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_matrixportal
-rwxrwxrwx  1 jliu  staff   6596 Feb 18 17:00 adafruit_miniqr.mpy
-rwxrwxrwx  1 jliu  staff   3010 Feb 18 17:00 adafruit_pixelbuf.mpy
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_portalbase
-rwxrwxrwx  1 jliu  staff   9188 Feb 18 17:00 adafruit_requests.mpy
-rwxrwxrwx  1 jliu  staff   1401 Feb 18 17:01 neopixel.mpy
-rwxrwxrwx  1 jliu  staff   1761 Feb 18 17:01 simpleio.mpy


User avatar
MakerMelissa
 
Posts: 304
Joined: Wed Jun 05, 2013 2:10 am

Re: MatrixPortal Memory Error

Post by MakerMelissa »

jliu70 wrote:Just out of curiosity, I gave it a try,
Looks like the memory error persists, but I don't know if it's because I'm not loading the "mpy" versions of the libraries.

Here's the contents of the CIRCUITPY/lib folder:

Code: Select all

(circup) jliu@JEFFREYs-MBP ~/git/jliu-circuitpython-examples  % ls -l /Volumes/CIRCUITPY/lib
total 104
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_bitmap_font
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_bus_device
-rwxrwxrwx  1 jliu  staff  18705 Feb 18 16:53 adafruit_datetime.mpy
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_display_shapes
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_display_text
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_esp32spi
-rwxrwxrwx  1 jliu  staff    363 Feb 18 17:00 adafruit_fakerequests.mpy
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_imageload
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_io
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_matrixportal
-rwxrwxrwx  1 jliu  staff   6596 Feb 18 17:00 adafruit_miniqr.mpy
-rwxrwxrwx  1 jliu  staff   3010 Feb 18 17:00 adafruit_pixelbuf.mpy
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_portalbase
-rwxrwxrwx  1 jliu  staff   9188 Feb 18 17:00 adafruit_requests.mpy
-rwxrwxrwx  1 jliu  staff   1401 Feb 18 17:01 neopixel.mpy
-rwxrwxrwx  1 jliu  staff   1761 Feb 18 17:01 simpleio.mpy

Are you using the new circuitpython image with the frozen libraries? If so, you can delete the following:
adafruit_portalbase
adafruit_requests.mpy
adafruit_esp32spi
neopixel.mpy
adafruit_bus_device

I believe having those will actually override the frozen libraries and use the files in your lib folder instead, which would result in a memory error.

Melissa.

User avatar
jliu70
 
Posts: 84
Joined: Wed Oct 21, 2020 9:27 pm

Re: MatrixPortal Memory Error

Post by jliu70 »

Yes, I'm using the latest UF2

Code: Select all

Adafruit CircuitPython 7.2.0-alpha.2-27-g57bbe7922 on 2022-02-18; Adafruit Matrix Portal M4 with samd51j19
Followed your directions and removed the libraries from the CIRCUITPY/lib folder.

Code: Select all


(circup) jliu@JEFFREYs-MBP ~  % ls -l /Volumes/CIRCUITPY/lib               
total 76
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_bitmap_font
-rwxrwxrwx  1 jliu  staff  18705 Feb 18 16:53 adafruit_datetime.mpy
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_display_shapes
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_display_text
-rwxrwxrwx  1 jliu  staff    363 Feb 18 17:23 adafruit_fakerequests.mpy
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_imageload
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_io
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:23 adafruit_matrixportal
-rwxrwxrwx  1 jliu  staff   6596 Feb 18 17:00 adafruit_miniqr.mpy
-rwxrwxrwx  1 jliu  staff   3010 Feb 18 17:00 adafruit_pixelbuf.mpy
-rwxrwxrwx  1 jliu  staff   1761 Feb 18 17:01 simpleio.mpy


Still seeing the memory error. REPL output below:

Code: Select all


Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Total memory free right before importing matrixportal library is 99024
Total memory free right after importing matrixportal library is 73136
Total memory free right before matrixportal call is 72512
Connecting to AP FiOS-ZG466
Retrieving data...Reply is OK!
Traceback (most recent call last):
  File "code.py", line 193, in <module>
  File "code.py", line 140, in fetch_sports_data
  File "adafruit_portalbase/__init__.py", line 412, in fetch
  File "adafruit_portalbase/network.py", line 560, in fetch_data
  File "adafruit_portalbase/network.py", line 591, in _parse_data
MemoryError:  (data is likely too large)

Code done running.

User avatar
Tortilla
 
Posts: 31
Joined: Sun Jan 09, 2022 5:03 am

Re: MatrixPortal Memory Error

Post by Tortilla »

jliu70 wrote:Yes, I'm using the latest UF2

Code: Select all

Adafruit CircuitPython 7.2.0-alpha.2-27-g57bbe7922 on 2022-02-18; Adafruit Matrix Portal M4 with samd51j19
Followed your directions and removed the libraries from the CIRCUITPY/lib folder.

Code: Select all


(circup) jliu@JEFFREYs-MBP ~  % ls -l /Volumes/CIRCUITPY/lib               
total 76
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_bitmap_font
-rwxrwxrwx  1 jliu  staff  18705 Feb 18 16:53 adafruit_datetime.mpy
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_display_shapes
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_display_text
-rwxrwxrwx  1 jliu  staff    363 Feb 18 17:23 adafruit_fakerequests.mpy
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_imageload
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:01 adafruit_io
drwxrwxrwx  1 jliu  staff   1024 Feb 18 17:23 adafruit_matrixportal
-rwxrwxrwx  1 jliu  staff   6596 Feb 18 17:00 adafruit_miniqr.mpy
-rwxrwxrwx  1 jliu  staff   3010 Feb 18 17:00 adafruit_pixelbuf.mpy
-rwxrwxrwx  1 jliu  staff   1761 Feb 18 17:01 simpleio.mpy


Still seeing the memory error. REPL output below:

Code: Select all


Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Total memory free right before importing matrixportal library is 99024
Total memory free right after importing matrixportal library is 73136
Total memory free right before matrixportal call is 72512
Connecting to AP FiOS-ZG466
Retrieving data...Reply is OK!
Traceback (most recent call last):
  File "code.py", line 193, in <module>
  File "code.py", line 140, in fetch_sports_data
  File "adafruit_portalbase/__init__.py", line 412, in fetch
  File "adafruit_portalbase/network.py", line 560, in fetch_data
  File "adafruit_portalbase/network.py", line 591, in _parse_data
MemoryError:  (data is likely too large)

Code done running.
I think the memory error that you are seeing now is because the ESPN api is too large for the version of code that I originally posted and which you have been testing. Melissa's frozen library resulted in very significant increases in free memory after the library imports and is doing what it is supposed to do. I have learned that this project has to approached differently to be able to use the ESPN api's with a matrixportal (due to the memory limitation of the board). Since I posted my original code, I have reworked it and used a lot of memory saving tactics such as only importing the necessary fuctions from the modules, streamlining my code, removing the "double" framebuffer memory and other techniques. My original code that I posted was not really useable anyway, but I couldn't even debug it because of the initial memory error which was the subject of this thread. I have the matrixportal displayer up and running, but nevertheless, Melissa's efforts are greatly appreciated since I can still have memory errors when there are many games covered by an ESPN api on a given day. I just wanted to let you know so you might save some time trying to continue to test my rather bad initial code that I have completely reworked.

User avatar
danhalbert
 
Posts: 3353
Joined: Tue Aug 08, 2017 12:37 pm

Re: MatrixPortal Memory Error

Post by danhalbert »

Is there any way to fetch less data at a time with the ESPN API, like a few hours at a time instead of a whole day, etc.?

User avatar
jliu70
 
Posts: 84
Joined: Wed Oct 21, 2020 9:27 pm

Re: MatrixPortal Memory Error

Post by jliu70 »

Just doing a raw dump of the two API calls, it looks like the data set returned are the following sizes:

Code: Select all

 75281 College Baseball
106729 NHL
It does seem to dump back "everything".



For example the NHL json data snippet:

Code: Select all

{"leagues":[{"id":"90","uid":"s:70~l:90","name":"National Hockey League","abbreviation":"NHL","slug":"nhl","season":{"year":2022,"startDate":"2021-07-21T07:00Z","endDate":"2022-07-01T06:59Z","type":{"id":"2","type":2,"name":"Regular Season","abbreviation":"reg"}},"calendarType":"day","calendarIsWhitelist":true,"calendarStartDate":"2021-07-21T07:00Z","calendarEndDate":"2022-07-01T06:59Z","calendar":["2021-09-25T07:00Z","2021-09-26T07:00Z","2021-09-27T07:00Z","2021-09-28T07:00Z","2021-09-29T07:00Z","2021-09-30T07:00Z","2021-10-01T07:00Z","2021-10-02T07:00Z","2021-10-03T07:00Z","2021-10-04T07:00Z","2021-10-05T07:00Z","2021-10-06T07:00Z","2021-10-07T07:00Z","2021-10-08T07:00Z","2021-10-09T07:00Z","2021-10-12T07:00Z","2021-10-13T07:00Z","2021-10-14T07:00Z","2021-10-15T07:00Z","2021-10-16T07:00Z","2021-10-17T07:00Z","2021-10-18T07:00Z","2021-10-19T07:00Z","2021-10-20T07:00Z","2021-10-21T07:00Z","2021-10-22T07:00Z","2021-10-23T07:00Z","2021-10-24T07:00Z","2021-10-25T07:00Z","2021-10-26T07:00Z","2021-10-27T07:00Z","2021-10-28T07:00Z","2021-10-29T07:00Z","2021-10-30T07:00Z","2021-10-31T07:00Z","2021-11-01T07:00Z","2021-11-02T07:00Z","2021-11-03T07:00Z","2021-11-04T07:00Z","2021-11-05T07:00Z","2021-11-06T07:00Z","2021-11-07T07:00Z","2021-11-08T08:00Z","2021-11-09T08:00Z","2021-11-10T08:00Z","2021-11-11T08:00Z","2021-11-12T08:00Z","2021-11-13T08:00Z","2021-11-14T08:00Z","2021-11-15T08:00Z","2021-11-16T08:00Z","2021-11-17T08:00Z","2021-11-18T08:00Z","2021-11-19T08:00Z","2021-11-20T08:00Z","2021-11-21T08:00Z","2021-11-22T08:00Z","2021-11-23T08:00Z","2021-11-24T08:00Z","2021-11-26T08:00Z","2021-11-27T08:00Z","2021-11-28T08:00Z","2021-11-29T08:00Z","2021-11-30T08:00Z","2021-12-01T08:00Z","2021-12-02T08:00Z","2021-12-03T08:00Z","2021-12-04T08:00Z","2021-12-05T08:00Z","2021-12-06T08:00Z","2021-12-07T08:00Z","2021-12-08T08:00Z","2021-12-09T08:00Z","2021-12-10T08:00Z","2021-12-11T08:00Z","2021-12-12T08:00Z","2021-12-13T08:00Z","2021-12-14T08:00Z"
...
...

So I searching online, I found a couple of gists which seem to provide info on the ESPN APIs.

The following API call seems to limit the returned data to the scores for the NHL for the current day:

Code: Select all

https://site.web.api.espn.com/apis/v2/scoreboard/header?sport=hockey&league=nhl
The size of the returned NHL dataset is 44KB which probably will work since it's smaller than the 75KB free memory reported on the MatrixPortal.



Unfortunately the size of the returned college baseball dataset is 108KB which still seems to exceed the free memory.

Code: Select all

https://site.web.api.espn.com/apis/v2/scoreboard/header?sport=baseball&league=college-baseball

User avatar
danhalbert
 
Posts: 3353
Joined: Tue Aug 08, 2017 12:37 pm

Re: MatrixPortal Memory Error

Post by danhalbert »

At least on the hockey one, I can add `&dates=20220201` or `&dates=20220201-20220210`, and get different results.

Locked
Please be positive and constructive with your questions and comments.

Return to “Adafruit CircuitPython”