Black Lives Matter - Action and Equality. ... Adafruit is open and shipping.

Need memory strategies
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Need memory strategies

by DarTheMadScientist on Thu Jan 09, 2020 5:14 pm

I'm using Feather M4 Express.

I keep using up memory. It is more than just fragmentation. A few kinds of things seem to never get collected.

Some strategies I have tried:
    gc.collect as things are built and often otherwise
    gc.collect after putting None into a variable containing a large object

Some I haven't tried yet:
    frozen cp
    calling some hidden method before setting the variable to None
    using some collection I don't know about
    rebuilding light stripped-down versions of standard mpy's
    using smaller names
    making comments less verbose
    adding more processors
    looking for a processor with more RAM

I welcome any advice.
(I've been pushing bits around for half a century, but I still consider myself to be a Python noob. I might be missing the obvious.)

Posts: 39
Joined: Mon May 07, 2018 7:47 pm

Re: Need memory strategies

by adafruit_support_mike on Fri Jan 10, 2020 5:15 am

Try creating a block of objects you tend to reuse, and recycle those instead of creating and destroying new ones. It's kind of like a mini version fo malloc().

Posts: 61197
Joined: Thu Feb 11, 2010 2:51 pm

Re: Need memory strategies

by kevinjwalters on Fri Jan 10, 2020 11:56 am

In my first post on How to handle memory error? I built up a list of factors that can affect and improve memory performance but these were more focussed on the heavily constrained world of the M0 processor.

Have you tried printing out free memory a lot with gc.mem_free() to see when/where it starts bloating? Could it just be a leak in your code or library code?

Posts: 665
Joined: Sun Oct 01, 2017 3:15 pm

Re: Need memory strategies

by tannewt on Fri Jan 10, 2020 3:21 pm

Please post your source code. It's much easier to give suggestions when looking at the source. Thanks!

Posts: 1716
Joined: Thu Oct 06, 2016 8:48 pm

Re: Need memory strategies

by DarTheMadScientist on Fri Jan 10, 2020 5:00 pm

Thanks adafruit_support_mike! I have a queue that is made with a simple list. It is used heavily. Perhaps that creates a new list at each dequeue or enqueue, leaving garbage in its wake. A revision of my queue code can benefit from your advice.

Thanks, kevinjwalters! I am using gc.mem_free() a lot. My goal is to get things to where the free memory is above a certain level at the top of my main loop. It would also be nice to see that the largest possible allocation doesn't drop below a certain level. If that continually drops, even by a byte, then eventually my program will crash. Fragmentation is one concern. At one point I got a memory allocation error for 8k when there was 70k available, or so, my memory leaks.

Besides the possible problems with the queue, the problem I was having seemed to be with the new MP3 decoder. I think it does release memory, but there is some quirk where it is hard to get it back. (I don't use with in this case, my design there is inside out, so I destroy it by setting the only variable holding it to None and doing a collect, but I don't know if that is the right way.) I have a workaround that seems to work. However, the good news is that the developer, Jeff Epler (jepler), has added a way to reuse the same decoder by changing the file attribute.

I don't think I have a memory leak, well, not in my code. I don't accumulate things or leave large things in variables when unneeded. All data are DiAGs with no loops. I hope. I will watch for this. At first I suspected the MP3 decoder leaked, but now I can repeatedly destroy and create without a measured shrinkage of free memory (but I'm not sure about fragmentation). It seems that in some conditions, gc.collect() does not recover all of what the MP3 decoder creates. I suspect this is a gc limitation, not an MP3 decoder bug. I have not seen any evidence of the underlying system leaking, or in any library code, but memory jumps around a lot.

I will look at your post, soon.

Thank you, tannewt2. You are quite right, but, alas, it is several hundred lines and depends on several modules, each over a hundred lines. I was prepared to create a simpler version or supply a snippet, but I stumbled onto a workaround.

My workaround: I aggressively gc in my tightest main module loops. Collection seems to be very fast.

(Excessive Detail: Each module runs as a state machine and has a method named spin that allows it to change state based on I/O or timing. I use time.monotonic_ns() a lot! It can even call back to main as needed. In main, I have a function called spin that calls all of the spin methods of the modules, but it can never be called in a callback. It is called in loops in main, and especially in wait(), my replacement for time.sleep(). So, I added gc to the spin list used by spin. The name spin comes from magicians spinning plates on wands; I'm sure there is a better name. This is my approach to pretending I have events and interrupts in CircuitPython.)

I'm not sure why this works any better than a gc at the top of my highest loop plus a gc in audio changes. Before the workaround, I would gc before removing an MP3 decoder from its variable and after, but the memory used (as seen from the increase with gc-build-gc) would not be freed. Now, with aggressive gc of all the other things I leave around, it is freed. I suspect it is related to the state of memory when it is created. I will switch to reusing the decoder using the new file attribute.

I need to learn more about how CircuitPython GC works.

I realize now that I should have been watching memory stability as my project grew.

Posts: 39
Joined: Mon May 07, 2018 7:47 pm

Re: Need memory strategies

by kevinjwalters on Sat Jul 18, 2020 11:44 am

I've suffered from a lot of memory issues recently. I think it's partly a reflection of having devices that let you write larger programs but could also be partly due to fragmentation, particularly with continuous processing of data and storing it from BLE Advertisement packets.

Posts: 665
Joined: Sun Oct 01, 2017 3:15 pm

Please be positive and constructive with your questions and comments.