Black Lives Matter - Action and Equality.
0

MemoryError - storing values from clue.magnetic in list
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

MemoryError - storing values from clue.magnetic in list

by kevinjwalters on Tue May 12, 2020 6:55 am

This happens on 5.1.0 and 5.3.0 in CircuitPython. I wanted to quickly test the read rate and was surprised by some MemoryError. This is on an nRF52840-based CLUE.

Code: Select all | TOGGLE FULL SIZE
Adafruit CircuitPython 5.3.0 on 2020-04-29; Adafruit CLUE nRF52840 Express with nRF52840
>>>
>>>
>>>
>>> from adafruit_clue import clue
>>> import time, gc
>>> gc.collect() ; gc.mem_free()
89904
>>> t1 = time.monotonic_ns() ; a = [clue.magnetic for x in range(500)] ; t2=time.monotonic_ns()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
MemoryError: memory allocation failed, allocating 2048 bytes
>>> gc.collect() ; gc.mem_free()
89616
>>> t1 = time.monotonic_ns() ; a = [clue.magnetic for x in range(500)] ; t2=time.monotonic_ns()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
MemoryError: memory allocation failed, allocating 2048 bytes
>>> t1 = time.monotonic_ns() ; a = [clue.magnetic for x in range(500)] ; t2=time.monotonic_ns()
>>> print((t2-t1)/1e9)
MemoryError: memory allocation failed, allocating 384 bytes


So that's strange because there's almost 90k free and it appears to be struggling to store 500 3-tuples of floats. To add to that, it finally does manage it and then by some coincidence there's not enough memory to even run a simple print statement with some basic maths in it.

Dropping the list size to 250 and this improves things but reveals that only ~37 bytes is needed per entry.

Code: Select all | TOGGLE FULL SIZE
Adafruit CircuitPython 5.3.0 on 2020-04-29; Adafruit CLUE nRF52840 Express with nRF52840
>>> from adafruit_clue import clue
>>> import time, gc
>>> t1 = time.monotonic_ns() ; a = [clue.magnetic for x in range(250)] ; t2=time.monotonic_ns()
>>> gc.collect() ; gc.mem_free()
80592
>>> del a
>>> t1 = time.monotonic_ns() ; a = [clue.magnetic for x in range(250)] ; t2=time.monotonic_ns()
>>> gc.collect() ; gc.mem_free()
80416
>>> del a
>>> t1 = time.monotonic_ns() ; a = [clue.magnetic for x in range(250)] ; t2=time.monotonic_ns()
>>> gc.collect() ; gc.mem_free()
80320
>>> del a
>>> t1 = time.monotonic_ns() ; a = [clue.magnetic for x in range(250)] ; t2=time.monotonic_ns()
>>> gc.collect() ; gc.mem_free()
80320
>>> del a
>>> t1 = time.monotonic_ns() ; a = [clue.magnetic for x in range(250)] ; t2=time.monotonic_ns()
>>> gc.collect() ; gc.mem_free()
80320
>>> del a
>>> t1 = time.monotonic_ns() ; a = [clue.magnetic for x in range(250)] ; t2=time.monotonic_ns()
>>> gc.collect() ; gc.mem_free()
80320
>>> print((89616-80320)/250)
37.184


Playing around a bit with data that's equivalent shows that two lists of 1000 easily fit into memory.

Code: Select all | TOGGLE FULL SIZE
Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 5.3.0 on 2020-04-29; Adafruit CLUE nRF52840 Express with nRF52840
>>>
>>> from adafruit_clue import clue
>>> import time, gc
>>> t1 = time.monotonic_ns() ; a = [(1.0, 2.0, 3.0) for x in range(1000)] ; t2=time.monotonic_ns()
>>> gc.collect() ; gc.mem_free()
53520
>>> t1 = time.monotonic_ns() ; b = [(4.0, 5.0, 6.0) for x in range(1000)] ; t2=time.monotonic_ns()
>>> id(b[0])
536963232
>>> id(b[1])
536963264
>>> gc.collect() ; gc.mem_free()
17216


I thought this was a bug at first but perhaps it's just an issue with the garabage collection that's going on and not going on inside clue.magnetic and during the loop leading to massive fragmentation?

I turned off auto_refresh to rule that out and the display updates does not appear to be a factor here. A for loop with list append does the same thing too so it's not a list-comprehension-ism. A for loop that pre-allocates enough space for it before going into the magnetic code does work.

Code: Select all | TOGGLE FULL SIZE
Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 5.3.0 on 2020-04-29; Adafruit CLUE nRF52840 Express with nRF52840
>>>
>>> from adafruit_clue import clue
>>> import time, gc, random
>>> gc.collect() ; gc.mem_free()
89872
>>> a=[(1.0, 2.0, 3.0) for x in range(1000)]
>>> gc.collect() ; gc.mem_free()
53616
>>> for idx in range(1000):
...     a[idx] = clue.magnetic
...
...
...
>>> gc.collect() ; gc.mem_free()
53504


Is the answer here essentially to pre-allocate anything big and/or use array.array or ulab's equivalent?

kevinjwalters
 
Posts: 576
Joined: Sun Oct 01, 2017 3:15 pm

Re: MemoryError - storing values from clue.magnetic in list

by kevinjwalters on Tue May 12, 2020 7:53 am

There might be more to this, this is odd, the MemoryError only pops out when I try and time it and I can't see a good reason for that being memory heavy?

Code: Select all | TOGGLE FULL SIZE
Adafruit CircuitPython 5.3.0 on 2020-04-29; Adafruit CLUE nRF52840 Express with nRF52840
>>> from adafruit_clue import clue
>>> import time, gc, random
>>> gc.collect() ; gc.mem_free()
89872
>>> a=[(1.0, 2.0, 3.0) for x in range(500)
... ]
>>>
>>> def read(n):
...     for idx in range(n):
...         a[idx] = clue.magnetic
...
...
...
>>> gc.collect() ; gc.mem_free()
71472
>>> read(500)
>>> gc.collect() ; gc.mem_free()
71504
>>> read(500)
>>> gc.collect() ; gc.mem_free()
71488
>>> read(500)
>>> gc.collect() ; gc.mem_free()
71504
>>> read(500)
>>> gc.collect() ; gc.mem_free()
71504
>>> t1 = time.monotonic_ns() ; read(500) ; t2 = time.monotonic_ns()
>>> print((t1-t2)/1e9)
MemoryError: memory allocation failed, allocating 512 bytes

kevinjwalters
 
Posts: 576
Joined: Sun Oct 01, 2017 3:15 pm

Re: MemoryError - storing values from clue.magnetic in list

by kevinjwalters on Tue May 12, 2020 1:27 pm


kevinjwalters
 
Posts: 576
Joined: Sun Oct 01, 2017 3:15 pm

Please be positive and constructive with your questions and comments.