0

memory leak on alpha 3.0 metro express m4
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

memory leak on alpha 3.0 metro express m4

by cfloryiv on Mon Apr 23, 2018 7:08 pm

I'm getting a memory allocation error when running the sample python program designed for the PM2.5 sensor
Code: Select all | TOGGLE FULL SIZE
Concentration Units (standard)
---------------------------------------
PM 1.0: 3   PM2.5: 3   PM10: 4
Concentration Units (environmental)
---------------------------------------
PM 1.0: 3   PM2.5: 3   PM10: 4
---------------------------------------
Particles > 0.3um / 0.1L air: 657
Particles > 0.5um / 0.1L air: 169
Particles > 1.0um / 0.1L air: 12
Particles > 2.5um / 0.1L air: 2
Particles > 5.0um / 0.1L air: 0
Particles > 10 um / 0.1L air: 0
---------------------------------------
Traceback (most recent call last):
  File "code.py", line 20, in <module>
MemoryError: memory allocation failed, allocating 42648 bytes

Here's the sample program
Code: Select all | TOGGLE FULL SIZE
from digitalio import DigitalInOut, Direction
import board
import busio
import time
import struct

led = DigitalInOut(board.D13)
led.direction = Direction.OUTPUT

# Connect the Sensor's TX pin to the board's RX pin
uart = busio.UART(board.TX, board.RX, baudrate=9600)

buffer = []

while True:
    data = uart.read(32)  # read up to 32 bytes
    data = list(data)
    #print("read: ", data)          # this is a bytearray type

    buffer += data
   
    while buffer and buffer[0] != 0x42:
        buffer.pop(0)
   
    if len(buffer) < 32:
        continue

    if buffer[1] != 0x4d:
        buffer.pop(0)
        continue

    frame_len = struct.unpack(">H", bytes(buffer[2:4]))[0]
    if frame_len != 28:
        continue

    frame = struct.unpack(">HHHHHHHHHHHHHH", bytes(buffer[4:]))

    pm10_standard, pm25_standard, pm100_standard, pm10_env, pm25_env, pm100_env, particles_03um, particles_05um, particles_10um, particles_25um, particles_50um, particles_100um, skip, checksum = frame

    check = sum(buffer[0:30])
   
    if check != checksum:
        continue
    print("Concentration Units (standard)")
    print("---------------------------------------")
    print("PM 1.0: %d\tPM2.5: %d\tPM10: %d" % (pm10_standard, pm25_standard, pm100_standard))
    print("Concentration Units (environmental)")
    print("---------------------------------------")
    print("PM 1.0: %d\tPM2.5: %d\tPM10: %d" % (pm10_env, pm25_env, pm100_env))
    print("---------------------------------------")
    print("Particles > 0.3um / 0.1L air:", particles_03um)
    print("Particles > 0.5um / 0.1L air:", particles_05um)
    print("Particles > 1.0um / 0.1L air:", particles_10um)
    print("Particles > 2.5um / 0.1L air:", particles_25um)
    print("Particles > 5.0um / 0.1L air:", particles_50um)
    print("Particles > 10 um / 0.1L air:", particles_100um)
    print("---------------------------------------")

    buffer = buffer[32:]
    #print("Buffer ", buffer)
    time.sleep(5.0)


cfloryiv
 
Posts: 37
Joined: Tue Dec 27, 2011 10:06 am

Re: memory leak on alpha 3.0 metro express m4

by tannewt2 on Mon Apr 23, 2018 7:25 pm

Where did you get this sample code? Its suspicious that its trying to allocate 42648 bytes.

What board are you running this on?

~Scott

tannewt2
 
Posts: 810
Joined: Thu Oct 06, 2016 8:48 pm

Re: memory leak on alpha 3.0 metro express m4

by adafruit2 on Mon Apr 23, 2018 7:39 pm

cliv can you post a photo of your wiring?

adafruit2
Site Admin
 
Posts: 17140
Joined: Fri Mar 11, 2005 7:36 pm

Re: memory leak on alpha 3.0 metro express m4

by cfloryiv on Mon Apr 23, 2018 7:48 pm

The code I'm using was downloaded from the product page for the sensor (adafruit part# p3868). The only change I made was to use the struct library rather than the ustruct library.

I using the wiring from the product page also. Note that the program runs for several minutes before crashing. This is why I think it's a memory leak.

I dont have a picture but the wiring is:

pm2.5 pin metro pin
vcc 5v
grnd grnd
txd rcv

cfloryiv
 
Posts: 37
Joined: Tue Dec 27, 2011 10:06 am

Re: memory leak on alpha 3.0 metro express m4

by cfloryiv on Mon Apr 23, 2018 7:51 pm

I forgot to add. I'm running the new metro express m4 board

cfloryiv
 
Posts: 37
Joined: Tue Dec 27, 2011 10:06 am

Re: memory leak on alpha 3.0 metro express m4

by cfloryiv on Mon Apr 23, 2018 10:05 pm

It appears a problem with data filling up the buffer. The program began crashing when a 5 second sleep was added to the loop.

Is there a "flush" method to the uart object?

cfloryiv
 
Posts: 37
Joined: Tue Dec 27, 2011 10:06 am

Re: memory leak on alpha 3.0 metro express m4

by cfloryiv on Mon Apr 23, 2018 10:21 pm

There's a issue in python when a buffer grows and get reallocated. This is not a memory leak issue. Rather, it's a limited memory issue. Microcontrollers dont have virtual memory like a desktop pc. If a buffer grows unchecked, it will eventually grow to fill all of memory.

cfloryiv
 
Posts: 37
Joined: Tue Dec 27, 2011 10:06 am

Re: memory leak on alpha 3.0 metro express m4

by danhalbert on Mon Apr 23, 2018 10:33 pm

Could you put in some print() statements to see why the buffer is growing out of control? Thanks. There's some logic in the loop that can let the buffer grow too large in some error cases. I've added some print statements marked with `### debug` to show the flow.

Code: Select all | TOGGLE FULL SIZE
while True:
    data = uart.read(32)  # read up to 32 bytes
    data = list(data)
    #print("read: ", data)          # this is a bytearray type
    print("read", len(data), "bytes")      ### debug

    buffer += data
    print("buffer length now", len(buffer))      ### debug
   
    while buffer and buffer[0] != 0x42:
        print("looking for 0x42")            ### debug
        buffer.pop(0)
   
    if len(buffer) < 32:
        print("len < 32")         ### debug
        continue

    if buffer[1] != 0x4d:
        print("buffer[1] != 0x4d")       ### debug
        buffer.pop(0)
        continue

    frame_len = struct.unpack(">H", bytes(buffer[2:4]))[0]
    if frame_len != 28:
        print("frame_length != 28")       ### debug
        continue

    frame = struct.unpack(">HHHHHHHHHHHHHH", bytes(buffer[4:]))

    pm10_standard, pm25_standard, pm100_standard, pm10_env, pm25_env, pm100_env, particles_03um, particles_05um, particles_10um, particles_25um, particles_50um, particles_100um, skip, checksum = frame

    check = sum(buffer[0:30])
   
    if check != checksum:
        print("checksum not correct")        ### debug
        continue
...

danhalbert
 
Posts: 1137
Joined: Tue Aug 08, 2017 12:37 pm

Re: memory leak on alpha 3.0 metro express m4

by cfloryiv on Tue Apr 24, 2018 4:49 am

I put the print statement in. The program runs fine for a while, then the count of the buffer increases past 32. It gets caught in the loop and never escapes.

cfloryiv
 
Posts: 37
Joined: Tue Dec 27, 2011 10:06 am

Re: memory leak on alpha 3.0 metro express m4

by danhalbert on Tue Apr 24, 2018 9:18 am

Did you put all the print statements in? Could you show some sample output, showing transitions from good to bad, and eliding multiple repetitions? Thanks.

danhalbert
 
Posts: 1137
Joined: Tue Aug 08, 2017 12:37 pm

Re: memory leak on alpha 3.0 metro express m4

by danhalbert on Wed Apr 25, 2018 3:11 pm

We've added some error checking to the sample program which should prevent the buffer overflow. Try the new version here: https://learn.adafruit.com/pm25-air-qua ... ython-code.

However, we don't expect you to be hitting the error cases: it appears you are sometimes reading what looks like bad data, and it would be good to know why. We haven't been able to replicate the problem you're seeing. So if you get long periods of bad data and can send us some buffer dumps or debugging printouts, that would be very helpful. Thanks.

danhalbert
 
Posts: 1137
Joined: Tue Aug 08, 2017 12:37 pm

Re: memory leak on alpha 3.0 metro express m4

by cfloryiv on Wed Apr 25, 2018 9:58 pm

That fixed the problem, thanks. I just modified your code change to include a "continue" statement.

Code: Select all | TOGGLE FULL SIZE
 
if len(buffer) > 200:
        buffer = []   # avoid an overrun if all bad data
        continue

cfloryiv
 
Posts: 37
Joined: Tue Dec 27, 2011 10:06 am

Re: memory leak on alpha 3.0 metro express m4

by danhalbert on Wed Apr 25, 2018 11:17 pm

Great, thanks! It's still weird we might be reading bad data; of course let us know if you see a lot of dropped data.

danhalbert
 
Posts: 1137
Joined: Tue Aug 08, 2017 12:37 pm

Please be positive and constructive with your questions and comments.