0

help with performance for datalogger
Moderators: adafruit_support_bill, adafruit

Forum rules
Adafruit MicroPython is currently EXPERIMENTAL and BETA - Please visit https://learn.adafruit.com/category/micropython and http://forum.micropython.org/ in addition to our section here!
Please be positive and constructive with your questions and comments.

help with performance for datalogger

by carl0s on Thu Nov 09, 2017 5:16 pm

Hi.

I have this code...

The "dump_byte_by_byte" function outputs to the console at about ~56 lines per second. That sort of speed is good for me.

but, to write to the sdcard, I suspect I'm going to want to write line by line, so I have been working on a function for that. This is the get_bytes_and_dump_line_by_line() function. With this function though my console output is about 22 lines per second.

Any hints / direction? I don't even have it writing to SD yet (although I tested your examples..) and I want to pull in a second i2c or uart source as well (OBD2).

Here's the code:
Code: Select all | TOGGLE FULL SIZE
uart = busio.UART(board.TX, board.RX, baudrate=9600, timeout=10)

#date = getdate() # will use this for the filename
#filename = str(date.tm_year) + "-" + str(date.tm_mon) + "-" + str(date.tm_mday) + "-" + str(date.tm_hour) + "" + str(date.tm_min)
#filename += ".csv"
def getPacket():
   rcdbyte = [None] * 3
   packet = [b'\x00'] * 14
   i = 0
   while True:
      rcdbyte[i] = uart.read(1)   #read a single byte
      if rcdbyte[i] == None:   #if we got an empty byte, assume the serial link is down, and return zeros.
         return packet
      if rcdbyte[i] == b'\x02' and rcdbyte[i-1] == b'\x01' and rcdbyte[i-2] == b'\x00': # we got our 3 bytes in order for the start of packet
         packet[0] = b'\x00'
         packet[1] = b'\x01'
         packet[2] = b'\x02'

         for x in range(3, 14):   #get the rest of the packet
            packet[x] = uart.read(1)
            if packet[x] == None:   #if we got an empty byte, assume the serial link is down, and return zeros.
               packet[x] = b'\x00'
               return packet
         return packet
      i += 1
      if i == 3:
         i = 0


def get_and_dump_byte_by_byte():
   packet = getPacket()
   print (time.monotonic(), end=',')
   for x in range(0,13):
      print(ord(packet[x]),end=',')
   print(ord(packet[13]),end='')
   print('\n')


def get_bytes_and_dump_line_by_line():
   packet = getPacket()
   logline = ','.join([str(ord(b)) for b in packet])
   print(str(time.monotonic()) + logline)


while True:
   get_and_dump_byte_by_byte()
   #get_bytes_and_dump_line_by_line()

carl0s
 
Posts: 9
Joined: Thu Sep 29, 2016 8:38 pm

Re: help with performance for datalogger

by carl0s on Thu Nov 09, 2017 5:52 pm

I guess it's the string conversion that's slow.

I have improved the main packet collection loop a bit, so that there are 3 less ord() functions on the bytes. i.e. the first 3 fixed bytes are set as integers in the first place, and ord() is used directly on the uart.read. this is instead of doing ord() on all 14 bytes for every packet.

Still only 22 lines per second though :-/

Can't wait for the M4 :D

Code: Select all | TOGGLE FULL SIZE
def getPacket():
   rcdbyte = [None] * 3
   packet = [0] * 14
   i = 0
   while True:
      rcdbyte[i] = uart.read(1)   #read a single byte
      if rcdbyte[i] == None:   #if we got an empty byte, assume the serial link is down, and return zeros.
         return packet
      if rcdbyte[i] == b'\x02' and rcdbyte[i-1] == b'\x01' and rcdbyte[i-2] == b'\x00': # we got our 3 bytes in order for the start of packet
         packet[0] = 0
         packet[1] = 1
         packet[2] = 2

         for x in range(3, 14):   #get the rest of the packet
            packet[x] = ord(uart.read(1))
            if packet[x] == None:   #if we got an empty byte, assume the serial link is down, and return zeros.
               packet[x] = 0
               return packet
         return packet
      i += 1
      if i == 3:
         i = 0

carl0s
 
Posts: 9
Joined: Thu Sep 29, 2016 8:38 pm

Re: help with performance for datalogger

by carl0s on Thu Nov 09, 2017 9:11 pm

I have settled on a solution!

I'm just storing everything in the list[] and just logging that. No string conversion or concatenation.

I am getting 49 - 51 lines per second written to the SD card. 50 x 14 bytes (plus my added timestamp). That is close to the maximum that the Zeitronix box could be pushing out anyway (9,600bps), so I'm pretty happy with that.

I can't get the rtc to work alongside the sd though. Not enough memory I think..

As you can see here, I tried a few different approaches!.. :)

Code: Select all | TOGGLE FULL SIZE
import board
import busio
import time
import adafruit_sdcard
#import adafruit_pcf8523
import microcontroller
import digitalio
import storage
import os


 

switch = digitalio.DigitalInOut(board.A5)
switch.direction = digitalio.Direction.INPUT
switch.pull = digitalio.Pull.UP



#Zeitronix Packet format, bytes[]
#[0] always 0
#[1] always 1
#[2] always 2
#[3] AFR
#[4] EGT Low
#[5] EGT High
#[6] RPM Low
#[7] RPM High
#[8] MAP Low
#[9] MAP High
#[10] TPS
#[11] USER1
#[12] Config Register1
#[13] Config Register2

#led = digitalio.DigitalInOut(board.D13)
#led.direction = digitalio.Direction.OUTPUT

# Use any pin that is not taken by SPI
SD_CS = board.D10
# Connect to the sdcard and mount the filesystem.
spi = busio.SPI(board.SCK, board.MOSI, board.MISO)
cs = digitalio.DigitalInOut(SD_CS)
sdcard = adafruit_sdcard.SDCard(spi, cs)
vfs = storage.VfsFat(sdcard)
storage.mount(vfs, "/sd")
 
# Use the filesystem as normal! Our files are under /sd


uart = busio.UART(board.TX, board.RX, baudrate=9600, timeout=10)


#date = rtc.datetime() # will use this for the filename
#filename = str(date.tm_year) + "-" + str(date.tm_mon) + "-" + str(date.tm_mday) + "-" + str(date.tm_hour) + "" + str(date.tm_min)
filename = "2017-09-11-test"
filename += ".csv"


def getBigPacket():
   packet = [0] * 14
   #print(data)          # this is a bytearray type
   i = 0
   while True:
      data = uart.read(32)  # read up to 32 bytes. Remember we must find our start bytes at most 11 bytes before this.. to get a complete packet
      if data[i] == 2 and data[i-1] == 1 and data[i-2] == 0:
         packet[0] = 0
         packet[1] = 1
         packet[2] = 2
         for x in range(3,14):
            i += 1
            packet[x] = data[i]
         return packet
      i += 1
      if i == 20:
         i = 0


def getPacket():
   rcdbyte = [None] * 3
   packet = [0] * 15
   packet[0] = time.monotonic()
   i = 0
   while True:
      rcdbyte[i] = uart.read(1)   #read a single byte
      if rcdbyte[i] == None:   #if we got an empty byte, assume the serial link is down, and return zeros.
         return packet
      if rcdbyte[i] == b'\x02' and rcdbyte[i-1] == b'\x01' and rcdbyte[i-2] == b'\x00': # we got our 3 bytes in order for the start of packet
         packet[1] = 0
         packet[2] = 1
         packet[3] = 2

         for x in range(4, 15):   #get the rest of the packet.
            packet[x] = ord(uart.read(1))
            if packet[x] == None:   #if we got an empty byte, assume the serial link is down, and return zeros.
               packet[x] = 0
               return packet
         return packet
         #return ', '.join(packet)
      i += 1
      if i == 3:
         i = 0

def getPacketalt():
   packet = [0] * 14
   i = 0
   while True:
      rcdbyte = uart.read(3)   #read 3 bytes
      if rcdbyte[i] == None:   #if we got an empty byte, assume the serial link is down, and return zeros.
         return packet
      if rcdbyte[i] == 2 and rcdbyte[i-1] == 1 and rcdbyte[i-2] == 0: # we got our 3 bytes in order for the start of packet
         packet[0] = 0
         packet[1] = 1
         packet[2] = 2

         for x in range(3, 14):   #get the rest of the packet
            packet[x] = ord(uart.read(1))
            if packet[x] == None:   #if we got an empty byte, assume the serial link is down, and return zeros.
               packet[x] = 0
               return packet
         return packet
      i += 1
      if i == 3:
         i = 0

def getPacketasString():
   rcdbyte = [None] * 3
   packet = "0,0,0,0,0,0,0,0,0,0,0,0,0,0"
   i = 0
   while True:
      rcdbyte[i] = uart.read(1)   #read a single byte
      if rcdbyte[i] == None:   #if we got an empty byte, assume the serial link is down, and return zeros.
         return packet
      if rcdbyte[i] == b'\x02' and rcdbyte[i-1] == b'\x01' and rcdbyte[i-2] == b'\x00': # we got our 3 bytes in order for the start of packet
         packet = "0,1,2,"
         for x in range(3, 14):   #get the rest of the packet
            byte = ord(uart.read(1))

            if byte == None:   #if we got an empty byte, assume the serial link is down, and return zeros.
               packet += "0,"
               return packet
            packet += str(byte) + ','
         return packet
      i += 1
      if i == 3:
         i = 0



def get_and_dump_byte_by_byte():
   packet = getPacket()
   print (time.monotonic(), end=',')
   for x in range(0,13):
      print(packet[x],end=',')
   print(packet[13],end='')
   print('\n')


def get_bytes_and_dump_line_by_line():
   packet = getPacket()
   logline = ','.join([str(b) for b in packet])
   print(str(time.monotonic()) + logline)


def first_attempt_at_sdwriting():
   while True:
      with open("/sd/" + filename, "a") as f:
         packet = getPacket()
         t = time.monotonic()
         print('.',end='')
         f.write('%f,' % (t))
         for x in range(0,13):
            f.write('%d,' % packet[x])
         f.write('%d' % packet[13])
         f.write("\n")
         if switch.value == False:
            f.close()
            print("closing")
            break


while True:
   if switch.value == False:
      time.sleep(0.1)
      f = open("/sd/" + filename, "w")
      while switch.value == True:
         packet = getPacket()
         f.write('{}\n'.format(packet))
         print('.',end='')
      f.close()
      time.sleep(0.1)

carl0s
 
Posts: 9
Joined: Thu Sep 29, 2016 8:38 pm

Re: help with performance for datalogger

by tannewt2 on Tue Nov 14, 2017 12:22 pm

Cool! Another thing to try would be writing to the SD card in 512 byte chunks because thats what it does internally anyway. You do risk losing a bit of data but it should reduce the amount of writes to the card.

tannewt2
 
Posts: 411
Joined: Thu Oct 06, 2016 8:48 pm

Re: help with performance for datalogger

by carl0s on Tue Nov 14, 2017 12:35 pm

tannewt2 wrote:Cool! Another thing to try would be writing to the SD card in 512 byte chunks because thats what it does internally anyway. You do risk losing a bit of data but it should reduce the amount of writes to the card.


I have re-written in Arduino now and things are super fast.

I get my 50hz input, and have loads of loop cycles wasted printing dots on the screen.

Python served really well as a prototyping thing I suppose in this case, and helped me to write it in Wire/C.

carl0s
 
Posts: 9
Joined: Thu Sep 29, 2016 8:38 pm

Re: help with performance for datalogger

by tannewt2 on Tue Nov 14, 2017 12:47 pm

Perfect! Arduino is the go-to for code execution speed! You can't beat a compiler with a VM.

tannewt2
 
Posts: 411
Joined: Thu Oct 06, 2016 8:48 pm

Please be positive and constructive with your questions and comments.