Incrementing a count, getting interpreted after 8?

CircuitPython on hardware including Adafruit's boards, and CircuitPython libraries using Blinka on host computers.

Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.
User avatar
xanatos
 
Posts: 110
Joined: Thu Jun 18, 2009 3:09 pm

Incrementing a count, getting interpreted after 8?

Post by xanatos »

I'm incrementing a count.

The base count is derived from taking the last two digits of a string returned from a device. I explicitly know it's a string because I use str(data).

It's great up to 08. On 09, it displays it as \t (which is a tab character in ascii) and everything bombs out.

I've tried converting to int, eval, repr, even str to str.... same behavior every time. Count hits what should be 09, it displays as \t and blows up.

I feel dumb. Please help :)

User avatar
BooleanMattock
 
Posts: 49
Joined: Wed Nov 16, 2022 8:14 am

Re: Incrementing a count, getting interpreted after 8?

Post by BooleanMattock »

xanatos wrote: Thu Jan 26, 2023 10:37 pm I'm incrementing a count.

The base count is derived from taking the last two digits of a string returned from a device. I explicitly know it's a string because I use str(data).

It's great up to 08. On 09, it displays it as \t (which is a tab character in ascii) and everything bombs out.

I've tried converting to int, eval, repr, even str to str.... same behavior every time. Count hits what should be 09, it displays as \t and blows up.

I feel dumb. Please help :)
Could you share the code with us to take a look at?

User avatar
xanatos
 
Posts: 110
Joined: Thu Jun 18, 2009 3:09 pm

Re: Incrementing a count, getting interpreted after 8?

Post by xanatos »

There's a lot of extra stuff in the full code - here's the parts that are doing the serial stuff, plus my full imports (in case anything there bumps into anything else):

Code: Select all

import board
import digitalio
import terminalio
import busio
import time
import adafruit_ds3231 # RTC, Hi Precision  I2C 0x68
from digitalio import DigitalInOut, Direction, Pull
from analogio import AnalogOut, AnalogIn
import microcontroller  # for uP temp measurements
import math
import binascii
...

Code: Select all

uart = busio.UART(board.TX, board.RX, baudrate=9600, timeout=.04)
i2c = board.I2C()
The following three are used to talk between the controller and a TED-96 display from BreadBoard Mates. They have been simplified for testing to send just a single command and receive a response. First, the code:

Code: Select all

def sendDisplay(data):
    message_bytes = binascii.unhexlify(data)
    uart.write(message_bytes)
    rxdata = uart.read(32) # read up to 32 bytes
    if rxdata == None:
        rxdata = 999
    return rxdata
    
def getCurrentPage():
    data = "240001" #Query Active Page
    print(rxdata)
    rxdata = sendDisplay(data)
    return rxdata

def disHexify(data):  #Removes all the b' and \x stuff to leave a raw text number string
     data = str(binascii.hexlify(data),'X')
     data = str(data)
     print("dishexified data, raw, = " + data)
     return data
Running getCurrentPage sends a hex string (240001) to the display, and the display responds fine with a string like:

Code: Select all

b'\x06\x00\x00'
So far so good. The 06 is an acknowledgement that the command was received and proccessed OK, and the 00 00 is the hex code for page 0.

I "dishexify" the string (not to be confused with binascii.unhexlify) to turn it into a string of numbers, in this case 060000. All good so far.

Now, once I have the Current Page, I have a navigation switch that I use to increment or decrement which page the display is showing. I can successfully increment up to page 9 (060009)

Now when I *have* incremented up to page 09, the hex raw string that the display returns is, I'm pretty sure,

Code: Select all

b'\x06\x00\x09'
But my code above, in getCurrentPage(), says it's getting a string of:

Code: Select all

b'\x06\x00\t'
So, note the final that I want to see as \x09 is now being interpreted as the ascii character "Horizontal Tab", \t.

I have tried to convert this string in many ways, from plain old str(rxdata), hex(rxdata)... I've tried several flavors of binascii.unhexlify(data) (and hexlify), with types of 'X', 'ascii', etc.

I succeed only in either turning my rxdata right back to it's original

Code: Select all

b'\x06\x00\t'
Or bombing out with unable to convert errors of one type or another.

So I know the display is more-than-likely NOT sending that \t. But everything I have ready to handle the -probable- \x09 the display is sending wants to make that into a \t - and that makes it a problem because I can't increment beyond that point. I have 20+ pages I need to access.

I have even done the very ugly manual replacement of the \t using the string.replace() method, but then the next character, \x0A - blows up with whatever interpreted character that is. I know there is a much more elegant "Pythonic" way to fix this issue...

Oh - and last point - yes, I do need to work with these hex bytes as strings because of what gets sent to the display and a very large library of hex-command-strings-as-ascii-text (like "24FFFE0000001454686973206973206120746573742E2020787878", which is built from hex prefixes, hex addresses, element IDs in hex, and some canned messages in hex...

Hopefully I have been clear here and it all makes sense. Thanks for help!

User avatar
neradoc
 
Posts: 542
Joined: Wed Apr 27, 2016 2:38 pm

Re: Incrementing a count, getting interpreted after 8?

Post by neradoc »

The display IS sending b"\t", because b"\t" is only the human representation of b"\x09". Bytes strings are represented like ASCII strings in the REPL (with a in front) and escaping bytes that are not characters. \t is 9, b"\x48" is b"H". But you seem to have accidentally converted it into it's representation by calling str() on it without an encoding.

You are using strings with hexadecimal representations of the bytes that you need to send to the display. And you need to stick data that comes as binary data from the display, if I understand correctly. All you need to do is hexlify the data.

Code: Select all

>>> data = b"\x06\x00\x09"
>>> data # same data, just represented differently, \t is 9
b'\x06\x00\t'
>>> repr(data) # just for display, if you print() that, you get the line above
"b'\\x06\\x00\\t'"
>>> str(data) # same, don't do that
"b'\\x06\\x00\\t'"
>>> binascii.hexlify(data) # the hex representation, \t is still 9
b'060009'
But then I assume you tried to add it to a string, which wouldn't work, you can't add objects of type bytes and str. One possibility is to convert it to a str with binascii.hexlify(data).decode(), but I think it would be better to use bytes for all of your hex strings (just add a b in front).

Code: Select all

def getCurrentPage():
    data = b"240001" #Query Active Page
    rxdata = sendDisplay(data)
    return rxdata

Code: Select all

SOME_COMMAND = b"32" # let's pretend that is the code for "go to page"
data = b"\x06\x00\x09"
page = data[1:] # skip the first byte
hex_data = SOME_COMMAND + binascii.hexlify(page)
sendDisplay(hex_data)

But I would advise using binary data to begin with, even if it's a little more annoying to type at first (you should only have to convert the source code once, by adding b in front and \x every 2 characters).

Code: Select all

def sendDisplay(data):
    uart.write(data)
    rxdata = uart.read(32) # read up to 32 bytes
    if rxdata == None:
        rxdata = 999
    return rxdata

def getCurrentPage():
    data = b"\x24\x00\x01" #Query Active Page
    rxdata = sendDisplay(data)
    return rxdata

Code: Select all

SOME_COMMAND = b"\x32" # binary data
sendDisplay(SOME_COMMAND + page)

User avatar
xanatos
 
Posts: 110
Joined: Thu Jun 18, 2009 3:09 pm

Re: Incrementing a count, getting interpreted after 8?

Post by xanatos »

Thank you. I'm playing with this now. Your last two suggestions about just going with b"\x24\x00.... may be my way out of this mess. I have half the issue solved doing this... This'll keep me occupied for most of the rest of the day :) I'll reply when/if I have complete success.

User avatar
xanatos
 
Posts: 110
Joined: Thu Jun 18, 2009 3:09 pm

Re: Incrementing a count, getting interpreted after 8?

Post by xanatos »

OK, this has gotten me a lot of the way there, but I now seem to be having issues with simple addition :)

So, to call a new page, I have this string:

Code: Select all

pageCallPrefix = b"\x24\x00\x00\x00"
To this string is added the page index. So page 0 is:

Code: Select all

pageCallPrefix = b"\x24\x00\x00\x00\x00"
and page 1 is:

Code: Select all

pageCallPrefix = b"\x24\x00\x00\x00\01"
Now when I call for the current page (say it's "\x00"), I want to add to it \x01, to make the last byte \x01.

Here's the code I'm trying to get running:

Code: Select all

def doPageUD(dir):
        cp = getCurrentPage()
        if cp[:1] == b"\x06":  #First 2 chars, 06 = RX Data Good")
            print("RX data OK in UD")
            currPage = cp[-1:]
            print("CurrPage in UD = " + str(currPage))
            if dir == "U":
                print("Direction UP")
                newPage = currPage + b"\x01"
                newPage = pageCallPrefix + newPage[-1:]
                print("new " + str(newPage))
                sendDisplay(newPage)
            if dir == "D":
                print("Direction Down")
                print(pageCallPrefix)
In that mess I am trying to add to the current page (gotten by querying the display and taking the last byte [-1:], and adding + b"\x01". What that is doing is *appending* another 01 to the end. When I take the newPage var and just the last byte and append it to the PageCallPrefix, it does call page 1 (the display successfully changes from page 0 to page 1), but it hasn't incremented the page value - it just took my appended \x01 and stuck it to the end of the pageCallPrefix. It seems the "+" is acting as glue rather than an operation lol.

I need to take the CurrPage value, which is the last byte of data received from getCurrentPage, and increment it by one each time (so if currPage is \x00, it'll be \x01 after I add 1. If it's \x09 it'll be \x0A, etc.)

I feel like an idiot here... lol This should be simple.

Everything else you gave me earlier (just going with bytes (b"\x24\x00\x00.....") has rectified a number of issues. The only remaining issue I can determine is this incrementing of the page call thing. I need to take the current page, add one, and send it back to the display to jump to the next page. I feel like I'm missing something fundamental here. You wouldn't believe how many ways I've tried to increment, rather than append, that last byte.... but feel free to show me how utterly obvious the answer is, really lol. I'm pulling out the very last fine bits of hair I have left on my head! :)

User avatar
xanatos
 
Posts: 110
Joined: Thu Jun 18, 2009 3:09 pm

Re: Incrementing a count, getting interpreted after 8?

Post by xanatos »

OK, so I completely re-invented the code based on your recommendations, and then fixed the stuff that still didn't work, and I came to the following (full code, unrelated stuff removed) that works:

Code: Select all

#v0.01   2023-01-27

import board
import digitalio
import terminalio
import busio
import time
import adafruit_ds3231 # RTC, Hi Precision  I2C 0x68
from digitalio import DigitalInOut, Direction, Pull
from analogio import AnalogOut, AnalogIn
import microcontroller  # for uP temp measurements
import math
import binascii

time.sleep(0.5) #Wait for display to come up

pageCallPrefix = "24000000"
setBackLightPrefix = "24000600" # + 01-FF

led = digitalio.DigitalInOut(board.LED)
led.direction = digitalio.Direction.OUTPUT

# Analog input on A1-3
analog1in = AnalogIn(board.A1)  #
analog2in = AnalogIn(board.A2)  #
analog3in = AnalogIn(board.A3)  #

navIn1 = DigitalInOut(board.D7)
navIn1.direction = Direction.INPUT
navIn1.pull = Pull.UP

navIn2 = DigitalInOut(board.D9)
navIn2.direction = Direction.INPUT
navIn2.pull = Pull.UP

navIn3 = DigitalInOut(board.D10)
navIn3.direction = Direction.INPUT
navIn3.pull = Pull.UP

navIn4 = DigitalInOut(board.D11)
navIn4.direction = Direction.INPUT
navIn4.pull = Pull.UP

navIn5 = DigitalInOut(board.D12)
navIn5.direction = Direction.INPUT
navIn5.pull = Pull.UP

uart = busio.UART(board.TX, board.RX, baudrate=9600, timeout=.04)
i2c = board.I2C()

ds3231 = adafruit_ds3231.DS3231(i2c)
days = ("Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun")
t = ds3231.datetime
DateText = "{} {}/{:02}/{:02}".format(days[int(t.tm_wday)], t.tm_year, t.tm_mon, t.tm_mday)
TimeText = "{}:{:02}:{:02}".format(t.tm_hour, t.tm_min, t.tm_sec)
text_time = DateText + " " + TimeText
print(text_time)

def makeBank(u,d,l,r,c):
    fullBank = ""
    rawBank = [u,d,l,r,c]
    for item in rawBank:
        #print("This item = " + str(item))
        if item == False:
            fullBank = fullBank + "1"
        else:
            fullBank = fullBank + "0"
    return fullBank

def sendDisplay(data):
    uart.write(data)
    rxdata = uart.read(32) # read up to 32 bytes
    if rxdata == None:
        rxdata = 999
    return rxdata
    
def getCurrentPage():
    data = binascii.unhexlify("240001") #Query Active Page
    rxdata = sendDisplay(data)
    return rxdata

def clamp(n, minn, maxn):
    return max(min(maxn, n), minn)

def doPageUD(dir):
        cp = getCurrentPage()
        cp2 = str(binascii.hexlify(cp))
        cp2 = cp2.replace("b'","")
        cp2 = cp2.replace("\'","")
        if cp[:1] == b"\x06":
            cpS = eval(cp2[-2:])
            if dir == "U":
                cpSnew = cpS + 1
                cpSnew = clamp(cpSnew,0,14)
            elif dir == "D":
                cpSnew = cpS - 1
                cpSnew = clamp(cpSnew,0,14)
            cpSnew = str(cpSnew)
            if len(cpSnew) < 2:
                cpSnew = "0" + cpSnew
            newPage = pageCallPrefix + cpSnew
            print(newPage)
            data = binascii.unhexlify(newPage)
            rxdata = sendDisplay(data)

def doPageLRC(dir):
    print(dir)

t = binascii.unhexlify("2400000000") #Reset to Page 0
reset = sendDisplay(t)

while True:
    navC = navIn1.value
    navL = navIn2.value
    navD = navIn3.value
    navR = navIn4.value
    navU = navIn5.value
    navBank = makeBank(navU,navD,navL,navR,navC)
    navBank = eval(navBank)

    if navBank == 10000 and lastNB == 0:  #Page Up Button
        doPageUD("U")

    if navBank == 1000 and lastNB == 0:  #Page Down Button
        doPageUD("D")

    if navBank == 100 and lastNB == 0:  #Page Up Button
        doPageLRC("L")

    if navBank == 10 and lastNB == 0:  #Page Down Button
        doPageLRC("R")

    if navBank == 1 and lastNB == 0:  #Page Down Button
        doPageLRC("C")

    cp = getCurrentPage()
   
    if cp == b"\x06\x00\x01":  #6x20 Text Screen
        if cp != lastPage:  #Keeps it from repeatedly sending the same string
            data = b"\x24\xFF\xFE\x00\x00\x00\x14\x54\x68\x69\x73\x20\x69\x73\x20\x61\x20\x74\x65\x73\x74\x2E\x20\x20\x78\x78\x78" #"This is a test.  xxx"
            color = b"\x24\x00\x08\x00\x00\xFF\xA0"  #Yellow
            sendDisplay(color)
            time.sleep(.02)
            sendDisplay(data)

    if cp == b"\x06\x00\x09":  #Clock Screen
        if cp != lastPage:  #Keeps it from repeatedly sending the same string
            print("Clock Screen")
            t = ds3231.datetime
            DateText = "{} {}/{:02}/{:02}".format(days[int(t.tm_wday)], t.tm_year, t.tm_mon, t.tm_mday)
            TimeText = "{}:{:02}:{:02}".format(t.tm_hour, t.tm_min, t.tm_sec)
            text_time = DateText + " " + TimeText
            print(text_time)
            D1 = b"\x24\x00\x02\x05\x0E\x00\x06"  # 6
            D2 = b"\x24\x00\x02\x05\x0F\x00\x0C"  # 12
            D3 = b"\x24\x00\x02\x05\x10\x00\x3B"  # 59
            TXT = b"\x24\xFF\xFF\x00\x00\x53\x41\x20\x32\x32\x2F\x30\x31\x2F\x32\x38\x00"  # SA 22/01/28
            sendDisplay(D1)
            time.sleep(.02)
            sendDisplay(D2)
            time.sleep(.02)
            sendDisplay(D3)
            time.sleep(.02)
            sendDisplay(TXT)
            time.sleep(.02)

    lastNB = navBank  #Must be at END of navigation Selection.  Prevents run-on-hold
    lastPage = cp
    time.sleep(.05)
The only thing that remains is that when I page up/down, it does so in regular old base 10, so it skips 0A-0F and jumps to 10... I just need to figure out now how to get it to increment is hex so I get the 0A-0F pages :)

Thank you for your help, it got me out of a stuck place :)

User avatar
neradoc
 
Posts: 542
Joined: Wed Apr 27, 2016 2:38 pm

Re: Incrementing a count, getting interpreted after 8?

Post by neradoc »

Hi, here is a version that should work for your page numbers, but since I can't test it there might a typo or error still. When you have a bytes string, like what uart.read() gives you, you can get an element as an int by indexing into it. Do not convert to strings (ever).
So to test if it starts with 6: cp[0] == 6
To get the page number: cpS = cp[2]
etc.
Attachments
screen_things.py
(4.91 KiB) Downloaded 1 time

User avatar
xanatos
 
Posts: 110
Joined: Thu Jun 18, 2009 3:09 pm

Re: Incrementing a count, getting interpreted after 8?

Post by xanatos »

Thank you Neradoc. I hadn't realized I could index the hex because it was represented as "\x00", etc. I thought those \x characters would screw it up. Apparently that's just a representation but NOT how the data is actually held in the variable...?

In any case, your modifications were in fact exactly what I was needing. The code increments/decrements through all the currently loaded pages without a hitch. I'm slowly going through this now to be sure my brain is properly wrapped around what's going on with hex representations.

I've been programming in Python for a few years now, but never having to deal with manipulating hex data "as such", and never on a microcontroller (all Raspberry Pi stuff). Learning on a microcontroller is a lot like I'm learning it all over again :) Kinda fun, actually!

Thanks again - your assistance is much appreciated.

Dave

User avatar
xanatos
 
Posts: 110
Joined: Thu Jun 18, 2009 3:09 pm

Re: Incrementing a count, getting interpreted after 8?

Post by xanatos »

One last confusion issue and I promise I'll get out of your hair! :)

I'm trying to convert a text string to hex. An example is "SU 22/01/29". I need this to return as, in this case, b'\x53\x55\x20\x32\x32\x2f\x30\x31\x2f\x32\x39'

I'm trying many things that are "close" but not it.

Using this function:

Code: Select all

def ascii_to_hex(ascii_str):
    hex_str = binascii.hexlify(ascii_str.encode())
    return hex_str
If I send my text string:

Code: Select all

hex_output = ascii_to_hex("SU 22/01/29")
print(hex_output) 
The result is b'53552032322f30312f3239'

The numbers are right, but no "\x".

To complicate matters, I need to append the (proper) output to a built-in prefix (b"\x24\xFF\xFF\x00\x00") so that the final resultant string would be b'\x24\xFF\xFF\x00\x00\x53\x55\x20\x32\x32\x2f\x30\x31\x2f\x32\x39'

binascii.hexlify doesn't seem to want to do this. And I know I can't make it a string and just stuff "\x" in between every two characters... I'm still missing something fundamental in my understanding of converting between ascii strings and hex bytes in CircuitPython....

Thanks again, very much. Wish I could buy you a coffee or something lol! :)

Dave

PS: To further my confusion:

Code: Select all

test = binascii.unhexlify('11abcdef')
print(test)  #result = b'\x11\xab\xcd\xef'
test = binascii.unhexlify('53552032322f30312f3239')
print(test)  #result = b'SU 22/01/29'
So I get exactly what I expect when I use random hex values that are NOT what I need, but when I use the exact same function, but feed it the hex values I want to use, it converts it back to text... which I am assuming is just the representation... how do I get it to represent with the b'x\...' format?

User avatar
neradoc
 
Posts: 542
Joined: Wed Apr 27, 2016 2:38 pm

Re: Incrementing a count, getting interpreted after 8?

Post by neradoc »

Converting a string to the matching bytes type is just encode().
This encodes the string (which is UTF-8) into an array of bytes using the default encoding (UTF-8).
And then, regardless of the encoding, the bytes type is presented in the REPL as an ASCII string.
In this case the string is only ASCII characters so it's the same.

Code: Select all

>>> "SU 22/01/29".encode()
b'SU 22/01/29'
>>> b'\x53\x55\x20\x32\x32\x2f\x30\x31\x2f\x32\x39'
b'SU 22/01/29'
With a non-ascii UTF-8 string, you will see all the bytes of the extended characters even though they are treated as a single character in the string:

Code: Select all

>>> len("🐍")
1
>>> "🐍".encode()
b'\xf0\x9f\x90\x8d'

User avatar
xanatos
 
Posts: 110
Joined: Thu Jun 18, 2009 3:09 pm

Re: Incrementing a count, getting interpreted after 8?

Post by xanatos »

So I can concatenate b"\x24\xFF\xFF\x00\x00" with "SU 22/01/29".encode() and get a valid string? Still confused...

>>> "SU".encode()
b'SU'
>>> b"\x24\xFF\xFF\x00\x00" + "SU".encode()
b'$\xff\xff\x00\x00SU'

😆. Feeling dumb still...

User avatar
neradoc
 
Posts: 542
Joined: Wed Apr 27, 2016 2:38 pm

Re: Incrementing a count, getting interpreted after 8?

Post by neradoc »

Talking about this is confusing because the type is called "bytes" but it's an array of bytes (a byte being an 8 bit number or a value between 0 and 255), I call the type "bytes string" because of the b"" syntax, or "bytes object". I have called them "bytes array" above too, as more of a description: it's an array of integers of 1 byte size each.

b"\x24\xFF\xFF\x00\x00" is an object of type bytes.
"SU 22/01/29".encode() returns an object of type bytes (also represented as b"SU 22/01/29")

So you can add them together to get:
b"\x24\xFF\xFF\x00\x00" + b"SU 22/01/29" == b"\x24\xFF\xFF\x00\x00SU 22/01/29"

UART.read() returns an object of type bytes.
UART.write() takes an object of type bytes.

Well actually write() takes any raw buffer type, but let's not get into that.

FYI the other main type of binary data object is called bytearray just to make everything less confusing. The difference is that bytes is unmutable (like strings, you cannot write val[x] = y), but bytearray is mutable.
You could lookup the python docs on bytes or some tutorial online to get to use them a bit and clarify what it is.

User avatar
xanatos
 
Posts: 110
Joined: Thu Jun 18, 2009 3:09 pm

Re: Incrementing a count, getting interpreted after 8?

Post by xanatos »

So that's why the first b'\x24 gets represented as b'$ in my above example, correct?

So that should mean I *can* send b"$\xFF\xFF\x00\x00SU 22/01/29" and it will get properly interpreted by the receiving display?

User avatar
neradoc
 
Posts: 542
Joined: Wed Apr 27, 2016 2:38 pm

Re: Incrementing a count, getting interpreted after 8?

Post by neradoc »

Yes, $ is 0x24:

Code: Select all

>>> hex(ord("$"))
'0x24'
>>> chr(0x24)
'$'

Locked
Please be positive and constructive with your questions and comments.

Return to “Adafruit CircuitPython”