0

Let's try again, more info on AIO problem, please help
Moderators: adafruit_support_bill, adafruit

Forum rules
If you're posting code, please make sure your code does not include your Adafruit IO Active Key or WiFi network credentials.
Please be positive and constructive with your questions and comments.

Let's try again, more info on AIO problem, please help

by Rcayot on Tue Mar 09, 2021 10:17 am

in a series of posts I have requested help on this issue. The basics are that I have a sort of working application that has a bme280 sensor on an itsybitsy nRF52840. The nRF sends tha tsenso data to an RPI Zero W via BLE, and the RPI Zero W sends the data to AIO via wifi.

The issue is that occasionally I get an error. Here is the error stack with the last few 'working' lines of the terminal output while running. As you can see, the last value before the error and program crash is 25.4125, which is a humidity reading. The next reading was to be a pressure. The program quits on an error.

Code: Select all | TOGGLE FULL SIZE
21.4195
25.4808
1006.81
21.3937
25.4125

Traceback (most recent call last):
  File "pibleuart_bme280.py", line 85, in <module>
    get_and_send_pressure()
  File "pibleuart_bme280.py", line 66, in get_and_send_pressure
    aio.send_data(pressure_feed.key, str(pressure))
  File "/usr/local/lib/python3.7/dist-packages/Adafruit_IO/client.py", line 154, in send_data
    return self.create_data(feed, payload)
  File "/usr/local/lib/python3.7/dist-packages/Adafruit_IO/client.py", line 254, in create_data
    return Data.from_dict(self._post(path, data._asdict()))
  File "/usr/local/lib/python3.7/dist-packages/Adafruit_IO/client.py", line 127, in _post
    self._handle_error(response)
  File "/usr/local/lib/python3.7/dist-packages/Adafruit_IO/client.py", line 108, in _handle_error
    raise RequestError(response)
Adafruit_IO.errors.RequestError: Adafruit IO request failed: 422 Unprocessable Entity - request failed - Record invalid. Failed to save data to Rcayot/feeds/pressure, data missing required value


The error is a "RequestError" and number 442, raised by the client.py portion of the AIO library. In the client portion of teh AIO code the lrelevent lines are:
Code: Select all | TOGGLE FULL SIZE
# Handle all other errors (400 & 500 level HTTP responses)
        elif response.status_code >= 400:
            raise RequestError(response)


So here the error is a 'request error" and is logged as error 442, which is an HTTP error evidently, but I can find no listing of what teh error actually is. The other informaiton in the error is:
Code: Select all | TOGGLE FULL SIZE
Adafruit_IO.errors.RequestError: Adafruit IO request failed: 422 Unprocessable Entity - request failed - Record invalid. Failed to save data to Rcayot/feeds/pressure, data missing required value


Unfortunately, it does not convey enough information. What "data missing required value" is missing? Now note above that the last printer value was a humidity value. In the code running on the nRF52840, the last few lines before the connection to the RPI Zero W is lost contains the pressue reading. So, as far as I know, the sensor was successfully queried and returned a pressure value. The nRF then loses its connection.

The RPI Zero W does not 'print' the pressue value, (the print statement is before the 'send' command. So the error is in actually sending the data to AIO, not recieving the data from the nRF board via BLE.

Okay, now here is something else:
the error logged on AIO is as follows:
Code: Select all | TOGGLE FULL SIZE
MQTT ERROR: error saving data to Rcayot/feeds/pressure, data missing required value


This indicates an MQTT error which is supposed to be different from a 'request error"

Anyway, I want to be able to capture the error in an exception(?) and resend or re-connect to not drop out of the program. If anyone can help, it would be appreciated.

Also, and suggestions on troubleshooting the error so I can fix it would also be helpful. My internet connection is via satellite so, long latency times are not uncommon.

Roger Ayotte

Rcayot
 
Posts: 148
Joined: Sat Feb 08, 2020 6:48 pm

Re: Let's try again, more info on AIO problem, please help

by adafruit_support_carter on Tue Mar 09, 2021 12:55 pm

Anyway, I want to be able to capture the error in an exception(?) and resend or re-connect to not drop out of the program. If anyone can help, it would be appreciated.

You could do something like that by surrounding the problematic lines in a try/except block.

Also, and suggestions on troubleshooting the error so I can fix it would also be helpful. My internet connection is via satellite so, long latency times are not uncommon.

Try printing out the value of pressure in your get_and_send_pressure() function. Somewhere before this line:
Code: Select all | TOGGLE FULL SIZE
      File "pibleuart_bme280.py", line 66, in get_and_send_pressure
        aio.send_data(pressure_feed.key, str(pressure))

adafruit_support_carter
 
Posts: 20297
Joined: Tue Nov 29, 2016 2:45 pm

Re: Let's try again, more info on AIO problem, please help

by Rcayot on Tue Mar 09, 2021 1:08 pm

Carter, thanks for the help.

1. I have added a print and a time.sleep(1) and it helps but does not solve the issue.

2. An error/except block would be great if I knew how to do it! I would need a good example or two to get there.

3. Lastly, I looked up the HTTP error 442. It says that the payload the recieving service is in the correct format, but that the contents are not able to be processed.

So, I know the code is sending SOMETHING, and the AIO rejects it. My suspicion is that either the string is empty, (even though the sending code has printed it) somehow.

I think I could recover from this and resend etc. My previous example, the RPI Zero W controlled the sensor through I2C and sent the data through wifi to AIO. That example had each read of sensor sent one after the other, with a 300 second sleep, and that worked for weeks without dropping.

Is there any way the BLE and wifi components of teh PI are interfering with each other? I have increased the wait time between sends to 15s.

Lastly what is the difference between

aio.send(temperature_feed.key, temperature)

and aio.send_data(temperature_feed.key, temperature)

Roger

Rcayot
 
Posts: 148
Joined: Sat Feb 08, 2020 6:48 pm

Re: Let's try again, more info on AIO problem, please help

by adafruit_support_carter on Tue Mar 09, 2021 2:52 pm

So, I know the code is sending SOMETHING, and the AIO rejects it. My suspicion is that either the string is empty, (even though the sending code has printed it) somehow.

Adding the print statement should have verified this. What gets printed for pressure prior to hitting the issue?

adafruit_support_carter
 
Posts: 20297
Joined: Tue Nov 29, 2016 2:45 pm

Re: Let's try again, more info on AIO problem, please help

by Rcayot on Tue Mar 09, 2021 3:23 pm

In the nRF console, the pressue is printed before it is sent:
Code: Select all | TOGGLE FULL SIZE
while True:
    ble.start_advertising(advertisement)
    print("Waiting to connect")
    while not ble.connected:
        pass
    print("Connected")
    while ble.connected:
        s = uart.readline()
        if s:
            try:
                result = (eval(s))
                #print(result)
            except Exception as e:
                result = repr(e)
                print(result, 'e')
            uart.write(str(result))
            print(result)


The last thing printed by teh nRF console (serial window) is the pressure. So, it was sent to the RPI via BLE. Also for the RPI, this is the pertinent portion:

Code: Select all | TOGGLE FULL SIZE
def get_and_send_pressure():
    s = "bme280.pressure"
    uart_service.write(s.encode("utf-8"))
    uart_service.write(b'\n')
    time.sleep(1)
    pressure = uart_service.readline().decode("utf-8")
    print(pressure)
    time.sleep(1)
    aio.send_data(pressure_feed.key, str(pressure))


the print(pressure) occurs before the send to AIO.

Thanks,
Roger

Rcayot
 
Posts: 148
Joined: Sat Feb 08, 2020 6:48 pm

Re: Let's try again, more info on AIO problem, please help

by adafruit_support_carter on Tue Mar 09, 2021 3:47 pm

And what showed up for the case where you hit the issue?

adafruit_support_carter
 
Posts: 20297
Joined: Tue Nov 29, 2016 2:45 pm

Re: Let's try again, more info on AIO problem, please help

by Rcayot on Tue Mar 09, 2021 5:13 pm

there appears to be an empty line feed on teh terminal of the RPI before teh printing of the errors. I have no idea if that line space would be there or not. For example the latest problem. The inserted codeblock below is the terminal output of the RPI. Normally the data is just printed. Here you see a blank line before the error list. On the nRF serial monitor, there was a pressue printed. I have now changed the code to print after the aio.send line.

Code: Select all | TOGGLE FULL SIZE
21.4195
25.4808
1006.81
21.3937
25.4125

Traceback (most recent call last):
  File "pibleuart_bme280.py", line 85, in <module>
    get_and_send_pressure()
  File "pibleuart_bme280.py", line 66, in get_and_send_pressure
    aio.send_data(pressure_feed.key, str(pressure))
  File "/usr/local/lib/python3.7/dist-packages/Adafruit_IO/client.py", line 154, in send_data.... etc....

Rcayot
 
Posts: 148
Joined: Sat Feb 08, 2020 6:48 pm

Re: Let's try again, more info on AIO problem, please help

by adafruit_support_carter on Tue Mar 09, 2021 5:42 pm

That probably means you're getting a bad value for pressure - which is what the blank line is. You could make it more verbose with:
Code: Select all | TOGGLE FULL SIZE
    print("Pressure = ", pressure)


When you try and then call:
Code: Select all | TOGGLE FULL SIZE
aio.send_data(pressure_feed.key, str(pressure))

it ends up with nothing for the value which is why it complains with "data missing required value".

You could add a conditional after the call:
Code: Select all | TOGGLE FULL SIZE
    pressure = uart_service.readline().decode("utf-8")

to check and make sure pressure is valid and then only send to AIO if true.

adafruit_support_carter
 
Posts: 20297
Joined: Tue Nov 29, 2016 2:45 pm

Re: Let's try again, more info on AIO problem, please help

by Rcayot on Tue Mar 09, 2021 9:16 pm

Okay, I have added some extra print stuff for diagnostics.

In the nRF code, I have added a print(result, 'sent') after the "uart.write(str(result))" line.

Here is the last few lines of output frmo the nRF serial terminal:
Code: Select all | TOGGLE FULL SIZE
2.6771 sent
28.1021 sent
1011.52 sent
22.6977 sent
28.0384 sent
1011.59 sent
22.6771 sent
Waiting to connect


So the last item sent was a temperature (comes after the 1000millibar pressure. The waiting to connect is due to teh RPI Zero W program stopping on an error.

In the RPI code, I added a print(temperature, 'recieved') after receiving the data from the nRF and a print(temperature, 'sent') after the aio.send line.
Code: Select all | TOGGLE FULL SIZE
def get_and_send_temperature():
    s = "bme280.temperature"
    uart_service.write(s.encode("utf-8"))
    uart_service.write(b'\n')
    time.sleep(1)
    temperature = uart_service.readline().decode("utf-8")
    print(temperature, 'received')
    time.sleep(1)
    aio.send_data(temperature_feed.key, str(temperature))
    print(temperature, 'sent')


Now, here is interesting part. The following are the last few lines of the serial output to terminal on the RPI.
Code: Select all | TOGGLE FULL SIZE
28.1021 received
28.1021 sent
1011.52 received
1011.52 sent
22.6977 received
22.6977 sent
28.0384 received
28.0384 sent
1011.59 received
1011.59 sent
 received
Traceback (most recent call last):
  File "pibleuart_bme280.py", line 84, in <module>
    get_and_send_temperature()
  File "pibleuart_bme280.py", line 46, in get_and_send_temperature
    aio.send_data(temperature_feed.key, str(temperature))
  File "/usr/local/lib/python3.7/dist-packages/Adafruit_IO/client.py", line 154, in send_data
    return self.create_data(feed, payload)
  File "/usr/local/lib/python3.7/dist-packages/Adafruit_IO/client.py", line 254, in create_data
    return Data.from_dict(self._post(path, data._asdict()))
  File "/usr/local/lib/python3.7/dist-packages/Adafruit_IO/client.py", line 127, in _post
    self._handle_error(response)
  File "/usr/local/lib/python3.7/dist-packages/Adafruit_IO/client.py", line 108, in _handle_error
    raise RequestError(response)
Adafruit_IO.errors.RequestError: Adafruit IO request failed: 422 Unprocessable Entity - request failed - Record invalid. Failed to save data to Rcayot/feeds/temperature, data missing required value

Note the 1011.59 recieved and sent, and then a blank space and 'received' just before the error/

So, the last line of the nRF was a temperature value having been sent. The last line on the RPI is nothing or a blank 'recieved.

So since a 'nothing' was recieved, it was also sent on to aio, hence the error.

Now, I guess I can test to see if the string received is what? Null? not string? I am not sure of the syntax.

And then I should send back a request to resent the last data?

I understand what need to be done, just not certain I know HOW to do it.

Thaks Carter.

Any more suggestions would be very helpful. Even a pointer to some examples of error checking etc. would be beneficial.

Roger

Rcayot
 
Posts: 148
Joined: Sat Feb 08, 2020 6:48 pm

Re: Let's try again, more info on AIO problem, please help

by jwcooper on Wed Mar 10, 2021 10:22 am

You could try something like this:

Code: Select all | TOGGLE FULL SIZE
print(temperature, 'received')
temp_str = str(temperature)
if temp_str and len(temp_str) > 0:
    time.sleep(1)
    aio.send_data(temperature_feed.key, temp_str)
    print(temp_str, 'sent')


Also, as carter suggested earlier, you can wrap your send_data call in a try except block to just catch the exception and continue on with the script. You can find examples of that here:
https://wiki.python.org/moin/HandlingExceptions

jwcooper
 
Posts: 782
Joined: Tue May 01, 2012 9:08 pm

Re: Let's try again, more info on AIO problem, please help

by adafruit_support_carter on Wed Mar 10, 2021 11:27 am

Yep. Something like that. There's no single answer. The basic idea is:
Code: Select all | TOGGLE FULL SIZE
if temp_str is OK:
    send it to AIO

but the specific logic to use for determining if it's "OK" may take some trial and error. In your case, it seems to be blank, so
Code: Select all | TOGGLE FULL SIZE
if temp_str and len(temp_str) > 0:

checks for that and hopefully might be good enough.

adafruit_support_carter
 
Posts: 20297
Joined: Tue Nov 29, 2016 2:45 pm

Re: Let's try again, more info on AIO problem, please help

by Rcayot on Wed Mar 10, 2021 8:32 pm

Thanks guys.

So far, I have had no issues for 12 hours. I have moved teh RPI away from the nRF board, a few feet, the RPI may also be in a slightly better reception area for wifi.

I will be able to modifyt the code with your suggestions, and I will let you know how it goes!

Roger

Rcayot
 
Posts: 148
Joined: Sat Feb 08, 2020 6:48 pm

Re: Let's try again, more info on AIO problem, please help

by Rcayot on Thu Mar 11, 2021 3:46 pm

Guys,

I have made some changes, and want to include:

if temp_str and len(temp_str) > 0:

But what happens if this returns False?

I was thinking I would add an else clause. But what would run the function over again?


Code: Select all | TOGGLE FULL SIZE
# pibleuart_ble280
# Connect to an "eval()" service over BLE UART.
# have nRF52840 evaluate temperature, humidity and pressure and return them to RPI bia BLE.
# Rpi will send data to aio via wifi
# Roger Ayotte March 5, 2021
# based on the eval example

# Connect to an "eval()" service over BLE UART.
import time
from adafruit_ble import BLERadio
from adafruit_ble.advertising.standard import ProvideServicesAdvertisement
from adafruit_ble.services.nordic import UARTService

ble = BLERadio()
uart_connection = None
sensor_read_delay = 300

from Adafruit_IO import Client, Feed, Data, RequestError

# Set to your Adafruit IO key.
# Remember, your key is a secret,
# so make sure not to publish it when you publish this code!
ADAFRUIT_IO_KEY = 'aio_Fzve6425emwZSbMDz9nTeAvjI1dj'

# Set to your Adafruit IO username.
# (go to https://accounts.adafruit.com to find your username).
ADAFRUIT_IO_USERNAME = 'Rcayot'

# Create an instance of the REST client.
aio = Client(ADAFRUIT_IO_USERNAME, ADAFRUIT_IO_KEY)

# Set up Adafruit IO Feeds.
temperature_feed = aio.feeds('temperature')
humidity_feed = aio.feeds('humidity')
pressure_feed = aio.feeds('pressure')

def get_and_send(data, key):
    #print(data, key)
    uart_service.write(data.encode("utf-8"))
    uart_service.write(b'\n')
    time.sleep(1)
    data = uart_service.readline().decode("utf-8")
    data_str = str(data)
    if len(data_str) > 0:
        print(data, 'received')
        time.sleep(1)
        aio.send_data(key.key, str(data))
    else:
        return
    print(data, 'sent')

while True:
    if not uart_connection:
        print("Trying to connect...")
        for adv in ble.start_scan(ProvideServicesAdvertisement):
            if UARTService in adv.services:
                uart_connection = ble.connect(adv)
                print("Connected")
                break
        ble.stop_scan()

    if uart_connection and uart_connection.connected:
        uart_service = uart_connection[UARTService]
        while uart_connection.connected:
            bme280request = 'bme280.temperature', 'bme280.humidity', 'bme280.pressure'
            keys = temperature_feed, humidity_feed, pressure_feed
            for data, key in zip(bme280request, keys):
                get_and_send(data, key)
                time.sleep(10)
            time.sleep(600) 

Rcayot
 
Posts: 148
Joined: Sat Feb 08, 2020 6:48 pm

Re: Let's try again, more info on AIO problem, please help

by jwcooper on Thu Mar 11, 2021 7:05 pm

It will skip data collection for that run of your loop. For most things, this would probably be fine, if you really need the data, you may need to get creative.

If you want it to try again, you could try recursively calling the same method:

Code: Select all | TOGGLE FULL SIZE
def get_and_send(data, key):
    #print(data, key)
    uart_service.write(data.encode("utf-8"))
    uart_service.write(b'\n')
    time.sleep(1)
    data = uart_service.readline().decode("utf-8")
    data_str = str(data)
    if len(data_str) > 0:
        print(data, 'received')
        time.sleep(1)
        aio.send_data(key.key, str(data))
    else:
        get_and_send(data, key)
    print(data, 'sent')


That could end up in an infinite loop if for some reason your data isn't being received, so you may want to fail after a few tries or so. You can track that with a counter.

jwcooper
 
Posts: 782
Joined: Tue May 01, 2012 9:08 pm

Re: Let's try again, more info on AIO problem, please help

by Rcayot on Fri Mar 12, 2021 10:55 am

JW thank you.

The system has been running for over 12hrs without a visible error. There may have been an error and dropped a data point, but I do not have a flag set to log the error.

Next step is to place the nRF52840 with sensor outside and powered by USB. The RPI will be about 20' away, right inside. Hopefully the BLE module will work at that distance.

Thanks again,

Roger

Rcayot
 
Posts: 148
Joined: Sat Feb 08, 2020 6:48 pm

Please be positive and constructive with your questions and comments.