🎁📬❄️📦 Holiday Shipping Deadlines are approaching! International customers and domestic ground - get your orders in 12/9/2020 📦❄️📬🎁
0

Matrixportal - Handling unexpected data
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Matrixportal - Handling unexpected data

by snaxter on Sun Nov 15, 2020 11:12 am

I'm getting an occasional crash (~every other day) when a web site returns data, but doesn't have JSON in the return.

The error is: ValueError: syntax error in JSON

And it happens when the web site, which normally produces perfectly fine JSON, returns the following:

<head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>500 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered an error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>

So, I'm occasionally hitting the web site when it has a server error (it is a bit flakey, but does come back).

I think this is caused by "network" (in adafruit_matrixportal) raising a ValueError (Couldn't parse json:) when it fails to parse the JSON. It seems like "network" should handle the case where the server is asking to try again, or it has parsed valid JSON from that server in the past, but it is temporarily unavailable.

What do you think?

snaxter
 
Posts: 19
Joined: Mon Mar 24, 2008 10:57 pm

Re: Matrixportal - Handling unexpected data

by dastels on Sun Nov 15, 2020 11:58 am

The easiest approach would be to put a try block around the erroring call/code and skip the data processing when the problem occurs. It's a perfect use of exception handling. It's expected, but infrequent, problem.

Dave

dastels
 
Posts: 4251
Joined: Tue Oct 20, 2015 3:22 pm

Re: Matrixportal - Handling unexpected data

by snaxter on Sun Nov 15, 2020 3:04 pm

Thanks for the reply.

I do have a try block around my code that calls network.fetch_data(). Snippet shown below:

Code: Select all | TOGGLE FULL SIZE
       try:
            outside = matrixportal.network.fetch_data(
                                            url=Outside_SOURCE,
                                            json_path=(PM2_Location,TEMP_Location,HUMI_Location),)

            inside = matrixportal.network.fetch_data(
                                            url=Inside_SOURCE,
                                            json_path=(PM2_Location,TEMP_Location,HUMI_Location),)

            wunder = matrixportal.network.fetch_data(
                                            url=Wunder_SOURCE,
                                            json_path=(WTemp_JSon),)

        except RuntimeError as e:
            print("Some error occured, retrying! -", e)
            continue

The code in network.fetch_data() has a try block around it that raises the exception:

Code: Select all | TOGGLE FULL SIZE
            try:
                gc.collect()
                print("1.25")
                json_out = response.json()
                print("1.3")
                if True:
                    print(json_out)
                print("1.4")
                gc.collect()
            except ValueError:  # failed to parse?
                print("Couldn't parse json: ", response.text)
                raise

I would have though my try block would have just done a continue from that error, but t does stop. Do I need to change the network.fetch_data() function to continue on the exception?

snaxter
 
Posts: 19
Joined: Mon Mar 24, 2008 10:57 pm

Re: Matrixportal - Handling unexpected data

by dastels on Sun Nov 15, 2020 4:44 pm

The continue will skip everything else in the enclosing loop, causing it to immediately begin the next iteration. As for the try block, once an exception causes it to exit that's it... you can't pickup after the call that raises the exception... which is what it sounds like you want to do. See https://stackoverflow.com/questions/19522990/python-catch-exception-and-continue-try-block.

Dave

dastels
 
Posts: 4251
Joined: Tue Oct 20, 2015 3:22 pm

Re: Matrixportal - Handling unexpected data

by snaxter on Sun Nov 15, 2020 9:16 pm

Yes, that is helpful. Thanks for the pointer - it makes it clear that the caller can't pickup after an exception is raised in a function.

To be clear, the exception is raised in the library adafruit_matrixport.network.fetch_data(). I can modify that library routine so that will only raise an exception parsing JSON on the first call (should catch true syntax errors), but on subsequent calls, the library will won't raise an exception. Or, perhaps it would be better if the library returned an error type and allowed the caller to handle it.

Should I file an issue since this is in one of the adafruit libraries?

snaxter
 
Posts: 19
Joined: Mon Mar 24, 2008 10:57 pm

Re: Matrixportal - Handling unexpected data

by dastels on Sun Nov 15, 2020 9:55 pm

By throwing an exception, it *is* letting you handle it.

You could wrap each fetch call in separate try blocks.

Dave

dastels
 
Posts: 4251
Joined: Tue Oct 20, 2015 3:22 pm

Please be positive and constructive with your questions and comments.