0

API and Dashboard Different Data
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

API and Dashboard Different Data

by pthacker on Fri Jan 12, 2018 10:50 am

I've created a dashboard with several switches. I use them to control lights in my house.

I have an issue: MQTT data and API data are different. The MQTT data has a switch OFF and the API data has it ON.

Is this a bug or am I doing something wrong?

pthacker
 
Posts: 35
Joined: Sun Jan 06, 2013 2:34 pm

Re: API and Dashboard Different Data

by pthacker on Sat Jan 13, 2018 12:04 pm

Ok here is a little more information. The Feeds shows the garage light switch on

Feeds view with garagelight feed - Switch ON

Feeds.png
Feeds.png (18.1 KiB) Viewed 478 times


If you look at the garagelight feed it is OFF - and the dashboard is also OFF

Feed.png
Feed.png (42.91 KiB) Viewed 478 times


I pull down with API data for an iPhone app and the garagelight feed last updated is ON

API.png
API.png (87.18 KiB) Viewed 478 times


The time stamp matches the feed (MQTT) but the API shows a different value at the same time stamp.

I need a little help here! Am I interpreting the API incorrectly?

Dave

pthacker
 
Posts: 35
Joined: Sun Jan 06, 2013 2:34 pm

Re: API and Dashboard Different Data

by pthacker on Sun Jan 14, 2018 2:17 pm

Ok, I have a little more information. I pulled down the detailed info on a feed using API DOCS. I looks like the "last updated" is using the value from the first entry not the last entry. I'm guessing here but I used the time stamps to make this assumption.

NOTE - the Switch is in the OFF position in on the dashboard and feed displays. See snapshot below.

details.png
details.png (118.07 KiB) Viewed 455 times

pthacker
 
Posts: 35
Joined: Sun Jan 06, 2013 2:34 pm

Re: API and Dashboard Different Data

by abachman on Mon Jan 15, 2018 12:30 pm

Hey pthacker,


Just wanted to ping you here and let you know we're taking a look. I'll post an update when I have more info.


- adam

abachman
 
Posts: 242
Joined: Mon Feb 01, 2010 12:48 pm

Re: API and Dashboard Different Data

by pthacker on Mon Jan 15, 2018 1:29 pm

Adam,

Thank you. Not urgent.

Dave

pthacker
 
Posts: 35
Joined: Sun Jan 06, 2013 2:34 pm

Re: API and Dashboard Different Data

by abachman on Mon Jan 15, 2018 3:42 pm

No worries, this one was very worthing looking into. The key bit was the /details lookup that had a different value in `last_value` than it did in the last data `value` field. There are two hard problems in computer programming: naming things, cache invalidation, and off by one errors. We totally nailed that second one.

We're aggressively caching in the MQTT layer to reduce DB traffic when writing multiple times to the same feed quickly. The result was that in cases where values repeated, the system thought the `last_value` field hadn't changed because the cached value was the same as the new value, so it didn't update the feed record. BUT, it was only when the new value was a repeat of a previously stored value AND when it was repeated inside a 5 minute window. Because we're storing actual data records separately, they were always up to date and accurate, so one consolation is that the incorrect value was never sent to MQTT subscribers or any API that listed data records, so in most cases, IO clients got correct data in spite of the last_value error.

End result, sometimes inconsistent /{username}/feed/{key} last_value and /{username}/feed/{key}/data/last value. Sorry about that.

Anyhow, fixed that bug and updated IO, you shouldn't see it again but do please check and make sure. I'm running a test script that was failing before the deployment with the same error you were seeing and is working fine now.


- adam

abachman
 
Posts: 242
Joined: Mon Feb 01, 2010 12:48 pm

Re: API and Dashboard Different Data

by pthacker on Mon Jan 15, 2018 4:27 pm

Adam,

Great troubleshooting! And thanks for the explaination. Years ago I was working on a MIPS processor and had a problem with stale data in a TLB cache line. It was really hard to fix.

I'll try to verify later today. (I'm trying to reconfigure a Pi Zero with a USB WIFI dongle. Its frustrating with the single micro-USB port. I will ONLY buy Zero Ws in the future)

Dave

pthacker
 
Posts: 35
Joined: Sun Jan 06, 2013 2:34 pm

Re: API and Dashboard Different Data

by pthacker on Tue Jan 16, 2018 12:37 am

Adam,

Problem fixed and verified. Thank you!

Dave

pthacker
 
Posts: 35
Joined: Sun Jan 06, 2013 2:34 pm

Please be positive and constructive with your questions and comments.