ESP32SPI / MatrixPortal wifi connect issues

CircuitPython on hardware including Adafruit's boards, and CircuitPython libraries using Blinka on host computers.

Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.
Locked
User avatar
Jrbski
 
Posts: 15
Joined: Thu Sep 26, 2019 11:27 pm

ESP32SPI / MatrixPortal wifi connect issues

Post by Jrbski »

OK, this is a weird one... I have a few different wifi networks, and I'm trying to get my matrixportal to send data to adafruit.io. I had used the matrixportal and adafruit.io previously for the text scroller demo. Now I am having issues connecting to 2 of 3 wifi networks. A unifi and asus router won't work, but a 'hacked' dd-wrt router does work. As an example, I attached the debug from wifi manager for the unifi router. Even though the dd-wrt has a weaker signal, it successfully connects. The only obvious difference is the working ssid is all lower case, and any ssid with uppercase fails. All use WPA2 personal.

I have updated from CP6.1 to CP6.2 and the 5/25 libraries with no change in behavior. I don't recall what version of CP the wifi previously worked on, but the unifi firmware was recently updated.

Any suggestions on what else to try? I may try setting up another 'guest' ssid with all lowercase as a last gasp. Note that the debug returns no error info after the 'retry' message that looks like should be printed out based on the code in github. I do have other ESP32's on the unifi wireless with no issues.

Code: Select all

ssID = Unifi
Connecting to WiFi...
ESP32 found and in idle mode
Firmware vers. bytearray(b'1.2.2\x00')
MAC addr: ['0x##', '0x##', '0x##', '0x##', '0x##', '0x##'
        Unifi           RSSI: -63
        RX          RSSI: -63
        JXR         RSSI: -66
        JX              RSSI: -66
        dd-wrt              RSSI: -81

Connecting to AP...
Failed to connect, retrying
 ('No such ssid', b'Unifi')
Connecting to AP...
Failed to connect, retrying
 ('Failed to connect to ssid', b'Unifi')

User avatar
Jrbski
 
Posts: 15
Joined: Thu Sep 26, 2019 11:27 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by Jrbski »

One update... I did look at the logs on Unifi, and I can see that the MatrixPortal did connect, and get an IP address from dhcp, but then I see a failure, so it looks like the issue is on the esp32 side.

User avatar
Jrbski
 
Posts: 15
Joined: Thu Sep 26, 2019 11:27 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by Jrbski »

Latest update: I did try adding an all lowercase ssid to the same access point, and that made no difference. I also updated the ESP32 firmware to 1.7.3, with no changes in behaviour. MatrixPortal connects to the access point, gets an IP from DHCP, and then disconnects. I never get a ping response from the ESP while it is connecting, so it seems like the wifi connection is never fully completed.

User avatar
Jrbski
 
Posts: 15
Joined: Thu Sep 26, 2019 11:27 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by Jrbski »

So I found a hint from another website that noted similar wifi issues related to CP 6.1... the difference is that my 'working' AP is also running as a DHCP server, and the AP's that aren't are using a separate DHCP server. The delay in getting DHCP may be causing the ESP32 to give-up. I see there is an 'AP_connect' method that has a time-out setting that I will try to replace the 'connect' method next before I try to roll back to CP 6.0.

User avatar
Jrbski
 
Posts: 15
Joined: Thu Sep 26, 2019 11:27 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by Jrbski »

I'm pretty much stuck now... the connect_AP delay didn't help, and neither did the rollback to CP 6.0. I enabled all the debugging I can. The only thing I see is that my AP reports the link in low power mode, but that is consistent with other esp's on the network...

User avatar
adafruit_support_mike
 
Posts: 67391
Joined: Thu Feb 11, 2010 2:51 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by adafruit_support_mike »

You'll probably need to sniff the network traffic with something like Wireshark to see which signals are being sent and which are being missed.

If the ESP32 can connect to one network but not to the others, you should be able to compare the traffic and see where things stop working.

User avatar
Jrbski
 
Posts: 15
Joined: Thu Sep 26, 2019 11:27 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by Jrbski »

Just to complete the documentation, this is what I see in the Unifi AP logs... I don't see any errors, so it looks like the ESP32 is just sending the disconnect.
details on the connect success log message
details on the connect success log message
Unifi_success_detail.png (7.06 KiB) Viewed 358 times
Log of wifi connect sequence
Log of wifi connect sequence
Unifi_log.png (28.11 KiB) Viewed 358 times

User avatar
Jrbski
 
Posts: 15
Joined: Thu Sep 26, 2019 11:27 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by Jrbski »

adafruit_support_mike wrote:You'll probably need to sniff the network traffic with something like Wireshark to see which signals are being sent and which are being missed.

If the ESP32 can connect to one network but not to the others, you should be able to compare the traffic and see where things stop working.
So by pure luck, I happened to catch a successful connection to the Unifi AP, so I was able to compare a failure to a successful connection. On the success, you can see that the matrix portal successful connects to adafruit.io. In the failure, it seems to time out processing the DHCP response, and makes another DHCP request. The DHCP packet sent in both cases is the same, so more data that points to the ESP32. Note that there is a delay second ARP request in the 'success' dump, so it looks like that is when the ESP32 is processing the DHCP packet.

The single success seems to point to a timing issue, either on the DHCP response, or the ESP32 setting the IP and/or DNS settings in the DHCP packet. As a note, the .11.22 address is my pi-hole DNS, and I am not using pi-hole on the dd-wrt AP.
TCPdump comparison between a successful and failed association
TCPdump comparison between a successful and failed association
Unifi_success_detail.png (101.85 KiB) Viewed 352 times

User avatar
adafruit_support_mike
 
Posts: 67391
Joined: Thu Feb 11, 2010 2:51 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by adafruit_support_mike »

Hmm.. the failure explicitly declines the address it gets from the DHCP server (line 71 on the right).

That makes sense from the traffic that follows the DHCP config packet. The ESP32 checks to see if any other device responds to the same address (who-has 192.168.11.183 tell 0.0.0.0), sees a reply from 70:85:c2:etc, and sends back a DECLINE packet because the address is already in use.

The part I can't understand is why the version on the left ignores those replies and takes the address anyway.

Do you happen to know which of your devices has the MAC beginning with 70:85:c2?

User avatar
Jrbski
 
Posts: 15
Joined: Thu Sep 26, 2019 11:27 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by Jrbski »

adafruit_support_mike wrote:Hmm.. the failure explicitly declines the address it gets from the DHCP server (line 71 on the right).

That makes sense from the traffic that follows the DHCP config packet. The ESP32 checks to see if any other device responds to the same address (who-has 192.168.11.183 tell 0.0.0.0), sees a reply from 70:85:c2:etc, and sends back a DECLINE packet because the address is already in use.

The part I can't understand is why the version on the left ignores those replies and takes the address anyway.

Do you happen to know which of your devices has the MAC beginning with 70:85:c2?
Well 70:85:c2 is a 'mystery device', but you definitely put your finger on the problem... I 'cheated' as a work-around by assigning a static IP for the matrixportal, and that solved the issue for now.

I checked arp tables on pi-hole, and a linux box, and they correctly had the matrixportal on .11.183 but a windows box still had 70:85:c2 in its arp table. It looks like a windows box on the network has a stale arp table linked to an old device. I guess the timing issue is related to how quickly the 'bad' arp response gets to the matrixportal. Unfortunately I was only watching TCPdump on the command line, so I don't know (yet) who was serving the ARP packet with the bad info. Some more detective work to track that part down, but it looks like windows may be 'helping'... NOT!

Thanks for the help!

User avatar
Jrbski
 
Posts: 15
Joined: Thu Sep 26, 2019 11:27 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by Jrbski »

Final mystery solved! 70:85:c2 = Asrock was a good hint. It turns out that a windows desktop (with an Asrock MB) was in an odd power state after a windows update. It looked like it was in 'sleep' mode, but it was mostly dead... no ping response, no display, not seen by dhcp server, etc. but the nic was 'alive' enough to respond to ARP requests! NMAP intense scan helped be persistent enough to hear an ARP response from it. Disconnect ethernet cable and its gone, plug back in an ARP response is back. I needed to power it off to bring it back.

Now is actually shows up in the DHCP tables, so all is well, and the network is ready for Adabox 018 which in now on the way!

User avatar
adafruit_support_mike
 
Posts: 67391
Joined: Thu Feb 11, 2010 2:51 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by adafruit_support_mike »

Jrbski wrote:It turns out that a windows desktop (with an Asrock MB) was in an odd power state after a windows update. It looked like it was in 'sleep' mode, but it was mostly dead... no ping response, no display, not seen by dhcp server, etc. but the nic was 'alive' enough to respond to ARP requests!
Oy.. gotta love network debugging. It's such a fertile source of 'wait 'til you hear this' stories that it's almost worth the therapy.

FWIW after the fact, check your router to see if it keeps a device table. If you're lucky it will have a list of devices it's seen that includes human readable names, IP addresses, and MACs.

I'm glad to hear that line of approach turned out to be useful, and that you were able to find/fix the problem. Happy hacking!

User avatar
Jrbski
 
Posts: 15
Joined: Thu Sep 26, 2019 11:27 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by Jrbski »

adafruit_support_mike wrote:
Jrbski wrote:It turns out that a windows desktop (with an Asrock MB) was in an odd power state after a windows update. It looked like it was in 'sleep' mode, but it was mostly dead... no ping response, no display, not seen by dhcp server, etc. but the nic was 'alive' enough to respond to ARP requests!
Oy.. gotta love network debugging. It's such a fertile source of 'wait 'til you hear this' stories that it's almost worth the therapy.

FWIW after the fact, check your router to see if it keeps a device table. If you're lucky it will have a list of devices it's seen that includes human readable names, IP addresses, and MACs.

I'm glad to hear that line of approach turned out to be useful, and that you were able to find/fix the problem. Happy hacking!
My router does have the full list of devices, and that is where I started the fun! unfortunately, the ethernet card on the desktop was alive enough to break the matixportal, but not enough to register with dhcp on the router. The router showed that the matrixportal owned 11.183, so it was a really weird failure mode.

User avatar
adafruit_support_mike
 
Posts: 67391
Joined: Thu Feb 11, 2010 2:51 pm

Re: ESP32SPI / MatrixPortal wifi connect issues

Post by adafruit_support_mike »

Yeah that was an oddball.

Locked
Please be positive and constructive with your questions and comments.

Return to “Adafruit CircuitPython”