Due to high demand expect some shipping delays at this time, orders may not ship for 1-2 business days.
0

Long term server issues with Adafruit Feather M0 WiFi with A
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Long term server issues with Adafruit Feather M0 WiFi with A

by tkr65536 on Sun Oct 17, 2021 12:04 am

Hi,
I have an Adafruit Feather M0 WiFi with ATWINC1500. Additionally, I have the Adalogger feather wing, and a 4x7-segment LED feather wing. I'm using WiFi101 library version 0.16.1 and the ATWINC1500 has firmware 19.5.4.

They read sensor values and store them internally. Periodically, another application connects to the board and queries it for values. I'm using the Server class in the WiFi101 library.

After an extended period of time (6 days, 7 days), the board stops responding to requests from the server. If I try to open a connection manually, I get a "connection refused". From the serial logs on the board, it's as if nothing happened - server.available() always evaluated to false.

When this happens, the board still thinks it's connected to the Wi-Fi network and can be pinged. Additionally, the board can ping the router, and if I restart the Wi-Fi AP, the board detects this loss of connection and reconnects automatically when the AP is back up.

So far, the biggest change I made is to close the client object (client.stop()) that server.available() returns. I hadn't been doing that before.

Back in March, I inquired about firmware version 19.6.1 as that's available. As of then, it hadn't been tested, but my plan is to test it now. See https://forums.adafruit.com/viewtopic.php?f=57&t=176984&p=864967&hilit=wifi+firmware for details. If this doesn't work, I'm next going to see if there's a way to restart the Server object, or reset the ATWINC1500 chip, but I'm looking for other recommendations.'


Thank you very much,
Teddy

tkr65536
 
Posts: 43
Joined: Sun Jun 25, 2017 12:08 pm

Re: Long term server issues with Adafruit Feather M0 WiFi wi

by tkr65536 on Mon Oct 25, 2021 11:11 pm

Just following up on this issue...

tkr65536
 
Posts: 43
Joined: Sun Jun 25, 2017 12:08 pm

Re: Long term server issues with Adafruit Feather M0 WiFi wi

by tkr65536 on Mon Oct 25, 2021 11:15 pm

For some context, this is a newer manifestation of the issue described here https://forums.adafruit.com/viewtopic.php?f=57&t=181107

However since then I found that I wasn't closing the client object returned by server.available(), but unfortunately that hasn't fixed my issue.

tkr65536
 
Posts: 43
Joined: Sun Jun 25, 2017 12:08 pm

Re: Long term server issues with Adafruit Feather M0 WiFi wi

by User_UMjT7KxnxP8YN8 on Tue Oct 26, 2021 12:25 am

I worked extensively with Adafruit's Metro M4 Express and WINC1500 shield a couple of years ago and one of the issues I found is that it has relatively few TCP sockets available, 7 if I recall correctly.

Getting the WINC1500 to run for extended periods of time (mine ran for months without issue) requires managing the sockets carefully. The WiFi101 library provides no direct way to do this, but one of the things you CAN do is to make sure that your webpages specify "Connection: close" rather than "Connection: keep-alive" in the HTTP header.

The downside to this is that every interaction with the server will require a new connection to be established, but usually only a few hundred milliseconds, which isn't a bad trade for having your server remain available.

Remember that if you load a webpage from your Feather M0 server that references multiple objects on the server (images, .css files, javascript files, etc), every one of them consumes a socket. If you are specifying "Connection: keep-alive" these sockets will remain open, which is fine if you plan to hit the server from the same device again soon. But let's say you load a page that consumes 4 sockets from one IP address, then try to load the same page from another IP address. You'll only have 3 sockets available and one of the 4 objects you're requesting won't load.

If the client machine closes the connection properly, you should quickly recover the open sockets. But if the client fails to send a socket close request your server sockets will sit there waiting for longer than you would probably like. I don't recall offhand how long that takes in the WiFi101 library.

One other issue you should consider is that if your Feather M0 is sitting naked on the internet (no firewall in front of it), script kiddies will pound it all day and all night from all over the globe. I have a Metro M4 Express server that's been exposed unprotected for over 2 years and it logs every access. You'd be amazed how much free time some people have.

If this is happening to you, one of their favorite tricks is the 'SYN Flood' attack, though with the WiFi101 library it doesn't take much of a 'flood' to shut your server down.

It works like this... the attacker sends your server a 'SYN' packet (request to connect) from a spoofed IP address. Your server allocates a socket to the request and sends a 'SYN/ACK' packet back. But there's nobody home on the other end, so your server sits there with a socket tied up until the request times out. Then they do it again, and again and... eventually all 7 sockets are tied up and when you try to access your server it is non-responsive because it has no sockets available for you to connect to.

You can combat this somewhat by reducing the default socket timeout of 60 seconds by calling the WIFICLASS::set_timeout(new_timeout) method and setting new_timeout (in milliseconds) to a lower value. It won't eliminate the problem, but it will help.

I replaced the WINC1500 in my system with a WizNet 5500 Ethernet shield and wrote extensive countermeasures involving low-level socket management and stochastic attack defenses but never went back and did the same with the WINC1500.

You can find my W5500 library (based on the excellent starting point provided by Paul Stoffregen) at https://github.com/SapientHetero/Ethernet. I've not published the code for my SYN Flood defense but you can look at manageSockets() in sockets.cpp in that library to get a sense of what I mean by 'socket management'. I used Kali Linux to launch a SYN Flood attack at a rate of 1 SYN packet every 500 microseconds against my server from within my own network (a worst-case scenario from a timing perspective) and it was slowed but still managed to load a webpage that used 7 sockets.

You can contact me directly via github if you'd like to discuss this in more detail.

User_UMjT7KxnxP8YN8
 
Posts: 272
Joined: Tue Jul 17, 2018 1:28 pm

Re: Long term server issues with Adafruit Feather M0 WiFi wi

by tkr65536 on Tue Oct 26, 2021 12:40 am

Thank you very much for the advice. My feathers are sitting behind firewalls and the plan is to keep them there.

The server querying the feather for data is also behind the firewall.

The note about the seven sockets available is good to know. I could definitely imagine one not being closed. Do you know if it's possible to query the number of open sockets and/or close them from the server side?


I'm actually working at a lower level than http - specifically I'm implementing a Zabbix agent - see https://www.zabbix.com/documentation/5. ... ivepassive for details on the protocol.


Thank you again for your suggestion. I'll definitely look into monitoring the number of open sockets and see where I can get.

Thank you,
Teddy

tkr65536
 
Posts: 43
Joined: Sun Jun 25, 2017 12:08 pm

Re: Long term server issues with Adafruit Feather M0 WiFi wi

by User_UMjT7KxnxP8YN8 on Tue Oct 26, 2021 1:12 am

I don't recall there being any application-level access to socket info in WiFi101 for WINC1500. And the code has so many layers that it wasn't easy to follow back in those days when I was just starting to program again after decades of managing.

Note that you also get 4 UDP sockets in addition to the 7 TCP sockets. I'm not familiar with the protocol you're using, but this may prove useful to you too.

You can always do what I did with the W5500 library and write your own method to allow you to query and close them, but be forewarned, you're going to want to understand exactly what each socket's current state is before closing it. I time-tagged each socket every time its state changed so I knew how long each had been idle, and closed 'orphaned' sockets such as those waiting for an ACK for a close packet, or that had received a SYN but no corresponding ACK. You can garner quite a bit of good info by looking at my W5500 code, as the socket state logic is broadly applicable. There are copious comments in it to help me remember what I did if I ever have to dig in again, and they'll help you do something similar with the WINC1500 library.

If you do rework the WiFi101 library, I'd like to hear what you do and how it works in case I go back to the WINC1500 again. A github link would be much appreciated.

Oh, and if you modify a library, make sure to keep a copy somewhere other than the Arduino 'libraries' folder or it may be lost during future update. I almost learned that the hard way after modifying WiFi101 to do WiFi SSID scans asynchronously only to find that it was gone after an update; fortunately I have a robust backup protocol.

User_UMjT7KxnxP8YN8
 
Posts: 272
Joined: Tue Jul 17, 2018 1:28 pm

Please be positive and constructive with your questions and comments.