32u4 serial port disappears - can't debug

[daimusou]

I can load code onto one of my Feather32u4's so I know the cable is good. I nearly always have to use the method of double-clicking the RST button just after starting the Upload process in the IDE. But shortly afterward the /dev/ttyACM0 device simply disappears on the Linux end and the Serial Monitor cannot open.

Watching in the system journal as I double-click the RST button I see this:

Code: Select all

Apr 20 09:01:39 bluebird kernel: usb 1-3.3: new full-speed USB device number 54 using xhci_hcd
Apr 20 09:01:40 bluebird kernel: usb 1-3.3: new full-speed USB device number 55 using xhci_hcd
Apr 20 09:01:40 bluebird kernel: usb 1-3.3: New USB device found, idVendor=239a, idProduct=000c
Apr 20 09:01:40 bluebird kernel: usb 1-3.3: New USB device strings: Mfr=2, Product=1, SerialNumber=0
Apr 20 09:01:40 bluebird kernel: usb 1-3.3: Product: Adafruit Feather
Apr 20 09:01:40 bluebird kernel: usb 1-3.3: Manufacturer: Adafruit In
Apr 20 09:01:40 bluebird kernel: cdc_acm 1-3.3:1.0: ttyACM0: USB ACM device
Apr 20 09:01:40 bluebird mtp-probe[7862]: checking bus 1, device 55: "/sys/devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.3"
Apr 20 09:01:40 bluebird mtp-probe[7862]: bus: 1, device: 55 was not an MTP device

But then a few seconds later it does this:

Code: Select all

Apr 20 09:01:50 bluebird kernel: usb 1-3.3: reset full-speed USB device number 55 using xhci_hcd
Apr 20 09:01:50 bluebird kernel: usb 1-3.3: device firmware changed
Apr 20 09:01:50 bluebird kernel: usb 1-3.3: USB disconnect, device number 55
Apr 20 09:01:51 bluebird kernel: usb 1-3.3: device not accepting address 58, error -71
Apr 20 09:01:51 bluebird kernel: usb 1-3.3: new full-speed USB device number 59 using xhci_hcd
Apr 20 09:01:51 bluebird kernel: usb 1-3.3: Device not responding to setup address.
Apr 20 09:01:52 bluebird kernel: usb 1-3.3: Device not responding to setup address.
Apr 20 09:01:52 bluebird kernel: usb 1-3.3: device not accepting address 59, error -71
Apr 20 09:01:52 bluebird kernel: usb 1-3-port3: unable to enumerate USB device

Something is not working in the Feather's USB emulation. And notice how the device address keeps changing. It is pretty difficult to debug my code if I can't keep the serial port working for more than a few seconds. Is there something wrong with the bootloader? Can I refresh it somehow? I have other Feather32u4 devices and they do not act like this.

Otherwise, the device seems to be working correctly. But then I go to change something in the programming, which of course breaks something and I have to debug it. But I can't because the Serial port will not keep working long enough.

[johnpark]

After upload, try checking Arduino > Tool > Port and pick the new port the OS may have just selected for your board. Then try to open the serial monitor. I've had this happen sometimes on mac os, not sure about the particulars of your setup in Linux, but it may be a similar workflow.
-John

[daimusou]

It completely disappears, even from 'lsusb' (list all USB devices).

[daimusou]

Here is a really small test case that might or might not be looking at the same problem. I wrote a tiny program that just prints "4" to the serial port every four seconds:

Code: Select all

void setup() {
  Serial.begin(9600);
}

void loop() {
  delay(4000);
  Serial.println("4");
}

The Feather is connected to a USB port and shows up as /dev/ttyACM0. I try to print out what is coming in through that port and it behaves as expected for a short while:

Code: Select all

Four> more /dev/ttyACM0
4
4

At that point it just hangs. Sometimes after just one "4" comes out. Sometimes an EOF condition or something happens and the 'more' command terminates.

[adafruit_support_mike]

The ATmega32u4 handles USB communication internally, and it takes a few milliseconds for it to set up the connection. The Serial object doesn't exist until then.

The setup() function usually takes a few microseconds to run, and will call Serial.begin() before that object exists. The class is written so that call will fail silently instead of crashing the code, but all Serial.print() calls later in the code will also fail.

You need to make setup() wait until the interface is ready:

Code: Select all

void setup () {
    while ( ! Serial ) { delay( 1 ); }
    Serial.begin( 9600 );
}

[daimusou]

I changed that, but it does not affect that it works for a few seconds and then stops. Is there a way I can reload the microcode?

[adafruit_support_mike]

That sounds like an OS level problem. What OS are you using?

[daimusou]

Ubuntu Linux 16.04.1. The Feather is connected to a powered USB hub.

The failure is erratic. Sometimes it works, but mostly it doesn't. Using the 'cat' command to print what comes in from the serial port either results in an EOF, or a message 'Input-Output error'.

[daimusou]

As a test, I ran my simple test program shown above on both an Arduino Nano (328p processor) and a Feather (32u4), with the Linux 'cat' program printing the results. I killed the Nano test case after 351 iterations (23 minutes). The Feather case would hang after just one or two iterations. In both cases I set the Linux end of the connection in advance using the 'stty' program, specifying "-raw" and a baud rate of 9600. Clearly there is something 'fragile' about the 32u4 serial port.

[daimusou]

Googling around, I have found several reports of problems with the 32u4 emulation of a serial port over USB. Most complaints are about the lack of direct access to the DTR pin, and that without seeing DTR high, attempts to write to the port simply do nothing. Several people have sworn off using 32u4 processors for anything involving Serial communication.

So I wonder if the ARM Cortex M0, particularly the version used in the Feather M0, has the same problem. I have not found the same complaints. The other approach I am going to try is putting the separate RFM69 radio breakout onto an Arduino Nano, which i happen to have at hand. I only need one of these, to act as an interface between my Linux computer and my growing cloud of Feather RFM69 devices.

[adafruit_support_mike]

Try using the udev rules from this tutorial:

https://learn.adafruit.com/adafruit-ard ... udev-rules

The Linux USB stack has some quirks of its own, and those often help.

[daimusou]

I already had the ModemManager-disabling rule and the changing access rights. But I found something interesting. If I keep the device busy sending stuff to the host, it keeps working. But if it waits too long between print calls, it stalls.

I modified my test program to start out leeping for 500ms before calls to Serial.println. Then every 10 iterations, I slow down by an additional 500ms. Everything works fine up until the delay becomes 3000ms, at which point no more input is seen on the Linux side.

In my application, it is relaying reports from various remote sensors linked by RFM69 radios. Whole minutes or hours might pass between reports. As a workaround, I have put a 'heartbeat' signal into my code that sends a short message over the serial port every 2 seconds to keep things working. On the Linux side it just ignores this message. But I would like to know why this is happening.

[daimusou]

Futher experiments. The Feather32u4 sends a simple message over the USB serial port every 2 secods. The message is ASCII text, about 4 bytes, with CRLF at the end. The host program watches for these and counts how many arrive in every 60-second period. There should be 30. It prints a one-line report for any period in which fewer than 100% of these are received. It can go half an hour with 100% reception, but then over a period of 5 minutes it will only be getting 60-93%. Then back to 100% again. The host is not busy doing anything else.

I repeated this using an Arduino Nano (328p) and got reliable 100% reception for 20 minutes. But a Nano does not have the onboard radio so is not useful for my real purpose.

Today I expect to receive a FeatherM0 (with radio) and will repeat the experiment with that.
Update: FeatherM0 with RFM69 radio dropped only two USB "keep alive" messages in 30 minutes, where the 32u4 is 100x worse. Radio works fine too. So I will use the M0 as my host-to-IoT gateway and use 32u4's only where no USB I/O is required.

[adafruit_support_mike]

The data you've collected is interesting, but I'm afraid we don't have an explanation for what's happening. We know the Linux USB stack has some low-level quirks that are unlikely to change, since USB is used for so many things.

For the sake of further data collection (if you're still interested in doing that), try putting a cheap/old USB-2 hub between the computer and the Feather. The hub will have its own USB client device, and that will talk directly to the computer. If the dropouts you're seeing are the result of a hardware/signal issue, that should change the rate of failure. If not, the dropouts are more likely to be related to protocol issues.

[daimusou]

Hmm, it was plugged into a powered hub (shared by my keyboard and mouse). I don't know if it is USB-2. I just tried plugging each Feather (32u4 and M0) directly into a USB port on the system box and Ubuntu did not recognize either one of them as a serial device at all! dmesg had errors like:

Code: Select all

[16145.426596] usb 1-1: new full-speed USB device number 22 using xhci_hcd
[16145.426737] usb 1-1: Device not responding to setup address.
[16145.634735] usb 1-1: Device not responding to setup address.
[16145.842588] usb 1-1: device not accepting address 22, error -71
[16145.842644] usb usb1-port1: unable to enumerate USB device

The system box is a relatively new System76 "Wild Dog".
Plugging the M0 back into the hub gives the succesful messages:

Code: Select all

[16212.186635] usb 1-3.3: new full-speed USB device number 23 using xhci_hcd
[16212.293801] usb 1-3.3: New USB device found, idVendor=239a, idProduct=800b
[16212.293804] usb 1-3.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[16212.293807] usb 1-3.3: Product: Feather M0
[16212.293809] usb 1-3.3: Manufacturer: Adafruit
[16212.295930] cdc_acm 1-3.3:1.0: ttyACM0: USB ACM device

As for the 'protocol', it is equivalent to Serial.println(4); every two seconds.