Serial USB Data Loss with Teensy

[MiloParis]

I'm currently trying to use a teensy to sample from an analog signal. I've been having problems transmitting the analog data to my program running on the computer (missing bytes, so when I reconstruct the 10-bit value the byte order is off and I get garbage), so I wrote a simpler version to test where it's going wrong.

It's based off the pjrc ADC auto sampler code, and basically sends 16384 bytes every time a request byte is sent from the computer.

The computer program is based off the pjrc bandwidth test: it now sends the request byte ('s' in this case), reads 16384 bytes, then checks to make sure every byte is in ascending order (with overflow handled). It's windows based, and I ran it off an ASUS 1201N 1.6GHz Win7 netbook.

I've included a sample output file generated from a run of the program, with the terminal output pasted at the top. I'm new to programming micro-controllers and serial ports but I'm thinking there must be a buffer overflow happening somewhere. I tried messing with the delay values between bytes sent on the teensy with no luck, I always end up losing data at some point. I've even changed several of the port parameters on the computer side without any difference.

I've been having trouble finding any information on this problem, so I'm thinking I'm doing something wrong somewhere in my program. I was wondering if anyone with more experience with serial programming could help me out.

[adafruit_support_bill]

Buffer overruns are a common problem with serial communication. If the receiver doesn't empty the buffer before it fills up, some data will be lost. Common methods to handle it are:

1) slow down the transmission rate.
2) use hardware handshaking (DTR/RTS)
3) use software handshaking (XON/XOFF)
4) packetize the data with checksums and re-transmit on error.

[philba]

It wasn't clear if your original code sends 16K blocks or just the test code. If it's the original code, try decreasing your block size. I'd test different sizes all the way down to 1K.

Also, what are you using to read the data on the Windows PC? If it stops listening, it will lose data.

[MiloParis]

Thanks for the clarification.

The problem is I'm trying to sample as fast as I can, so slowing down the transmission rate is a last resort. The absolute minimum sampling frequency I can have is around 400 Hz, which leaves a maximum 2.5 ms between samples.

I have a few questions about AVR:
Where are interrupts called? Do they cause a jump in the code whenever they're ready or are they processed at a certain point in the program?

If they're processed whenever they're ready, I was thinking of setting an auto-sampling frequency, making the sampling ISR dump the sample to a buffer of a certain length, signal when that buffer's full, then switch to another buffer. In that case I can just make the main while loop wait for the full buffer signal then write that buffer to serial without having to worry about timing messing up since the ISR will cause a jump whenever it's ready. Are these safe assumptions? The alternative will be to manage the timing myself.

Finally, do you have any suggestions for fast checksum algorithms?

@philba
It's the teensy that sends data to the computer program.

I've tried many different combinations of block size and timing delays, yet have never been able to not lose data. I'll double check as soon as I can get access to a teensy. The program I'm running should be included in the storage.zip file I uploaded, the file inside called "main.cpp".

The computer program basically just checks for data until an error occurred or 16384 bytes are sent. If the bytes were transferred completely, it tests to make sure they were sent in order (the data sent by the teensy is simply 0->255 repeating) by checking if the next number in the array is equal to the current one + 1 in a for loop.

[adafruit_support_bill]

Do you need to transmit the data continuously while sampling? It is possible to log to an SD card at up to 40,000 Hz. You could transmit the data to the PC later.
http://forums.adafruit.com/viewtopic.php?f=31&t=21728&p=113915&hilit=high+speed+data+logger#p113915

[MiloParis]

The end result of this sampling is to perform real-time DSP on the computer, so the delay between sampling and receiving it on the computer has to be as small as possible. Unfortunately storage on an SD card wouldn't be ideal. The analog source is low frequency and there is a bandpass before the signal reaches the ADC, so even 1KHz sampling rate would be great. The problem is I just need to figure out how to safely transmit the data.

[adafruit_support_bill]

1KHz should be achievable - especially if you transmit the samples in raw binary form. The double-buffering scheme you suggest is a reasonable approach. Since latency is an issue, you want to keep the buffers small - in case a re-transmit is necessary. Something like a 128 byte packet with a 16 bit CRC sounds about right. An alternative approach is to use 'forward error correction', where enough redundant data is transmitted to reconstruct a corrupted packet at the destination.