Page 1 of 2

Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Sun Aug 11, 2019 1:35 pm
by UART_PL
Hi,

For few days I'm struggling to establish backup my files from SD card to FTP server using Arduino C++ and Metro M4 Airlift.

In my program, I'm using Adafruit forked WiFiNINA library, which supposes to work best with this hardware setup.

Code: Select all

uint8_t clientBuf[1024];
size_t bufferSize = 1024;

uint32_t WriteBuffered(char * ftpfile) {
  
  file.open(ftpfile, O_RDONLY);  
  uint32_t dataLength = file.fileSize();
  Serial.println (dataLength);
  
  uint32_t m;
  while((m = file.read(clientBuf, bufferSize))>0) {
    dclient.write(clientBuf, m);
  }

} 
In this setup I can write to the ftp server only 5,7KB (5744b --> 5x 1024 + 624)
I've tried to debug how the WiFiClient.h, server_drv.h and spi_drv.h are working but I could not find any reason why on the 6th loop data are not transferred fully and no data transferred on next portions of the same file.

Then I understand that communication between M4 and ESP32 via SPI seems to be working fine, so I received real confirmation how much data was transferred to ESP32, but it doesn't mean that ESP32 have already sent it to the FTP server via TCP, am I right?

The question is what is the size of buffer inside ESP32 which is buffering data received by SPI from M4 to be sent to Network?
Is there any function which I can check the status of this buffer inside ESP32?
How to make sure that ESP32 has already sent the data and "cleaned" the buffer before I send another portion of data via SPI?

Does my understanding of how the WiFiNINA works are correct?

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Mon Aug 12, 2019 2:52 pm
by adafruit2
we've transferred some pretty big files so we know its possible, what if you add delays and reduce to 32 byes at a time?

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Mon Aug 12, 2019 3:44 pm
by UART_PL
Yes, the 32buffer size and some delays improve the situation, but one of the first thing which I've learned when I started programming was to avoid using delays.

Is there any particular reason why 32-bit buffer size is working and any buffer size above starting from 64-bit already causing issues? Please do not take me wrong, I'm going to use my code in different environments including different network connection speed, and I care about stability. I would like to avoid the situation that on some networks defined delay will work and with other networks it will not. Also if for me to find some walk around took me already a few days, I'm sure that I'm not the only one which was struggling to find the right solution for at least few hours. Can we consider adding some guidelines or function definition improvements to make it more user friendly?

Now I'm trying the following solution, which seems to be not so much buffer size-dependent:

Code: Select all

void WriteBuffered(char * ftpfile) {
  
  file.open(ftpfile, O_RDONLY);  
  uint32_t dataLength = file.fileSize();
  Serial.println (dataLength);
 
  uint32_t m;
  size_t n;
  while((m = file.read(clientBuf, bufferSize))>0) {
    n = dclient.write(clientBuf, m);
    while (n != m) {
      delay(100);
      n += dclient.write(&clientBuf[n], m-n-1);
    }
  }

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Mon Aug 12, 2019 3:46 pm
by adafruit2
we're not sure, a lot of the stuff internal to ESP32 is not easily identifiable and is inside the non-open stack, it could be that it just needs smaller chunks of data to avoid errors. you can probably remove the delay and stick to 32 byte buffers.

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Tue Aug 13, 2019 3:10 am
by UART_PL
Is there the same behavior also using CircuitPython?

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Tue Aug 13, 2019 1:15 pm
by adafruit2
yes

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Thu Aug 22, 2019 4:29 pm
by lenshustek
I'm having similar problems using the Adafruit Airlift ESP32 WiFI coprocessor board connected to a Teensy 3.5. I'm using it for a web server that needs to reply to requests for embedded images, and it takes fiddling with the maximum buffer length in each client.write() call and playing with delays to get it to work correctly "most" of the time. When it fails, the calls to client.write() return zero, and it prematurely terminates the connection by sending an ACK packet with the FIN flag on. Very disturbing. It's hard to see how this can be used in production. Is ESP32 firmware version 1.2.2, which is what came on the board, still the latest?

Also, I'm struggling now with how to accept two simultaneous TCP connections from a single client, which is what the Chrome browser routinely does. The ESP32 stack accepts the first SYN packet with a proper handshake to start the connection, but ignores the second SYN packet even though it has a different source port. Doing another call to server.available() doesn't help. Does anyone know how to get it to correctly start both connections?

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Thu Aug 22, 2019 6:32 pm
by adafruit2
we haven't used it as a server much - but we do know that it's working as a server in circuitpython.
does web *client* work?

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Thu Aug 22, 2019 10:13 pm
by lenshustek
I haven't written code to try to use it as an HTTP/HTML client. I'm focused on making it a web server that standard browsers (Chrome, Firefox, IE) can talk to.

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Thu Aug 22, 2019 10:38 pm
by adafruit2
ok what if you read/write one byte at a time? also try underclocking the chip - see if that helps

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Thu Aug 22, 2019 11:54 pm
by lenshustek
Well, I've gotten the embedded image file transfer to work most of the time by writing 500 bytes at a time and delaying 10 msec between writes. Clearly there's some bug that we're invoking magic to avoid.

Any suggestions about how to get it to accept two simultaneous connections from the same web browser client?

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Fri Aug 23, 2019 12:04 am
by adafruit2
there's an issue in the nina-fw that someone brought up that could be related but we haven't had the time to dig into (probably cause we are using circuitpython and its slow enuf)
https://github.com/adafruit/nina-fw/issues/9

for multi-connect we've never done it so no direct advice - do you not get two sockets?

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Sat Aug 24, 2019 12:48 pm
by User_UMjT7KxnxP8YN8
I recently purchased one of these boards, thinking it would be a nice single-board replacement for the Metro M4 Express/ATWINC1500 shield combo I've been using. But I routinely write 1024 bytes at a time to the ATWINC1500 and have sent it 2048 at a time in the past, so I'm having second thoughts about investing time porting my code to the Airlift unless this crippling throughput issue is corrected.

One of the posts here mentions the opacity of the WiFiNINA firmware; hopefully the problem isn't there.

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Sun Aug 25, 2019 1:46 pm
by adafruit2
the wifinina firmware is *pretty* simple in that it calls directly to the ESP32 API. if you have a simplified reproducable example that you can submit an issue on, we'll be able to test it out! we did do an MP3 streaming example which did chunks of about 32-64 bytes and it can pretty good for a long time. the benefits of the ESP32 is that its less opaque than the WINC1500, less expensive, and is more easily updatable with new featuers!that said, we're still stocking the WINC1500 since some people like it more :)

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Posted: Tue Aug 27, 2019 3:02 pm
by lenshustek
I continue to struggle with multiple simultaneous connections. I posted the question and some simplified sample code over at StackExchange:
https://arduino.stackexchange.com/quest ... onnections
Maybe someone will have ideas...