Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Please tell us which board you are using.

Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.
User avatar
UART_PL
 
Posts: 12
Joined: Tue Jul 09, 2019 1:46 pm

Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by UART_PL »

Hi,

For few days I'm struggling to establish backup my files from SD card to FTP server using Arduino C++ and Metro M4 Airlift.

In my program, I'm using Adafruit forked WiFiNINA library, which supposes to work best with this hardware setup.

Code: Select all

uint8_t clientBuf[1024];
size_t bufferSize = 1024;

uint32_t WriteBuffered(char * ftpfile) {
  
  file.open(ftpfile, O_RDONLY);  
  uint32_t dataLength = file.fileSize();
  Serial.println (dataLength);
  
  uint32_t m;
  while((m = file.read(clientBuf, bufferSize))>0) {
    dclient.write(clientBuf, m);
  }

} 
In this setup I can write to the ftp server only 5,7KB (5744b --> 5x 1024 + 624)
I've tried to debug how the WiFiClient.h, server_drv.h and spi_drv.h are working but I could not find any reason why on the 6th loop data are not transferred fully and no data transferred on next portions of the same file.

Then I understand that communication between M4 and ESP32 via SPI seems to be working fine, so I received real confirmation how much data was transferred to ESP32, but it doesn't mean that ESP32 have already sent it to the FTP server via TCP, am I right?

The question is what is the size of buffer inside ESP32 which is buffering data received by SPI from M4 to be sent to Network?
Is there any function which I can check the status of this buffer inside ESP32?
How to make sure that ESP32 has already sent the data and "cleaned" the buffer before I send another portion of data via SPI?

Does my understanding of how the WiFiNINA works are correct?

User avatar
adafruit2
 
Posts: 21281
Joined: Fri Mar 11, 2005 7:36 pm

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by adafruit2 »

we've transferred some pretty big files so we know its possible, what if you add delays and reduce to 32 byes at a time?

User avatar
UART_PL
 
Posts: 12
Joined: Tue Jul 09, 2019 1:46 pm

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by UART_PL »

Yes, the 32buffer size and some delays improve the situation, but one of the first thing which I've learned when I started programming was to avoid using delays.

Is there any particular reason why 32-bit buffer size is working and any buffer size above starting from 64-bit already causing issues? Please do not take me wrong, I'm going to use my code in different environments including different network connection speed, and I care about stability. I would like to avoid the situation that on some networks defined delay will work and with other networks it will not. Also if for me to find some walk around took me already a few days, I'm sure that I'm not the only one which was struggling to find the right solution for at least few hours. Can we consider adding some guidelines or function definition improvements to make it more user friendly?

Now I'm trying the following solution, which seems to be not so much buffer size-dependent:

Code: Select all

void WriteBuffered(char * ftpfile) {
  
  file.open(ftpfile, O_RDONLY);  
  uint32_t dataLength = file.fileSize();
  Serial.println (dataLength);
 
  uint32_t m;
  size_t n;
  while((m = file.read(clientBuf, bufferSize))>0) {
    n = dclient.write(clientBuf, m);
    while (n != m) {
      delay(100);
      n += dclient.write(&clientBuf[n], m-n-1);
    }
  }

User avatar
adafruit2
 
Posts: 21281
Joined: Fri Mar 11, 2005 7:36 pm

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by adafruit2 »

we're not sure, a lot of the stuff internal to ESP32 is not easily identifiable and is inside the non-open stack, it could be that it just needs smaller chunks of data to avoid errors. you can probably remove the delay and stick to 32 byte buffers.

User avatar
UART_PL
 
Posts: 12
Joined: Tue Jul 09, 2019 1:46 pm

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by UART_PL »

Is there the same behavior also using CircuitPython?

User avatar
adafruit2
 
Posts: 21281
Joined: Fri Mar 11, 2005 7:36 pm

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by adafruit2 »

yes

User avatar
lenshustek
 
Posts: 18
Joined: Mon Nov 10, 2014 1:03 am

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by lenshustek »

I'm having similar problems using the Adafruit Airlift ESP32 WiFI coprocessor board connected to a Teensy 3.5. I'm using it for a web server that needs to reply to requests for embedded images, and it takes fiddling with the maximum buffer length in each client.write() call and playing with delays to get it to work correctly "most" of the time. When it fails, the calls to client.write() return zero, and it prematurely terminates the connection by sending an ACK packet with the FIN flag on. Very disturbing. It's hard to see how this can be used in production. Is ESP32 firmware version 1.2.2, which is what came on the board, still the latest?

Also, I'm struggling now with how to accept two simultaneous TCP connections from a single client, which is what the Chrome browser routinely does. The ESP32 stack accepts the first SYN packet with a proper handshake to start the connection, but ignores the second SYN packet even though it has a different source port. Doing another call to server.available() doesn't help. Does anyone know how to get it to correctly start both connections?

User avatar
adafruit2
 
Posts: 21281
Joined: Fri Mar 11, 2005 7:36 pm

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by adafruit2 »

we haven't used it as a server much - but we do know that it's working as a server in circuitpython.
does web *client* work?

User avatar
lenshustek
 
Posts: 18
Joined: Mon Nov 10, 2014 1:03 am

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by lenshustek »

I haven't written code to try to use it as an HTTP/HTML client. I'm focused on making it a web server that standard browsers (Chrome, Firefox, IE) can talk to.

User avatar
adafruit2
 
Posts: 21281
Joined: Fri Mar 11, 2005 7:36 pm

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by adafruit2 »

ok what if you read/write one byte at a time? also try underclocking the chip - see if that helps

User avatar
lenshustek
 
Posts: 18
Joined: Mon Nov 10, 2014 1:03 am

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by lenshustek »

Well, I've gotten the embedded image file transfer to work most of the time by writing 500 bytes at a time and delaying 10 msec between writes. Clearly there's some bug that we're invoking magic to avoid.

Any suggestions about how to get it to accept two simultaneous connections from the same web browser client?

User avatar
adafruit2
 
Posts: 21281
Joined: Fri Mar 11, 2005 7:36 pm

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by adafruit2 »

there's an issue in the nina-fw that someone brought up that could be related but we haven't had the time to dig into (probably cause we are using circuitpython and its slow enuf)
https://github.com/adafruit/nina-fw/issues/9

for multi-connect we've never done it so no direct advice - do you not get two sockets?

User avatar
User_UMjT7KxnxP8YN8
 
Posts: 322
Joined: Tue Jul 17, 2018 1:28 pm

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by User_UMjT7KxnxP8YN8 »

I recently purchased one of these boards, thinking it would be a nice single-board replacement for the Metro M4 Express/ATWINC1500 shield combo I've been using. But I routinely write 1024 bytes at a time to the ATWINC1500 and have sent it 2048 at a time in the past, so I'm having second thoughts about investing time porting my code to the Airlift unless this crippling throughput issue is corrected.

One of the posts here mentions the opacity of the WiFiNINA firmware; hopefully the problem isn't there.

User avatar
adafruit2
 
Posts: 21281
Joined: Fri Mar 11, 2005 7:36 pm

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by adafruit2 »

the wifinina firmware is *pretty* simple in that it calls directly to the ESP32 API. if you have a simplified reproducable example that you can submit an issue on, we'll be able to test it out! we did do an MP3 streaming example which did chunks of about 32-64 bytes and it can pretty good for a long time. the benefits of the ESP32 is that its less opaque than the WINC1500, less expensive, and is more easily updatable with new featuers!that said, we're still stocking the WINC1500 since some people like it more :)

User avatar
lenshustek
 
Posts: 18
Joined: Mon Nov 10, 2014 1:03 am

Re: Metro M4 Airlift buffer size at ESP32 cooprocesor issue

Post by lenshustek »

I continue to struggle with multiple simultaneous connections. I posted the question and some simplified sample code over at StackExchange:
https://arduino.stackexchange.com/quest ... onnections
Maybe someone will have ideas...

Locked
Please be positive and constructive with your questions and comments.

Return to “Metro and Metro Express, Grand Central”