The above did not change the issue of delayed writing to the flash memory.
supervisor.reload() is what the RP2040 uses for a soft reboot (softreset). I will implement that if we don't have a 'eureka moment'.
supervisor.reload will only provide current weather information if I soft reboot the hardware every 5 minutes or so. That does not look to me to be an elegant solution.
I am confused as to what updating the bootloader of the AirLift will do toward a solution.
I am working with a Feather RP2040 with AirLift WiFi also a Nano RP2040 Connect, both RP2040s with SD Card storage. From what I understand they are functionally equivalent for what I need. I want to publish 30 weather web pages that are each updated once every 5 minutes on 20 second intervals. e.g. Update page 1, wait 20 seconds, update page 2, wait 20 seconds.
I have an option I can select in the boot.py file to switch the RP2040 permissions to 'write' between the SD Card or the flash memory, but not both. When the SD Card option is selected, it is mounted for read/write.
When the RP2040 can write to the SD Card, the data is not available as a web page. Web pages on the flash memory are available but the flash memory can't be written to.
When the RP2040 can write to the flash memory, the SD Card is unavailable to the RP2040 for write. I have not tested read because if I can't write updated data why would I want to read from it.
When the RP2040 can write to the flash memory the data is available to the web page BUT it is only updated to the web viewer every 9 hours or after a reboot. The reboot can be a power reboot, a RESET button reboot or supervisor.reload() reboot. None of which are optimal solutions. The supervisor.reload() option is least bad option.
My preference would be to write the web pages to the SD Card for posting to the web. The web client would have access to that data only. That would eliminate the need for the boot.py routines, and protect the flash memory from corruption by unexpected events from code.py. The first index.html web page can reside on the flash memory, pointing to all the other real time web pages on the SD Card. I don't update the first web page real time, it is static.
I hope I have done a good job of describing what I would like to do and some of the issues.
I did a test to see if supervisor.reload() would be a big issue. supervisor.reload() does not cause the most recent data to be displayed after the soft reboot.
I did a test where I deleted all the web pages and booted the system. I waited for three loops of 30 web pages per loop to be "written" with a supervisor.reload() at the end of each loop of 30 web pages. The web pages were not showing on the flash memory. I pushed the RESET button and like magic they were there.
With that information, I think the data displayed on the web pages is pulled into a buffer area at power on or after a RESET button push or a ~9 hour wait. That buffered data is what is displayed to the web client. The data created every 20 seconds is written to the flash memory as it is created. The buffer is not updated with the new data.
A power reboot or RESET button reboot is not practical every 5 to 6 minutes. The 8 to 9 hour data refresh shows that there is something that can cause the data in the flash memory to be updated to the buffer location. We just need to find out what that magic command is.
Thank you for the excellent summary and congratulations on the small wins this weekend.
I think you are right that there is something that can cause flash to write out to SD.
Two thoughts occurred to me while reading your updates.
1) Are you toggling the SPI CS pin HIGH for the device not in use. EG. after web download turning the AirLift CS HIGH (disable) and during SD write turning SD CS pin low (enable)? Maybe that would explain the conflict of downloading data from web and trying to write to the SD card if they are sharing the same SPI port.
2) Is the web server code itself causing any sort of buffering. EG. if you print the index.html contents over the serial console are the files being updated?
One final thought. Is it possible to buffer all your webpage downloads in memory then write them out at once to the SD card to only have to do a single microcontroller.reset()?
1) I do nothing to/with the SPI CS line. It is totally controlled by imports of storage, digitalio, adafruit_sdcard, and adafruit_esp32spi.
2) I do nothing to buffer data. All web calls are handled by imports of adafrui_requests, os, server, and server.WSGIServer.
I may have confused you. code.py writes the data to the flash memory. The web server shows data from the suspected buffer memory. The web pages need to be updated every from flash memory to buffer memory so we can get the weather information on our iPad, iPhone, or PC any time we want it.
I am writing all 30 web pages to the flash memory as they are created over a 9 to 10 minute period. When the RP2040 is reboot by microcontroller.reset(), every 30 pages,the web pages are copied to the buffer memory. code.py then shows the flash memory copies of the web pages to the client.
Rebooting the RP2040 every 30 web pages/9.5 minutes is not a optimal solution. Real-time access to the web pages without rebooting is my goal. Without microcontroller.reset() the buffer memory is updated every 8 to 9 hours from the data in the flash memory, therefore there must be some magic code that makes that happen on command I can put in code.py.
Thank you for stating your situation yet another way.
I was looking into any magic happening with the soft microcontroller.reset. This thread is a nice summary of what is going on with flash and buffer states during a soft reset, but ultimate does not point to a solution.
Stupid question for you. What model of SD card are you using?
Did you read about some of the patch that was done to the adafruit_sdcard CircuitPython library around large writes and 32GB Samsung EVO cards? Some of the instrumentation used for debugging might be helpful if you are not experiencing the same issue.
NOTE: The SD Card is not in playing a role when code.py is writing map data to the flash memory or sending web data from the buffer memory.
I am using a Samsung EVO Select, 32GB SD card. Several times over the past trials and tribulations I have pulled the SD Card (power removed from Feather) and put it in my Windows machine and run diagnostics. Each time "No Errors".
The 'index.html' is static in size at 5,558 bytes. It is not written during operation and does not change.
The Open Weather Map HTML files are only 367 bytes to 391 bytes in size. Cluster size is 1kb, so no very large file is written that would cause a write buffer overflow. But each small HTML file looks to be immediately written to the SD Card, so buffer size or overflow should not be an issue.
These are the only files on the SD Card.
For the flash memory. ******************
The 'index.html' is static in size at 5,558 bytes. It is not written during operation and does not change.
The Open Weather Map HTML files are only 367 bytes to 391 bytes in size. Each small HTML file looks to be immediately written to the flash memory, so buffer size or overflow should not be an issue.
These are the only files in the flash memory in the subdirectory accessed for web pages.
********************************
I looked at the references you found. They did not look to address my issue. They were interesting and I did learn stuff. Thanks for the deep dive.
I've enjoyed working on this problem with you, but I believe it is now resolved. You had the right idea all along. It just took me 20 minutes to find what you were asking for. There is a filesystem_flush() funciton being called from by microcontroller.reset just before the system softboots with the reset_cpu().
I agree with you we are very close to a solution because of your searches.
I downloaded the file your message pointed to - adafruit-circuitpython-bundle-py-20210928.zip.
I was not able to find circuitpython/ports/nrf/common-hal/microcontroller/__init__.c.
I tried filesystem_flush() and got the error - NameError: name 'filesystem_flush' is not defined Not an unexpected error.
I then tried file.filesystem_flush() and got the error - AttributeError: 'TextIOWrapper' object has no attribute 'filesystem_flush'
I may well have missed the mark on what you wanted me to try. Can you give me a step-by-step of what I should do?
I have a file.flush() statement in my code.py that is not causing the result needed.
I don't think we are having a problem flushing the files to the flash memory. We have not found out why and how the OS causes the files in flash memory to be transferred to the buffer memory that is displayed through the web calls. Would it help if I uploaded my code.py and boot.py? ~ 750 lines. Its only requirements in flash memory mode is a Feather RP2040 and an AirLift.
void filesystem_flush(void) {
// Reset interval before next flush.
filesystem_flush_interval_ms = CIRCUITPY_FILESYSTEM_FLUSH_INTERVAL_MS;
supervisor_flash_flush();
// Don't keep caches because this is called when starting or stopping the VM.
supervisor_flash_release_cache();
}
Does that meal I should do/try os.sync(filesystem_flush())? I fear disaster. ;-)
When I add os.sync to my code.py it does not crash. But it does not update the web pages.
The more I think about it, the web buffer memory is updated at power on boot, RESET button release boot, and the magic 9 hour thing. It is not updated at power down, or RESET push. There is no warning to those functions to for to update the web buffer memory. So something at power on, RESET release, or 9 hour thing that copies flash memory to buffer memory.
As a WAG the 9 hour thing may be triggered by an unknown number of bytes written, a time thing for a buffer update. 9 hours is 32,400 seconds (32,768 seconds???). We update a ~400 byte file every ~20 seconds in that 9 hours. That would be a total of about 648,000 bytes. That number does not correspond to anything I can correlate it to. So what happens every 32,768 seconds?
FYI Only - I know it is common on older Windows machines with slow BIOS ROMs to copy the BIOS into RAM for faster execution. This may be happening in the RP2030. Flash memory exterior to the RP2040 almost has to be slower than the internal RP2040 RAM. The RP2040 has 264kB on-chip SRAM in six independent banks (44kB each bank). I would bet your bank account (pun intended) that the buffer memory we have been chasing is in there some place. This is probably a Red Herring, but it is probably "why" what is happening, is happening. It does not give us a "how" we fix the issue.
It seems we have run out of options in terms of google search troubelshooting.
Do you want to write up a short summary of what you have learned and open an issue with the CircuitPython issue. Ideally you would have a tiny piece of example code that shows a buffer not being written to flash until microcontroller.reset() is called.
This way the the circuitpython experts can chime in on whether this is a shutdown or boot up process that is moving the dumping the buffer and maybe how to do it without the reset.