In this article I'll talk about making use of FLASH to store information in the ESP8266 wireless units (I refer here to those with 4Meg or more of FLASH – on the assumption that only the first 2 meg is in use plus a tiny bit at the end)…
To clarify - The ESP can make use of 1 meg of FLASH for program space, plus another meg if using OTA (online updating) - that is- the second meg is used to upload a new program while the original is still running. In addition Espressif use a tiny bit at the very top of whatever size FLASH you have. I'm going to talk here about those units with 4MB of FLASH - for example the very popular and cheap ESP-12 and variants. If you had a unit with MORE memory then this could be expanded. LESS and this isn't worth reading.
Espressif, in order to arrange for secure user data – use three 4k blocks…
The first two contain data up to 4096 bytes, the third says which block is the current one in use – accordingly that block is mostly wasted.
So when you want to update the data – you see which one or the two is NOT current – wipe it… and then update it, then update the third block with the pointer.
So you use 3, 4k blocks to get one good block of data. And I have to say it works, 100% – I’ve never lost data. Should the write procedure fail part way through due to power cut or whatever, all you lose is the latest update. The current block remains the previous one you were using.
In the SDK there is a routine to use this - I recall TUANPM had his own but then when Espressif incorporated this into the SDK I moved to their routines. They work perfectly.
The only issue with this is the waste of a block which if taken to extreme – using other spare blocks of FLASH would be quite wasteful. I thought of the various existing file systems out there and figured they were overkill for my needs and not necessarily as secure as this.
So I had this idea to take TWO 4k blocks and use the top 32-bits of each as a counter.
So to read a block – look to see which of the two has the highest number that is NOT FFFFFFFF - that is the block to read.
When writing, to whichever is NOT the current block – write the data including an incremented count (avoiding 0 and FFFFFFFF). As the count is the last thing to be written, any failure due to power loss will surely result in that 32-bit number NOT being updated – OR being reset to FFFFFFFF. In which case we stick with the current block and hence only the latest update is lost – just like the Espressif version but only using 2 blocks.
Reader Gary came in with a slightly better idea – to write the block – whatever that may be with the first 4 bytes set to FF. (you must write on 4-byte boundaries to avoid the ESP having a fit). Once the sector is written you can then go back and fill those 4 bytes in – because overwriting with 0 is always ok – what you can’t do is over-write with 1 as that is the function of the erase-block routine (4k). I’ve now tested this –checking adjacent 32-bit words – it all works a treat.
The next stage was to formulate this into something tangible…
To read into a structure in RAM
Check both FLASH blocks first 4 bytes.. if both FF – current block is first else if one block is FF – current block is other block else current block is highest.
If not new, Read struct size in from relevant offset into block. If new, fill struct with FF !! No point in reading all.
To write a structure to FLASH
Check both FLASH blocks first 4 bytes.. if both FF – this is NEW. Else if either is FF current block is other – else current block is highest.
If NEW – ERASE first block, write data into relevant part of FLASH – write number 1 into bottom 32 bits. Check – report any failure.
Otherwise read current block into 4K RAM buffer. FFFFFFFF into bottom 4 bytes. Erase OTHER block, write RAM structure with updates to new block. Take original counter – increment – write to bottom of new block. Check, report any failure.
Apart from that double read-write which might be small depending on your structure, this seems reasonably efficient.
And the result…
Assumptions for the above being that the sector would in fact be blocks of two sectors – so from the start of the third Meg to nearly the end of a 4 Meg FLASH (ESP-12 for example) you’d be able to use sectors 0-253 of protected data. The offset would be the offset from the start of the sector where your data is to go, the buffer the address of your buffer and the buffer_size to be no more than 4092 bytes. Ok, so any block can only be written to, maybe 100,000 times - but if you wanted to – you could keep track of that.
For those interested the working test code is here – not the most concise – but it works. Given it a good hammering this morning – yet to decide how best to use this – but it’s a great move forward from thinking of that extra space as a great black hole. I've just put in a second version of the WRITE routine that does NOT use a 4K buffer - initial testing suggests it is working fine.. mind you - the tests are small.