In this article I'll talk about making  use of FLASH to store information in the ESP8266 wireless units (I refer here to those with 4Meg or more of FLASH – on the assumption that only the first 2 meg is in use plus a tiny bit at the end)…

To clarify - The ESP can make use of 1 meg of FLASH for program space, plus another meg if using OTA (online updating) - that is-  the second meg is used to upload a new program while the original is still running.  In addition Espressif use a tiny bit at the very top of whatever size FLASH you have. I'm going to talk here about those units with 4MB of FLASH - for example the very popular and cheap ESP-12 and variants.  If you had a unit with MORE memory then this could be expanded. LESS and this isn't worth reading.

Espressif, in order to arrange for secure user data – use three 4k blocks…

The first two contain data up to 4096 bytes, the third says which block is the current one in use – accordingly that block is mostly wasted.

So when you want to update the data – you see which one or the two is NOT current – wipe it… and then update it, then update the third block with the pointer.

So you use 3, 4k blocks to get one good block of data. And I have to say it works, 100% – I’ve never lost data. Should the write procedure fail part way through due to power cut or whatever, all you lose is the latest update. The current block remains the previous one you were using.

In the SDK there is a routine to use this - I recall TUANPM had his own but then when Espressif incorporated this into the SDK I moved to their routines. They work perfectly.

The only issue with this is the waste of a block which if taken to extreme – using other spare blocks of FLASH would be quite wasteful.  I thought of the various existing file systems out there and figured they were overkill for my needs and not necessarily as secure as this.

So I had this idea to take TWO 4k blocks and use the top 32-bits of each as a counter.

So to read a block – look to see which of the  two has the highest number that is NOT FFFFFFFF  - that is the block to read.

When writing, to whichever is NOT the current block – write the data including an incremented count (avoiding 0 and FFFFFFFF). As the count is the last thing to be written, any failure due to power loss will surely result in that 32-bit number NOT being updated – OR being reset to FFFFFFFF. In which case we stick with the current block and hence only the latest update is lost – just like the Espressif version but only using 2 blocks.

Reader Gary came in with a slightly better idea – to write the block – whatever that may be with the first 4 bytes set to FF.  (you must write on 4-byte boundaries to avoid the ESP having a fit).  Once the sector is written you can then go back and fill those 4 bytes in – because overwriting with 0 is always ok – what you can’t do is over-write with 1 as that is the function of the erase-block routine (4k).  I’ve now tested this –checking adjacent 32-bit words – it all works a treat.

The next stage was to formulate this into something tangible…

To read into a structure in RAM

Check both FLASH blocks first 4 bytes.. if both FF – current block is first else if one block is FF – current block is other block else current block is highest.

If not new, Read struct size in from relevant offset into block. If new, fill struct with FF !! No point in reading all.

To write a structure to FLASH

Check both FLASH blocks first 4 bytes.. if both FF – this is NEW. Else if either is FF current block is other – else current block is highest.

If NEW –  ERASE first block, write data into relevant part of FLASH – write number 1 into bottom 32 bits. Check – report any failure.

Otherwise read current block into 4K RAM buffer. FFFFFFFF into bottom 4 bytes. Erase OTHER block, write  RAM structure with updates to new block. Take original counter – increment – write to bottom of new block. Check, report any failure.

Apart from that double read-write which might be small  depending on your structure, this seems reasonably efficient.

And the result…

my_flash_read(sector, offset,buffer,buffer_size)

and

my_flash_write(sector,offset,buffer,buffer_size)

Assumptions for the above being that the sector would in fact be blocks of two sectors – so from the start of the third Meg to nearly the end of a 4 Meg FLASH (ESP-12 for example) you’d be able to use sectors 0-253 of protected data. The offset would be the offset from the start of the sector where your data is to go, the buffer the address of your buffer and the buffer_size to be no more than 4092 bytes. Ok, so any block can only be written to, maybe 100,000 times  - but if you wanted to – you could keep track of that.

For those interested the working test code is here – not the most concise – but it works. Given it a good hammering this morning – yet to decide how best to use this – but it’s a great move forward from thinking of that extra space as a great black hole. I've just put in a second version of the WRITE routine that does NOT use a 4K buffer - initial testing suggests it is working fine.. mind you - the tests are small.

[pcsh lang="cpp" tab_size="4" message="" hl_lines="" provider="manual"]

// Some assumptions-  your offset is on a 4-byte boundary...(size doesn't have to be) - your sector is 0-253 (for the top 2 meg)
// and your offset + buffer does not go over the 4096 (remember - first 4 bytes used up.

uint8_t IFA my_flash_read(uint32_t fsec,uint32_t foff, uint32_t *fbuf,uint32_t fsiz)
{ //READ
uint32_t first;
uint32_t second;
uint32_t current;
uint32_t structsize;
uint32_t flashbase;
uint32_t flashoffs;
uint32_t tmp;
uint8_t good;

if (fsec>253) { return 0; } // assuming starting at third meg and can't touch very top 4 secs (2 for each of these)
uint32_t flashStart=(fsec*2)+0x200000;

// size of struct must be rounded up - offset from base=arg1, ram buffer = arg2,
// size=arg3
// when writing if length not 32-bit rounded, will write to nearest boundary up

flashbase=foff&0xfffff000;
flashoffs=foff&0xfff;

structsize=fsiz; if (structsize&3) structsize+=(4-(structsize&3));
current=4096;
spi_flash_read(flashStart+flashbase,&first,4);
spi_flash_read(flashStart+flashbase+current,&second,4);

if ((first==0xffffffff) && (second==0xffffffff)) // ie all new
	{
	good=0;
	spi_flash_erase_sector(0x200+(flashbase/4096));
	tmp=1;
	spi_flash_write(flashbase+flashStart,&tmp,4);
	spi_flash_read(flashbase+flashStart,&tmp,4);
    current=0;
	if (tmp==1) good=1; else good=0;
	}
else if (first==0xffffffff) current=1;
else if (second==0xffffffff) current=0;
else if (second>first) current=1;
else current=0;

// can't read whole in once go if struct not on 4 byte boundary

current*=4096;
if (structsize==fsiz)
	{
    spi_flash_read(flashoffs+flashStart+4+current,&first,4);
	if (first!=0xffffffff) good=1; else good=0;
	spi_flash_read(flashoffs+flashStart+4+current,fbuf,fsiz);
	}
	else
	{
    spi_flash_read(flashoffs+flashStart+4+current,&first,4);
	if (first!=0xffffffff) good=1; else good=0;
	spi_flash_read(flashoffs+flashStart+4+current,fbuf,(fsiz&0xfffffffc));
	spi_flash_read(flashoffs+flashStart+4+current+(fsiz&0xfffffffc),&tmp,4);    //// CHECK
	memcpy(fbuf+(fsiz&0xfffffffc),&tmp,(fsiz&3)); // move those last 1,3 or 3 bytes
	}

return good;  // so you know if it is the first time or not.
} // done with READ operation - phew!!


// and now to tackle writing - same rules as reading - this version uses a 4k buffer

uint8_t IFA my_flash_write(uint32_t fsec,uint32_t foff, uint32_t *fbuf,uint32_t fsiz)
{ //WRITE
uint32_t first;
uint32_t second;
uint32_t current;
uint32_t structsize;
uint32_t counter;
uint32_t tmp;
uint32_t flashbase;
uint32_t flashoffs;
uint8_t good;
uint8_t bigbuf[4096];

if (fsec>253) { return 0; } // assuming starting at third meg and can't touch very top 4 secs (2 for each of these)
uint32_t flashStart=(fsec*2)+0x200000;

flashbase=foff&0xfffff000;
flashoffs=foff&0xfff;

// size of struct must be rounded up - offset from base=arg1, ram buffer = arg2,
// size=arg3
// when writing if length not 32-bit rounded, will write to nearest boundary up

structsize=fsiz; if (structsize&3) structsize+=(4-(structsize&3));
current=4096;
spi_flash_read(flashbase+flashStart,&first,4);
spi_flash_read(flashbase+flashStart+current,&second,4);

if ((first==0xffffffff) && (second==0xffffffff)) current=0;
else if (first==0xffffffff) current=1;
else if (second==0xffffffff) current=0;
else if (second>first) current=1;
else current=0;
	{
	good=0;

	spi_flash_erase_sector(0x200+(flashbase/4096)+(current^1)); // erase the OTHER one
	current *=4096;
	spi_flash_read(flashbase+flashStart+current,bigbuf,4096);
    memcpy(&counter,bigbuf,4); // copy counter
	memset(bigbuf,0xff,4);
	memcpy(bigbuf+4+flashoffs,fbuf,fsiz); // write the new data into the buffer
	if (current) current=0; else current=4096;
	spi_flash_write(flashbase+flashStart+current,bigbuf,4096);
	if (counter==0xffffffff) counter++;
    counter++;
	memcpy(bigbuf,&counter,4); // copy counter back
	spi_flash_write(flashbase+flashStart+current,bigbuf,4);
	spi_flash_read(flashbase+flashStart+current,&tmp,4);
    if (tmp==counter) good=1;
	}
return good;  // so you know if it is the first time or not.
} // done with WRITE operation 

// this version of writing does NOT use a 4k buffer - seems to work just as well.

uint8_t IFA my_flash_write_no_buffer(uint32_t fsec,uint32_t foff, uint32_t *fbuf,uint32_t fsiz)
{ //WRITE
uint32_t first;
uint32_t second;
uint32_t current;
uint32_t newcurrent;
uint32_t structsize;
uint32_t counter;
uint32_t tmp;
uint32_t flashbase;
uint32_t flashoffs;
uint8_t good;

if (fsec>253) { return 0; } // assuming starting at third meg and can't touch very top 4 secs (2 for each of these)
uint32_t flashStart=(fsec*2)+0x200000;

flashbase=foff&0xfffff000;
flashoffs=foff&0xfff;

// size of struct must be rounded up - offset from base=arg1, ram buffer = arg2,
// size=arg3
// when writing if length not 32-bit rounded, will write to nearest boundary up

structsize=fsiz; if (structsize&3) structsize+=(4-(structsize&3));
current=4096;
spi_flash_read(flashbase+flashStart,&first,4);
spi_flash_read(flashbase+flashStart+current,&second,4);

if ((first==0xffffffff) && (second==0xffffffff)) current=0;
else if (first==0xffffffff) current=1;
else if (second==0xffffffff) current=0;
else if (second>first) current=1;
else current=0;
	{
	good=0;
	spi_flash_erase_sector(0x200+(flashbase/4096)+(current^1)); // erase the OTHER one
	current *=4096;
	if (current) newcurrent=0; else newcurrent=4096;
	if (current) counter=second; else counter=first; if (counter==0xffffffff) counter++;

	tmp=0xffffffff;
	spi_flash_write(flashbase+flashStart+newcurrent,&tmp,4);	// write a blank counter

	uint32_t tstart,tstart2,tend;
	tstart=flashbase+flashStart+current+4;
	tstart2=flashbase+flashStart+newcurrent+4;

	tend=flashbase+flashStart+current+4+flashoffs;

	while (tstart<tend)
		{
			spi_flash_read(tstart,&tmp,4);
			spi_flash_write(tstart2,&tmp,4);
			tstart+=4; tstart2+=4;
		}

	spi_flash_write(tstart2,fbuf,structsize);
	tstart+=structsize; tstart2+=structsize;

	while (tstart<4096)
		{
			spi_flash_read(tstart,&tmp,4);
			spi_flash_write(tstart2,&tmp,4);
			tstart+=4; tstart2+=4;
		}


    counter++;
	spi_flash_write(flashbase+flashStart+newcurrent,&counter,4);
	spi_flash_read(flashbase+flashStart+newcurrent,&tmp,4);
    if (tmp==counter) good=1;
	}
return good;  // so you know if it is the first time or not.
} // done with WRITE operation

[/pcsh]