Accessing ESP8266 Arrays in FLASH

23 thoughts on “Accessing ESP8266 Arrays in FLASH”

1. Hi Pete,

   Today I was experimenting with this, trying to get all my user interface messages (a lot!) into rodata, so they wouldn’t occupy dram. I’ve always wanted to do this but was hasitant because of the alignment restrictions, it would get quite complex.

   The first thing I did was see what section read-only strings (const char *string) end up in. That appears to be rodata section! Not the bss section like I expected (that is default behaviour for C-compilers because some very old very naughty code assumes that pointers to characters are always writeable!). Now I must say I append about 100 extra options to cc, to drop all and all legacy compatibility, so I guess for my setup that was to be expected, good.

   Then I would make a test string of 8k and see what happens, const char test[8192]. Not static like I do usually, don’t give the compiler a reason to optimise it away. No referencing though. Result: no heap size decrease. At all. Added a reference. No change. Only when I added an explicit write to the string, the heap size would decrease by 8k. As soon as I removed the write, the heap size was back to normal.

   I guess this means that if the compiler deems a variable to be read-only (you can help it make the decision by using const, but that’s not even required!) it is NOT copied to dram. This also means the esp actually can read bytes from random locations in the flash, no aligment necessary (maybe for 16 or 32 bit arrays, but that’s not my main concern).

   I don’t know if this is a recent “accomplishment” of the ESP SDK or the compiler/linker suite, or if this has been working for ages, but I must say I am quite surprised. It also explains why my dram hasn’t filled up with the lots and lots of user interface messages in my code.

   So at least for character arrays, the compiler/linker/sdk do the “right thing” already fully automatically!

   This is quite different from ATmel microcontrollers, with it’s Harvard architecture, where a program can never read (by the usual means) from program memory, so all strings, all data, everything must be copied to ram on startup (or read with special instructions).

   Happily surprised 🙂

2. Hello,
   I want to store the data using array(max size) and post to internet.How to do it any idea.

3. You are such a gentleman Richard – right – I’m having that. GUYS – take a look at Richard’s blog… nice simple solution while retaining the array format..

4. Wrote this last week after you posted your issue, but forgot to hit publish, doh. So here it is anyway, slightly later than is useful…

   Accessing byte data stored on flash

5. Ken – you are 100% right – thank you for that – just me not paying atttention – so one step forward, but, sadly 2 steps back. I was indeed me missing a parameter off… SO I tried 16 bit tables – they work – 8 bit tables – they work – I tried reading an array subscript from the 8-bit table – and reading one 3 bytes back – to ensure I could read off-boundary – no problems at all.

   Sadly, when I put the attribute onto the 8-bit table I use for PWM lookup – the processor crashed again – and I don’t understand why as it is only 3 lookups every 20ms.

   More work to do….

   1. It’s always good to have many eyes on a problem.

      If you’re still stuck on the PWM lookup issue, feel free to post again, and several readers will attempt to beat me to finding a solution 😉

      Regarding the message filtering, please confirm the following: you’re generating messages in your code with a few predefined prefixes (eg critical, error, warning, info, debug), and then you want to conditionally output them to serial according to your filter setting. Is this correct?

      That’s one way to approach it. Another way is to surround each message statement with a define test, such as:

      #ifdef __MSG_WARNING
      [print the warning message]
      #endif

      … and before a compile, just define the message types you want output.

      This second way may seem like more work at the outset but I see two advantages:
      – for the release version, with minimal messaging, the bin will be smaller
      – you only have print messages when you want them, and you’re not running an extra process on every single message.

      1. Hi Ken

         All is revealed. The lookups work great. However, you simply cannot lookup 16 bit or 8 bit values at speed.. it causes a reboot. So – I took my 16 bit PWM table and my 8 bit brilliance table and made them 32 bits – and lo – they work perfectly out of FLASH. The perfectionist might create a function to get the relevant 1 of 4 bytes out – but that then makes the table a little confusing – OR you could do what I’ve done and say… to hel with it, since employing the bootloader I have LOADS of Flash and little RAM – so it’s a no-brainer really.

         All working perfectly and I’ve even had time to redesign my LED CLOCK, code is in the repository – and THAT is working a treat. I have a 60 LED array, I have a led flying right through the array every 60th of a second in one colour, a blue second hand, green minute hand and the hour hand is 5 LEDs wide, it looks pretty good on a 60-LED strip – it’s going to look stunning in a round infinity mirror!! Just a slight matter of getting 2 round pieces of glass and some means of making a border around it to handle the LEDS that doesn’t look like amateur-night-out. That is always the killer. So today’s been a good day! I put a short video of the clock software running here. https://www.youtube.com/watch?v=WJK2KpnsmLk

         1. Hi Peter
            In your last paragraph of this post you say ‘code is in the repository’ where would this be? I’ve searched quite a few of your other blog posts and cannot find a link.
            It would be nice if you would link this in the actual post.

            1. Home control 2015 – it’s in there

6. sorry, correction to last line for hex output:
   ets_uart_printf(“Pete’s pointer: 0x%x\r\n”, petes);// address of first element of array

7. So hey, this kept eating at me, and my gude wyffe turned in early, so I thought I’d leave the comforts of my preferred C++ framework to fire up the C examples.

   Once I remembered how to build them, I went into the user_main.c of the ‘Hello World’, and I added your array declaration like so, and started to play with print statements…

   static const uint32_t petes[] ICACHE_RODATA_ATTR = {
   55,45,66,77,88,12,76,23,45,67,34,65,25,74,234,67,3,643,646,
   55,45,66,77,88,12,76,23,45,67,34,65,25,74,234,67,3,643,646,
   55,45,66,77,88,12,76,23,45,67,34,65,25,74,234,67,3,643,646};

   LOCAL void ICACHE_FLASH_ATTR hello_cb(void *arg)
   {
   ets_uart_printf(“Hello World!\r\n”);

   ets_uart_printf(“Pete’s test [0]: %d\r\n”, petes[0]);
   ets_uart_printf(“Pete’s test [37]: %d\r\n”, petes[37]);
   ets_uart_printf(“Pete’s pointer: %d\r\n”, petes); // address of first element of array
   }

   It compiled, and I got as output:

   Hello World!
   Pete’s test [0]: 55
   Pete’s test [37]: 646
   Pete’s pointer: 0x40264b54

   So… hope it works for you as well.

   1. WOAH!!!! You’re onto something here, Ken – unbelievably – this works! Well done – but…

      So what I wanted was a simple bit of code to allow me to hide boring system messages as debug – with various levels. Here’s the code.

      void ICACHE_FLASH_ATTR iprintf(uint16_t debug_type, char *fmt, … ){
      char buf[128]; // resulting string limited to 127 chars – change it if you like
      va_list args;
      va_start (args, fmt);
      ets_vsnprintf(buf, sizeof(buf), fmt, args);
      va_end (args);
      if (debug_type & enable_debug_messages) uart0_tx_buffer(buf,os_strlen(buf));
      }

      And so as you see, I build up the string into a buffer and IF the first parameter is correct, I print out the result serially.. so things like MQTT debug messages can be hidden unless I really want them. This works PERFECTLY – or so I thought until now. So what you have shown is that it is NOT the array definition that is in error… (you’ve also surprised me in that %d seem to be handling 32 bit vars, I was expecting it to only handle 16 bit vars, that in itself is worthwhile) but something else. it isn’t the “uart0_tx_buffer command but something else. I tried your direct use of ets_uart_printf and it absolutely DOES work.

      So for clarity, defining the array in FLASH WORKS, apparently without any alignment #defines…. it is the access method that is having a problem.

      Ideas anyone?

      1. Your use: iprintf(“Data= %ld”,petes[37]);
         your def: iptintf(uint16_t debug_type, char …

         I guess you only missed the debug_type parameter.

8. I found this, which discusses some aspects of flash read/write: https://cesanta.com/blog/esp8266_using_flash.shtml

   It also links to an Espressif doc “99A-ESP8266__Flash_RW_Operation__EN_v0.2.pdf” which covers the same ground.

   I haven’t tried this yet myself.

   1. Well we know we can read and write 4k blocks given an address… the code that Espressif use – or if you want source code, TuanPM uses for backing up vars (I save a single large struct) works – and indeed the code to read from web pages works – but both of these assume you have an address and in the case of web pages, a separate mechanism for storing the stuff in the first place. What I was hoping for was the ability to place an array in human readable form in the main code itself, just as you define structs – and then have a mechanism to access that. As yet, I don’t think we do..

9. Hi Peter,

   Since no lowlevel logs (like reboot reason) provided I can only guess it could be memory alignment problem.
   You might want use combination of attributes: “ICACHE_STORE_ATTR ICACHE_RODATA_ATTR” from
   user_config.h

   Best Regards,
   McGr3g0r

   1. “ICACHE_STORE_ATTR ICACHE_RODATA_ATTR” Compiler didn’t seem to want to know about half of that…

      1. Hi Peter,

         Please see the ICACHE_STORE_ATTR macro definition:
         https://github.com/nodemcu/nodemcu-firmware/blob/master/app/include/user_config.h

         You may define it in your code – user_config.h is probably the best place for it.

         #define ICACHE_STORE_ATTR __attribute__((aligned(4)))

         1. A brave attempt with the aligned #define – compiled ok – but – as soon as I run the line that reads the array – REBOOT…. any other ideas anyone?

Comments are closed.