Accessing ESP8266 Arrays in FLASH

A new challenge – the answer to which I will post in here.

Accessing an ESP8266 C array in FLASH – i.e. never touching RAM until you actually access something.  One suggestion was to ensure you use 4-byte variables.. so I tried this - it compiled - the array is in FLASH - but the access IMMEDIATELY reboots the ESP8266. iprintf is just a macro for printf - no issues there..

static const uint32_t petes[] ICACHE_RODATA_ATTR = {
55,45,66,77,88,12,76,23,45,67,34,65,25,74,234,67,3,643,646,
55,45,66,77,88,12,76,23,45,67,34,65,25,74,234,67,3,643,646,
55,45,66,77,88,12,76,23,45,67,34,65,25,74,234,67,3,643,646};

iprintf is just a macro for a normal printf function...

iprintf("Data= %ld",petes[37]);

Result: Immediate reboot....

Thoughts anyone? Why do we need this? Because FLASH is in plentiful supply on these boards, RAM is not.  Ok, you could use tricks from the web page lookup and make functions – but that involves knowing exact addresses etc.. If possible I’d rather let the compiler worry about that.

I’ve posted this on the Espressif forum – I’ll post a working solution in here – but I’m out of ideas….

Facebooktwittergoogle_pluspinterestlinkedin

23 thoughts on “Accessing ESP8266 Arrays in FLASH

  1. Hi Peter,

    Since no lowlevel logs (like reboot reason) provided I can only guess it could be memory alignment problem.
    You might want use combination of attributes: "ICACHE_STORE_ATTR ICACHE_RODATA_ATTR" from
    user_config.h

    Best Regards,
    McGr3g0r

    1. Well we know we can read and write 4k blocks given an address... the code that Espressif use - or if you want source code, TuanPM uses for backing up vars (I save a single large struct) works - and indeed the code to read from web pages works - but both of these assume you have an address and in the case of web pages, a separate mechanism for storing the stuff in the first place. What I was hoping for was the ability to place an array in human readable form in the main code itself, just as you define structs - and then have a mechanism to access that. As yet, I don't think we do..

  2. So hey, this kept eating at me, and my gude wyffe turned in early, so I thought I'd leave the comforts of my preferred C++ framework to fire up the C examples.

    Once I remembered how to build them, I went into the user_main.c of the 'Hello World', and I added your array declaration like so, and started to play with print statements...

    static const uint32_t petes[] ICACHE_RODATA_ATTR = {
    55,45,66,77,88,12,76,23,45,67,34,65,25,74,234,67,3,643,646,
    55,45,66,77,88,12,76,23,45,67,34,65,25,74,234,67,3,643,646,
    55,45,66,77,88,12,76,23,45,67,34,65,25,74,234,67,3,643,646};

    LOCAL void ICACHE_FLASH_ATTR hello_cb(void *arg)
    {
    ets_uart_printf("Hello World!\r\n");

    ets_uart_printf("Pete's test [0]: %d\r\n", petes[0]);
    ets_uart_printf("Pete's test [37]: %d\r\n", petes[37]);
    ets_uart_printf("Pete's pointer: %d\r\n", petes); // address of first element of array
    }

    It compiled, and I got as output:

    Hello World!
    Pete's test [0]: 55
    Pete's test [37]: 646
    Pete's pointer: 0x40264b54

    So... hope it works for you as well.

    1. WOAH!!!! You're onto something here, Ken - unbelievably - this works! Well done - but...

      So what I wanted was a simple bit of code to allow me to hide boring system messages as debug - with various levels. Here's the code.

      void ICACHE_FLASH_ATTR iprintf(uint16_t debug_type, char *fmt, ... ){
      char buf[128]; // resulting string limited to 127 chars - change it if you like
      va_list args;
      va_start (args, fmt);
      ets_vsnprintf(buf, sizeof(buf), fmt, args);
      va_end (args);
      if (debug_type & enable_debug_messages) uart0_tx_buffer(buf,os_strlen(buf));
      }

      And so as you see, I build up the string into a buffer and IF the first parameter is correct, I print out the result serially.. so things like MQTT debug messages can be hidden unless I really want them. This works PERFECTLY - or so I thought until now. So what you have shown is that it is NOT the array definition that is in error... (you've also surprised me in that %d seem to be handling 32 bit vars, I was expecting it to only handle 16 bit vars, that in itself is worthwhile) but something else. it isn't the "uart0_tx_buffer command but something else. I tried your direct use of ets_uart_printf and it absolutely DOES work.

      So for clarity, defining the array in FLASH WORKS, apparently without any alignment #defines.... it is the access method that is having a problem.

      Ideas anyone?

      1. Your use: iprintf(“Data= %ld”,petes[37]);
        your def: iptintf(uint16_t debug_type, char ...

        I guess you only missed the debug_type parameter.

  3. sorry, correction to last line for hex output:
    ets_uart_printf("Pete's pointer: 0x%x\r\n", petes);// address of first element of array

  4. Ken - you are 100% right - thank you for that - just me not paying atttention - so one step forward, but, sadly 2 steps back. I was indeed me missing a parameter off... SO I tried 16 bit tables - they work - 8 bit tables - they work - I tried reading an array subscript from the 8-bit table - and reading one 3 bytes back - to ensure I could read off-boundary - no problems at all.

    Sadly, when I put the attribute onto the 8-bit table I use for PWM lookup - the processor crashed again - and I don't understand why as it is only 3 lookups every 20ms.

    More work to do....

    1. It's always good to have many eyes on a problem.

      If you're still stuck on the PWM lookup issue, feel free to post again, and several readers will attempt to beat me to finding a solution 😉

      Regarding the message filtering, please confirm the following: you're generating messages in your code with a few predefined prefixes (eg critical, error, warning, info, debug), and then you want to conditionally output them to serial according to your filter setting. Is this correct?

      That's one way to approach it. Another way is to surround each message statement with a define test, such as:

      #ifdef __MSG_WARNING
      [print the warning message]
      #endif

      ... and before a compile, just define the message types you want output.

      This second way may seem like more work at the outset but I see two advantages:
      - for the release version, with minimal messaging, the bin will be smaller
      - you only have print messages when you want them, and you're not running an extra process on every single message.

      1. Hi Ken

        All is revealed. The lookups work great. However, you simply cannot lookup 16 bit or 8 bit values at speed.. it causes a reboot. So - I took my 16 bit PWM table and my 8 bit brilliance table and made them 32 bits - and lo - they work perfectly out of FLASH. The perfectionist might create a function to get the relevant 1 of 4 bytes out - but that then makes the table a little confusing - OR you could do what I've done and say... to hel with it, since employing the bootloader I have LOADS of Flash and little RAM - so it's a no-brainer really.

        All working perfectly and I've even had time to redesign my LED CLOCK, code is in the repository - and THAT is working a treat. I have a 60 LED array, I have a led flying right through the array every 60th of a second in one colour, a blue second hand, green minute hand and the hour hand is 5 LEDs wide, it looks pretty good on a 60-LED strip - it's going to look stunning in a round infinity mirror!! Just a slight matter of getting 2 round pieces of glass and some means of making a border around it to handle the LEDS that doesn't look like amateur-night-out. That is always the killer. So today's been a good day! I put a short video of the clock software running here. https://www.youtube.com/watch?v=WJK2KpnsmLk

        1. Hi Peter
          In your last paragraph of this post you say 'code is in the repository' where would this be? I've searched quite a few of your other blog posts and cannot find a link.
          It would be nice if you would link this in the actual post.

      1. I've updated the post with a uint16 example. While I was testing with gcc under cygwin on windows I hit a nasty optimisation bug at -O2 and above - it completely removed the "bytes = *(uint32*)((uint32)addr & ~3);" line, leaving the bytes variable uninitialised and so ultimately returning garbage. Don't know if the gcc version we are using for esp8266 has this same bug, so do just check you get the result you expect when you first use it.

        1. The latest version of my software now uses the 8 bit version (with acknowledgement!). In the future I plan to have some fancy table-based lighting effects and this is absolutely ideal as I can put very large tables in FLASH without running into RAM problems. Marvelous.

  5. You are such a gentleman Richard - right - I'm having that. GUYS - take a look at Richard's blog... nice simple solution while retaining the array format..

  6. Hi Pete,

    Today I was experimenting with this, trying to get all my user interface messages (a lot!) into rodata, so they wouldn't occupy dram. I've always wanted to do this but was hasitant because of the alignment restrictions, it would get quite complex.

    The first thing I did was see what section read-only strings (const char *string) end up in. That appears to be rodata section! Not the bss section like I expected (that is default behaviour for C-compilers because some very old very naughty code assumes that pointers to characters are always writeable!). Now I must say I append about 100 extra options to cc, to drop all and all legacy compatibility, so I guess for my setup that was to be expected, good.

    Then I would make a test string of 8k and see what happens, const char test[8192]. Not static like I do usually, don't give the compiler a reason to optimise it away. No referencing though. Result: no heap size decrease. At all. Added a reference. No change. Only when I added an explicit write to the string, the heap size would decrease by 8k. As soon as I removed the write, the heap size was back to normal.

    I guess this means that if the compiler deems a variable to be read-only (you can help it make the decision by using const, but that's not even required!) it is NOT copied to dram. This also means the esp actually can read bytes from random locations in the flash, no aligment necessary (maybe for 16 or 32 bit arrays, but that's not my main concern).

    I don't know if this is a recent "accomplishment" of the ESP SDK or the compiler/linker suite, or if this has been working for ages, but I must say I am quite surprised. It also explains why my dram hasn't filled up with the lots and lots of user interface messages in my code.

    So at least for character arrays, the compiler/linker/sdk do the "right thing" already fully automatically!

    This is quite different from ATmel microcontrollers, with it's Harvard architecture, where a program can never read (by the usual means) from program memory, so all strings, all data, everything must be copied to ram on startup (or read with special instructions).

    Happily surprised 🙂

Comments are closed.