ESP8266 FLASH– Something to Ponder

Update: You’ll see from the original article before the break, I was having serious problems with ESP-01 versus ESP-12 etc where code running in the 12 would simply not work on the ESP-01. Well, it turns out none of this was helped by the MAKEFILE..

You’ll often see references to files at 00000 and 40000 (hex) and like me probably wondered what’s happening with all that space in the middle. Turns out much of it is thrown away.  I’ll soon put the new Makefile up that we’re using but perhaps the mere fact of this email will help some. You don’t have to have config information stored at 3c000 and you don’t have to have your code at 40000 either!

Thanks to reader Richard Burton who clearly knows more about Makefiles than I do, we’re now compiling our code starting at 20000.  It turns out that 7c000 is used by SDK – but 78000 is available and that is exactly where I am now storing non-volatile information (and this works equally well on the ESP-01, ESP-07 and ESP-12 boards and means we can now code from all the way down at 20000 continuously upwards through to 78000).  Whereas before my code was up to 73000 it now goes from 20000 to 53000 leaving a TON of room for more FLASH code.  It has to be said however that I’d not realised how close to the wind I was sailing with RAM and the outcome of that is – wherever at all possible add the ICACHE_FLASH_ATTR to your C code so that the code runs from FLASH rather than being permanently cached in RAM.

So..

If you have the start of a function like this..

LOCAL void temperature_cb(void *arg)

Just amend like this..

LOCAL ICACHE_FLASH_ATTR void temperature_cb(void *arg)

The only place I’ve found I could NOT use this was a couple of small, tight routines to handle WS2812B LEDS where time was of the essense (microsecond or so).

Shortly I’ll update this and put a link to the Makefile so you can compare with what you are using. I still don’t entirely understand the Makefile but I’m slowly getting there. Just a tad short of blogging time as we’re in the process of kitting out a house and we’re on a deadline but I plan a LOT more on this subject in the coming months as everything is starting to come together.

Below, the original article where you can see I was getting quite frustrated.

----------------------

For now, I am defeated and I’m hoping one of you readers is a step ahead of me.

For a while now, we’ve had an issue blowing ESP-01 with my code which is based on TuanPMs MQTT code but just keeps getting bigger and bigger.  I’d have no problems blowing an ESP12 but once we got past API 0.96, ESP-01s would not work – then I hit on the idea of reversing the order of the two files I was blowing. That worked for a while.

 

Now the code goes all the way from 0 to 73000…. so the solution to get ESP-01s working is to use an option you see in some ECLIPSE setups for Windows – the Makefile there is a FLASHONEFILE option… this makes a single file that goes all the way up to 0x73000.  LOVELY… except – TUAN stores his variables like API and password at 3c000… so this gets wiped every time.

I thought I was being really clever by moving his pointer to 7c – so that must-keep variables would be stored at 7c000 – WELL above my code.

Nope… turn the thing off and on and the variables are remembered no problem. Update the code with FLASHONEFILE – and BANG! Variables gone, defaults loaded. I have ABSOLUTELY no idea why this is happening.

Any thoughts

Pete.

Facebooktwittergoogle_pluspinterestlinkedin

13 thoughts on “ESP8266 FLASH– Something to Ponder

  1. Hi Peter, reading very fast your issue I wouldsay to consider that settinging Cfg location of mqtt libs at 7c, libs uses 7c + 3 so up to 7f... May be that you're overwriting one of this segment .... Try a dump of flash mem to see what happens in detail... Hope you help ... Rgds

    1. Hi
      Not exactly sure how to do a dump of flash memory... the thing is - I can see the compiler/linker/flasher blowing the FLASH up to 73000 then stopping - there is no way it is writing above that - so I can't understand why I can't write to 7c000. Yes but that's a good point, it is probally worth writing some kind of hex display job to print out 7c000 upwards.... However there is an ongoing issue which I've never gotten to the bottom of - of the erase process overwriting further than it is supposed to - at least that's what I think it is - and it seems to be the reason why flashing large programs in 2 or more pieces to ESP-01 fails - yet it works if flashed all into one place. It really would be nice if someone had an easy to understand memory map of the whole available memory so we could see what we can use and what we cant. As far as I can see for example, from 00000 to 40000 not a LOT of that is used up - and much of it is presumably wasted, I've never understood why people start the programs at 40000 instead of findout out where the end of the SDK is and starting from there. Maybe in 100 years I'll understand these MAKE files...

  2. When you talk about people working from 0x4000 do you mean the position of the .irom0.text elf section, where people put most of their code (using ICACHE_FLASH_ATTR)? If so you're right it would make more sense to not have to pre-define your irom/iram split. But you can avoid that by using the espressif boot loader or rBoot (http://richard.burtons.org/2015/05/18/rboot-a-new-boot-loader-for-esp8266/) and building your roms as for the "new" type loader v1.2+ (as I mentioned in my previous reply to your blog). Looking at the make file in that post it looks like you're using eagle.app.v6.ld which is the "old" pre-boot loader way. That's fine, that method isn't deprecated, but there are advantages to doing it the new way instead.

    As for writing your settings to 0x7c000 this is a bad idea if you're writing it to a 4Mbit SPI flash. The sdk appears to reserve the last 4 sectors for it's own use. If you have a 4Mbit chip (I'd take a guess from your problem that's what you get on the ESP-01) that means 0x7c000 to 0x7ffff isn't available for your use. I played with these sectors when I was writing rBoot and they just aren't safe - in various configurations they were all overwritten. The SDK does it in a destructive way as well, it erases the whole sector but only writes back it's own stuff, it doesn't write back a modified version of the the original sector. This means you can't stick your own config half way through the sector, where the sdk config is unlikely to ever reach.

    1. Thanks for that Richard... though quite familiar with C, the workings of the Makefile are currently bordering just above black magic for me. I took a look here.. https://github.com/raburton/esp8266/tree/master/rboot but am now not too much further on. Even the various terms iRam/iRom seem vaguely ove rcomplicated some someone used to FLASH, RAM and DYNAMIC RAM.

      So normally, TuanPMs code is at 3c000 and it works there. But clearly writing one file from 00000 to 73000 I'm overwriteing it. So I tried 7c000 - I have to say that DOES work, ie you can power the board down - and up - and it will continue to work - but if you re-blow the FLASH - ie 00000 to 73000 again - it's gone - for the life of my I can't think why that would be if it can survive power cycling unless you're saying the SDK does some over-writing once it is running.

      Here's the relevant makefile. It does not help make it clear to Makefile beginners like me when there are so many options..... if I knew what I was doing with this I'd clear it out to make it more understandable. I'd just assumed there was FLASH available from 00000 to 80000...

      https://bitbucket.org/snippets/scargill/7dqe

      I was using the FLASH section at 392 but this was causing issues with ESP-01 boards so I'm using the FLASHONEFILE section at 342.

  3. I'm not surprised you're struggling with that Makefile, it's horrible! It appears the author has taken large chunks from samples with the SDK, which were pretty poor to start with (pulling in loads of stuff that really shouldn't be in the make file in the first place), and then tried to build on top of that - not a good starting point. I've just started writing a tutorial on how to use rBoot for my blog - an easy guide to get up and running it, though the general principals apply to the SDK boot loader too. With luck I'll have it finished tonight and I suspect it might help you. Although sorting the wheat from the chaff in that Makefile to implement my suggestions is going to be a job if you're not familiar a lot of that stuff in there, so if I get chance (I'm not too busy over the weekend) I'll try and actually compile the code you're working with and see if I can make a sane Makefile. If you can point me to the source. You're welcome to email me if you'd rather keep the blog tidy.

    1. You've just made a new friend - Makefiles are currently the bane of my life. If I highlight your email it doesn't make sense. Want to start the ball rolling by emailing pete@scargill.org and I'l figure out how to make the code available for you to have a go at...

  4. I'm interested to see what happens here. Although I'm mainly using esp12s, I have a few esp1s lying around for a simple mqtt temperature project

  5. I'm also very much interested in a more efficient way of using available space. I'm also using TuamPM's MQTT code and almost run out of space now but want to add some more code to it.
    @Richard, where can we find your blog on rboot?

    1. Click on my name for my blog (or just go to http://richard.burtons.org). The easiest way to make more efficient use of the space, and without having to keep adjusting things, is to use a boot loader (either rBoot or SDK boot loader v1.2/v1.3), even if you only intend to have one rom and don't want to OTA update. Build your rom image with an appropriate linker script and boot mode (use -boot2 for esptool2) and you'll get a single file instead of two, which are packed together - so no wasted space between them.

      The only complication for Peter was that he also wanted to flash a mini-filesystem with webpages on the rom, and when I tried to move this it didn't seem to be able to find it at runtime. Can't remember if I mentioned that Peter, but I did experiment with that. Ideally you'd use newer rom format to have all the the app code together at the bottom of the flash with room to grow upwards, and flash the filesystem up top, moving it down a little if you add extra content. Anyway, we ended up compromising and not using a boot loader, but readjusting the split on the flash in a way that seems to have worked out fine (if we could move that FS we could probably do a nicer job). If you want more help feel free to contact me, my email address is on the about page of my blog, or ask questions on the blog itself. I'm flying out of the country today for 10 days, so I may not be able to help much from my tablet and dodgy hotel wifi till I get back.

      1. Once again Richard, thanks for your help. I do want to understand this better and would like to try your bootloader, perhaps when you are back from your travels and I'm done house-moving we can re-engage - I'd like to figure out how to get that RAM usage down and understand how program ram space affects RAM overall... Seems to me you're the guy to talk to 🙂

  6. That's a great update, thanks. I'm still coming to grips with the ESP8266 memory map as well, and Richard's info on the bootloader has been very helpful.

    I'm mainly using the Sming 'Arduino-ish' C++ ESP8266 framework these days. In their makefiles, their load addresses are 0x0000 and 0x9000, so they're sliding the second load down even lower!

  7. I am rather new to the ESP and firmware in general, but I'm reading this pdf (Kolban's Book) and I'm seeing that the ESP has it's memory split in half in order to provide over-the-air updates. One half would contain the [buggy] user uploaded program and the other half is meant to accept the new updated version. Your variable wipe could be because of this?

Comments are closed.