You may have seen comments in here from Richard Burton about his bootloader. Well, I have to tell you – what a game-changer. If you look at a previous blog where I showed you how we’d moved code from the fairly normal 0x40000 starting point down to 0x20000 – freeing up LOADS of space for more FLASH code. Well Richard helped with that but he also suggested I give his bootloader a shot. This weekend after clearing out my many other jobs, I decided to get in touch with him. He pointed me to the source of his bootloader and with a minor tweak to my Windows PATH I managed to get MAKE to work in a DOS box so I could compile it – the upshot of that was a small bootloader .BIN file.
I’m still no use with Makefiles but I managed to generate the .bin file and Richard updated my Makefile. The result.. astounding.
What you see above – the left 2 columns represent the very original code I used with TuanPM’s MQTT as the base. In the middle one I’d moved non-volatile storage up to the top of available FLASH space (which applies to most of the current crop of boards from ESP-01 upwards) and moved the start of code all the way down to 0x10000 giving a massive amount of room for development – unlike the original case where I was running out.
Well, now with the bootloader you can see that virtually the whole LOT is available for programs! No extra cost, no extra work – just getting it right. I’m also now using the non-volatile storage routines in the SDK itself rather than the ones that TuanPM made (pointless having the same code twice). Right now my code (all in including the SDK) finishes at 0x54c00 – not much more than half way up – and that’s in an ESP-01 – the ESP-12 potentially has WAY more FLASH available. All of this is running with Espressif SDK 1.11
All that is needed now is a bit of careful thought as to how to best use the FLASH… there is still a good amount of RAM used up and I suspect that large routines in FLASH will just make matters worse – on the assumption that the whole routine gets pulled out into RAM to be used, I’m think it might better to split large routines up into smaller ones so that RAM can more readily be re-used. Time will tell whether that theory is sensible or a load of old codswallop.
This opens up some possibilities for large FLASH-based lookup tables for light shows – which brings me to the realisation I haven’t a clue how to make good use of that without pulling the lot into RAM – oh, well a challenge for another day. My thanks to Richard and lest we forget – here’s the link to his article on his bootloader.. http://richard.burtons.org/2015/05/18/rboot-a-new-boot-loader-for-esp8266/
Hello Scargill,
Its been months since you have configured your ESPs to work with rBoot. I hope you have stabilized the crash problem. I am working on a project that requires the combination of rBoot and TUAN PM’s MQTT. Can you share your code? Or can you point me to a source where same combination can be found?
Hi there Ashubhatt
I never resolved that – I’ve had other projects to work on. I have successfully combined MQTT and OTA using the Arduino setup – but not yet with Eclipse etc.. and I would like to. I never did quite grasp the situation with the second file or why it needed to be installed twice – by second file I mean the second BIN file with web pages.
As I recall when running OTA it would work – but every now and then when updating it would simply die. I’ve not had time to get into that properly.
Thanks for the response.
I have achieved the basic OTA using Eclipse. But I have no idea about integrating MQTT with OTA. May be some one can guide us here.
Plus may I have your basic code that has the combination of MQTT with rBoot? I may tweak the code and make it work somehow, if you don’t mind.
Just wondering, whether those crashes are result of insufficient memory space!
For ESP-12 modules having 4Mbit memory space, the size for rom (both rom0 and rom1) should be less than approx 240KB; Just the simple rBoot sample code takes 216KBs, which will significantly increase once we combine MQTT code into it. This might crash the downloaded OTA program..
I did notice that the Arduino environment chucks out extremely large code. in Eclipse I could not fill the FLASH if I tried..
To respond to Squonk (there’s no reply just below his comment – whoever designed this WordPress plugin needs shooting) the short answer is I have no idea how the paging mechanism works – I do thanks to Richard know that there’s a block of 32K RAM for pulling these functions out of FLASH to execute them – but how it does it and what is actually pulls out (a page or the exact function) heaven only knows for now. If anyone knows, please by all means step in here.
We ALSO discovered last night that most likely from SDK 1.01 onward, messing with the watchdog timer appears to do nothing. I had in the init code a flashing light using delays.. before any WIFI was setup – and I noted (thinking it was something else to blame) that the board was resetting… I’m now using a callback to flash the light – but the fact is you USED to be able to use delays (which would clearly mess up any background stuff but that didn’t matter at power up) and now – they cause reboots.
Hi Peter,
I opened a topic with all my questions (and guesses) regarding the ICACHE_FLASH_ATTR and the magic cache mechanism on the official Espressif forum, but I got no answer yet:
http://bbs.espressif.com/viewtopic.php?f=7&t=622&p=2295&hilit=i_cache_flash_attr#p2295
If you have a chance to speak with someone in charge at Espressif, please relay these!
Awesome timing! I’m just now getting serious and moving beyond simple esp builds. I came across rboot from comments in your previous posts. I liked the idea of rboot, but wasn’t sure if it was ready for use. It’s tough to figure out which projects to run with because all of this is so new.
You’ve convinced me to give it a shot! I’ve forked the repo and have made a bee changes to the Makefiles that simplify the build process a bit, I’ll get that PR’d ASAP.
But I also want your code, if you don’t mind sharing. An example combining rboot with tuan’s mqtt is exactly what I need. Any chance I could take a look?
Thanks for all your hard work and spending the time to write up your discoveries, it’s been a huge help in finding my way around!
Ian
You’re welcome Ian once it’s working! Part of the code ever since we started moving code around (even before the rboot change) simply crashes and I’ve no idea why. The fellow who is working on that, my pal Aidan has just come back from hols and isn’t up to speed yet – I’m hoping he’ll take a look tomorrow night – I’ll put a note in here when it’s working.
I would also like to thank Richard and his rBOOT package and tools – he spent alot of time this weekend helping me update my project (IoTBox) with OTA updates. I can now have these devices updating to the latest codebase without having to open them up.
I’m using ESP12s so have the ability to have 2 complete setups in rom and switch between them easily.
Interesting – my ESP-12 says it just has 4Mbits… I guess it must be the luck of the draw which one you get.
When you boot your ESP-12s – at 78k – the boot up code – how much FLASH does it say it has – my ESP12s say 4Mbit!?!!?
The size displayed is just based on a few bits set in the header flashed to the chip. If you set the size right on the rom (and esptool, as that overwrites the one in the rom when it flashes, iirc). You can also query the flash chip id with esptool and look it up to be sure what you have.
This is how Richard explained the flash layout plan for my application (based on Martin Harizanov’s 3 relay board. http://harizanov.com/wiki/wiki-home/three-channel-wifi-relaythermostat-board/:
“0x000000 rBoot
0x001000 rBoot config (default generated one will suffice)
0x002000 rom1
0x080000 espfs1 (#define ESPFS_POS 0x80000 in httpdconfig.h)
0x202000 rom2
0x280000 espfs2 (same as above + 0x200000)
0x3f8000 user config (#define CFG_LOCATION 0x3f8 in user_config.h)
0x3fc000 sdk config
This gives you about half a mb for your code and same for your web pages. You can change that split if you like, but you are unlikely to need to.”
There is a separate 0x8000 byte area of memory used for caching reads from the flash, so don’t worry about trying to write functions in small chunks, they don’t end up in “normal” ram.
For balance I should also say the SDK boot loader could have provided you with the same flash layout advantages over the standard boot process (or as the BBC might say ‘other boot loaders are available’), but in a less balanced way I’ll point out that the SDK boot loader isn’t open source, seems a bit buggy, and doesn’t do as much as rBoot 😉
So for clarity Richard.. if you had NOTHING market with “ICACHE_FLASH_ATTR” there would be 32Kbytes (or less) for storing programs? I’m assuming in that case the whole lot would be copied out of flash to remain permanently in that area (until power down). Does that sound right?
Pete.
Code not marked as ICACHE_FLASH_ATTR will end in the .text section, which is copied to normal iram (0x40100000) at boot and stays there. Code marked with ICACHE_FLASH_ATTR stays on the rom at start, but the memory mapped flash has a read cache in a separate section of memory (0x40108000), so when you call a function it is copied there, which means first run is slow but for as long as it remains in the cache it will run again fast.
You have a lot less than 32k for your own code in .text/iram because the SDK puts lot a lot in there too. If you write bare metal code, like rBoot itself, you have the whole 32k to play with, but you’ll soon miss the SDK!
Just a clarification request Richard… so code NOT marked as ICACHE_FLASH_ATTR ends up in the .text section… then copied to IRAM – so the .text section – that must presumably be part of the FLASH space….. as it has to start in flash at some point…
Can you confirm that .text is just another part of the FLASH or am I going mad?
Yes.
.text .data and .rodata are all on the flash and get copied to various bits of memory when booted
.rom0.text is also and the flash and stays there until chunks are temporarily copied to cache, on demand
I am just wondering if functions are copied one by one on demand into the cache when required, or more likely (from a pure hardware point of view!), if the whole Flash page containing the required routine is mirrored into ram cache, much like a non-transparent page fault mechanism?
If the cache works in whole page chunks, this would mean that the routine size is irrelevant, and would also bring some interesting questions on what happens when a routine code crosses a page boundary…