As the blog entry on using my home control code ROMS for ESP8266 is filling up and of course as much of the information is now dating I thought I’d do a new blog to bring everyone up to speed and move the conversation to this blog entry. The good news is - the code is now running under SDK 2.0 – i.e. bang up to date at the time of writing.
Firstly – yes, everything works – but the procedure for flashing ESP-12 etc has changed ever so slightly. I have updated the binary files to run on Espressif SDK 2.0 and updated the RBOOT code I use for OTA (over-the-air updating) to the latest version.
To backtrack a little…
For anyone new in here – some background - I’ve written software for the ESP8266 which has been in operation for some time and is under constant development. It allows the ESP8266 to control things like simple lights, serial LEDs, PWM LEDs and read sensors etc, talking via WIFI and MQTT to whatever central system you have – for example a Raspberry Pi with Node-Red. I’ve written a node to allow endless units to log in and to provide central communications for them. The software is rock solid and runs 24-7.
I use Windows to develop the software using the Cherts unofficial development environment. This provides an IDE using Eclipse for running and compiling C code. It is not necessary to use any of this if you simply FLASH the ROMS which are available – but of course if you do use the IDE you can customise code for your own use. The projects Home Control 2016 and Nextion WiFi Touch Display both use the same software and are the end result of many years developing home control. Personally I access much of this equipment using Apps like Blynk and more recently Imperihome (see also this article, this and this). Tools I use in Node-Red include my own BigTimer node. All of these options were chosen after much deliberation.
So what have I learned and what has changed?
All has been going well until recently when the Espressif SDK for ESP8266 was updated, firstly to 1.54 then very quickly followed by 2.0 and all of a sudden I and other started to have issues starting up ESP8266 boards. This is now resolved incidentally.
Firstly one of my problems was lack of serial output on power-on. Some serial (115Kb) status messages I have on power up, just point-blank refused to show. I figured the code might be starting at the wrong place. But no, a quick port flick at power-up proved that everything was working – but no serial out.
After much debugging, I discovered that there would be no serial output until this function:
had been called. Prior to SDK 1.54 this was NOT the case. Simple enough - I put this at the start of my code – I’ve just checked and web updating works just fine - but merely moving this function to the start solved all the serial output issues.
Incidentally much is made of the “garbage” that comes out of the ESP8266 on power up – it is just that Espressif chose to use 78,600 baud instead of a more popular one. If user software is set to this speed rather than, say, 11500 (which is what I use) then things look very different – diagnostic on power up followed by user code – all very seamless. Problem is – not that many serial terminals support 78k.. but I digress.
The second problem was a tendency for the unit to run away with itself, firing out garbage as if the ESP-12 was broken. It would SEEM that post-SDK-1.53 or thereabouts – it is NECESSARY on initial run, to flash location starting 0x3fc000 (end of the 4th page in a 4MB Flash chip) with the default file esp_init_data_default.bin which is found in the Espressif development kit. If this has already been blown (previous installation) then it is not necessary to redo that, but with a CLEAN FLASH, failure to flash this area results in the continuous rubbish on output. Again this did not seem to be the case before SDK 1.53/4 – I have NEVER flashed that location in the past. Once blown it is not necessary to blow this again for OTA. Still – now we know.
I’ve just had a Skype chat with Espressif engineers to try to dig deeper. They have confirmed the small file at the end is essential (I probably didn’t notice it before as I’d generally be using boards already programmed) – the serial issue we’re still trying to understand – but it all works.
There are 2 relevant files here. A file called blank.bin sits at 0x3fe000 (top of the third MB in a 4MB Flash i.e. ESP-12) and contains default system parameters produced in the SDK. NOT setting this initially seems to have no effect on my code.
A file called esp_init_data_default.bin sits at 3fc000 and contains default system parameters stored in the SDK. This is a 1K file. This HAS to be blown when first setting up a chip.
What to do
There is a binary file called esp_init_data_default.bin which I have added to the server at www.scargill.net – this is in addition to rboot.bin and romx.bin
In the Espressif setup there is a file called ESPTOOL – there are exe versions and .py versions of this floating around (see below). This is what I use to flash the ESP8266 chips – there are others – you may use them – but I can’t VOUCH for them!
In Windows this works for me – you will have to check for yourself.
To completely wipe an ESP12 – including all data and setup etc and assuming COM3 in this case (not necessary but if it makes you feel better)…
esptool.exe -p COM3 -b 115200 erase_flash
To initially flash the code from scratch… assuming you have the 3 files mentioned above…
esptool.exe -p $(COM3) -b 115200 write_flash -fs 32m -ff 80m -fm qio 0x00000 rboot.bin 0x02000 romx.bin 0x3FC000 esp_init_data_default.bin
Obviously you can change port, speed and file locations. I usually double up on that baud rate but this one is safe.
If you insist on trying this with an ESP-01 (no way will OTA work) – that 3FC000 address becomes just 0xFC000 as they only have 1MB of Flash unless you upgrade them (discussed in another blog in here – done – easy).
As an alternative, I’ve tried NODEMCU FIRMARE PROGRAMMER and that seems to work…
And so there it is – the source is updated on the web and requires SDK 2.0 or better to run. The ROMS are updated and the new file is up there.
Bear in mind there is some data here NOT being erased (see full erase command higher up) – in my case at 0x79000 – or around half way up the first megabyte – this is USER data – stored passwords etc. it was put here for compatibility with ESP-01 but I’m about to move this way up to 0xf800 leaving the best part of a MB for code – plenty of room.
Why worry about updates indeed? Well, one good reason – is that recently, updates to the SDK have used LESS precious RAM – the kind that programs (not data) run in – so with 2.0 there is more margin for expansion – which means more features and more room for more features! It is fine having 1MB to put code in but that is generally NOT the limiting factor with the ESP8266. The code ultimately has to run in RAM and there’s a slight bottleneck there – the gains of later SDK updates help a lot.
Having now added i2c and parallel LCD displays to the board, I set a parallel LCD running from MQTT and updating all 4 lines of the display, once a second from Node-Red. While this was running (it’s been running all afternoon – I thought I’d try a test)… I set my serial terminal program fire out lots of on-off commands in a row – which is WAY over the top…
I set up 30 on/off commands for GPIO0 – to run at 80ms each – as you can see clean as a whistle. I dropped that to 50ms per instruction and the board rebooted – I’m guessing that is a buffer overflow – but then – that simply is not going to happen in real life or anything like it.
I figured I’d go for broke - with 390 (I got sick of cut and paste) commands sent at 80ms it lasts almost to the end then rebooted – again a sure sign of the buffer being in trouble either overflow or wrap – backing that off to 100ms gave zero errors so not a wrap issues – Very happy with this – typically the boards get no more than a few instructions a seconds – often MUCH less. For the sake of it I think this coming week I may just take a look at this – firstly to see if at such speed, my terminal is screwing up – and if not – I’ll see if I can narrow that down but it is utterly academic
A complete and utter aside
I don’t use the Arduino setup but as I was thinking about user data, I just checked the Arduino EEPROM library SOURCE for ESP8266 and the Arduino environment does NOT use the 3-sector trick employed in the SDK. When you update the EEPROM (which is actually FLASH on the ESP8266), the sector is wiped (even if you have user data smaller than 4k, a 4k sector is still used) and then updated from RAM – not ideal. I have suggested to the designer that it might be a good idea to change this. Right now he’s busy working on ESP32 code.
So there it is – all working – hopefully I’ll soon get some clarification on the routine I mention above that I had to move. Time to start adding new features!!!