I’m often asked about setting up MQTT, Node-Red etc. on the likes of Raspberry Pi, NanoPi or similar, with a view to controlling ESP8266 devices.
Novice readers start here
Usually it is difficult for me to see this from a beginner perspective, but having spent 2 months in hospital thanks to an untimely stroke from which I’m still recovering, 4 months after the event, I recently found myself doing new installs without the benefit of doing this stuff every day. I am just SO glad I blog everything.
Over time and with lots of help, I’ve developed an install script to put all the tools I need onto a Raspberry Pi or similar board. This should be used on a virgin board with nothing but the full operating system. We’ve never tested and don’t intend to test NOOBS. Instead, the script has been tested on Raspberry Pi 2 and 3 (and 3B+) with Raspbian including the STRETCH variety.. and also on various (detailed in the code) boards such as Orange Pi Zero and others, Roseapple Pi, various FriendlyArm boards and more, using DIETPI, Ubuntu Core, Armbian and Debian.
The script
The script includes full setup for Mosquito MQTT, Nod-Red and lots of nodes including my own, SQLITE and many great utilities. Please check the script to see what it has been tested on, if a board or operating system is not mentioned, it has NOT been tested. Please don’t try “pick and mix” – use the script as indicated.
All work at the PC end, i.e. setting up, is generally done on a Windows PC, not mandatory but I sit at a PC, not a MAC or Linux box so that’s my perspective.
Here is the link for “the script”. https://bitbucket.org/snippets/scargill/ekon6o/the-script
The main blog entry https://tech.scargill.net/the-script/ has information to wget the script straight onto the likes of a PI as root user.
In the comments inside the script, you will find all information on systems tested.
Typically “the script will be run on a board with nothing but a clean operating system on it. It helps if remote root access and SCP is enabled. On a Raspberry Pi, user “pi” will usually exist, on other boards, the script will set up this user and the needed group access. You might also want to ensure that I2c is enabled in the board setup if you wish to use this.
You need nothing else but a copy of the script as this will install software as needed from online resources.
ESP8266 code
For ESP8266, you can install code from the binaries I provide or compile the source. https://bitbucket.org/scargill/esp-mqtt-dev2 – this now even has its own brand identity ESP-GO
No, I don’t use the Arduino model, I just code in C with the Espressif SDK version 2.0 (later versions don’t leave enough RAM, earlier versions work but have minor bugs so grab 2.0 – better, just use the binaries). RAM used specifically for code as against data, soon runs short with ambitious coding on the ESP8266 no matter what you do – don’t confuse this with heap RAM, shortage of which could affect reliability.
I use ESP8266Flasher.exe from NODEMCU to blow ROMS. For flashing generally use QIO mode as set up in the but the WEMOS boards seem to want DIO mode, all of which is down to the kind of FLASH used on the board.
At 00000, flash rboot.bin from my firmware folder, at 0x2000 (and assuming an ESP12 or similar with 4MB Flash – I no longer support the old ESP01 with only 512KB Flash) you need my rom.bin file from the same place and lastly at 3fc00 you should flash esp_init_data_default.bin from the ESPRESSIF site. It is also in my ROMS public folder with the other ROMS. http://roms.scargill.net
Occasionally on new boards I find it useful to erase the entire flash before doing the above, using esptool or similar. I use the .exe version of esptool out of preference.
In the above photo, two separate projects on one board, I’ve used a LED on GPIO13 (to use the Espressif naming) which flashes automatically and another on GPIO12 which turns on and off depending on the state of a Node Red BigTimer (I usually use my own Node-Red-Contrib-Bigtimer for this, with various devices able to trigger manual timer overide via MQTT messages. Also, in each case above, GPIO4 and 5 are I2c data and clock to the displays, with pull up resistors to 3v3.
As for talking to the ESP8266 from the Pi, I exclusively use Node=Red and mostly MQTT, all of which you will find detailed in the WORD document that is stored in the BitBucket repository along with the source. the code supports a wide range of LCD and OLED displays as well as 2 line LCDs, a range of temperature, humidity, pressure and other sensors (some simple one-wire, some I2c on GPIO4 and 5) including DHT11 and 22, Dallas temperature sensors, SHT30 light sensors, BME280, BMC180 and many more. More device support than I’ve seen elsewhere all at once.
There are several large blog entries in here covering all of this.
Options out there
Why this and not other ways? Well, all of the above has been widely tested over time in different countries and proven to be rock-solid. I use this ESP8266 code except for the odd occasion I’m using SONOFF boards. SO Much work has gone into the Tasmota software it seems daft to re-invent the wheel if Sonoffs are involved and the software supports Alexa.
You will find lots of discussion in here and also lots of info in my YouTube channel and on my Facebook pages ESP8266WIFI and IOTSTUFF.
See “Home Control 2017” (I may rename that to 2018 or later in time), “The Script”, “Nano I2c Peripheral” and “Big Timer” for starters.
When I started to see limits to I/O on the ESP8266, I added what I call my NANO i2c peripheral as an option to the project to give loads of extra I/O, PWM (now 6 channels i.e. 2 RGB channels) and analog inputs via a dirt cheap Arduino NANO type clone. One variation of this, not fully tested yet, includes infra-red. The code for these boards is available below and they will (via i2c) provide lots of extra I/O to the ESP8266 along with serial, PWM, analog inputs and servo control.
https://bitbucket.org/snippets/scargill/kRg5o/nano-peripheral
In the example photo above, the ESP12-based board drives the Nano via I2c, with two pull-ups to 3v3. GPIO 4 and 5 on the ESP8266, A4 and A5 on the Nano. GPIO12 on the ESP8266 can be turned on and off by Node-Red and D6 on the Nano similarly, or depending on the instruction in Node-Red (via MQTT) turned up or down. Note the latter only works with pins able to do PWM. All I/O pins on the Nano are available to be used in the software (except 0,1 and A4,5),
At £1.50 from AliExpress there simply is no more cost effective way to add I/O to the ESP8266 if you need it. As for the ESP8266 itself, I find the ESP12-based Wemos D1 Mini (again usually from AliExpress) to be unbeatable unless you need a board with relays etc. already fitted.The two together come in way cheaper than an ESP32 for example while (depending on your needs) offering more. If you are convinced you need an ESP32 as the ESP8266 isn’t fast enough for you, try double clock speed (160Mhz). I’ve an instruction for that.
This home control project is on-going, I started messing with control boards for the home, last century, way back before WIFI was so universal and continue to enjoy new innovations as they appear. The ESP32 may eventually compete on cost, right now there is almost nothing I can do with the ESP32 that I cannot do equally well with the ESP8266. No doubt about it though, more program space RAM (not FLASH, got loads left) would be good.
Fast flashing generally indicates startup condition when not logged into wifi.
No They are all on a bus bar and have been fine for months with the odd drop out and re-login. I must have a problem with wifi as I am getting hundreds of re-logins from the devices as well as the fast flashing. What does the flashing signify?
I’m running your firmware.
thanks
John
Perhaps I’m being stupid but.. I cannot find a reference anywhere to the flash rate of the led on pin 13. I have a load of Sonoffs flashed with Pete’s firmware and normal slow flash I assumed to be MQTT good. My devices are now all flashing furiously at approx every 500ms but strangely MQTT connection works fine?
Sonoffs… is it possible you have the output LED set to RGB? Matters not if the MQTT is working.
Peter, what about this alias for NPM?
alias npm=”cd ~/.node-red;npm ”
(that’s a tilde sign after cd and a space, NOT a dash/hyphen…)
Take a look: https://pastebin.com/qwt3dNP4
new version of The Script just sent to Peter, changelog:
## 01/05/2018 Redone InfluxDB/Grafana/Chronograf install, now works on every platform, and succesfully tested on Rock64+eMMc
Just wanted to say a general “THANK YOU!” for “The Script”.
I just got my new Pi Zero W and 3B+ (I used up my previous ones in projects that are in daily use) and automatically and without thinking used “The Script” to get them ready to rumble.
Thanks, Pete! You cannot imagine how much time and inventing new colorful curses you saved me over the past year 🙂
Especially when one is not 100% on the best foot with Linux, “The Script” is perfect to nontheless be able to spend the precious little time I have tinkering, and not reading though endless forums.
Heartfelt thanks!
And I of course acknowledge all the other people who have contributed – thx everyone!
A good friend in the US will start the long and interesting Raspberry PI march soon, I guess I better introduce him to your magnum opus.
Greetings from Canada, BC 🙂
Pete (Scargill), just wanted to give you an update that I was able to flash my chips successfully. Thank you for the help. Since Pete (Knight) brought up the subject, what is your current though on front end/dashboard application – Blynk, Node-Red dashboard, ImperiHome or something else?
On to getting a server set up for MQTT & Node-Red.
Good luck. Not happy with Imperihome. Looking into both node-red dashboard and blynk
Rushing for a flight.
Pete
I’m not a great fan of the Node-Red dashboard. It has its uses, but I’ve abandoned trying to use it for my projects.
Pete.
Hi Pete, glad to see that you’re actively blogging again. Hope the weather is good in your part of Spain. It’s been a bit breezy over here on the coast, but still very nice.
It’s been a while since I used ‘the script’ for a fresh install, but last time I did it was installing the old version of the Blynk websockets node for Node-Red.
It’s not a big deal to uninstall it and install the latest version instead, but some newcomers may not realise that there’s a greatly improved version available. The one to use is “node-red-contrib-Blynk-ws” which is being actively developed by Gabriele Lau.
Pete.
Well now you mention it I might just give it a go.. but I noted the other day that my nearly endless Blynk credit has magically disappeared.
Are you logged-in to your Blynk app?
If you are and the credit isn’t showing-up then just message @Pavel and let him know, I’m sure he’ll sort it out for you.
Pete.
you can have a local install of blynk server using my automatic install script: https://gist.github.com/fragolinux/a5e7d6153afab4c52d91d41ff9860b8c
please let me know if you have problems, i’ve not updated in last months and don’t know if they changed anything server side breaking my script… i’ll fix it, eventually…
as it turns out, those geniuses at blynk are doing everything they could to avoid you run your own local server on a SBC… they’re using latest version of java, which is kind of difficult to have on these little computers… i just found version 10, needed to run latest blynk server, on this site: https://www.bell-sw.com/java-for-raspberry-pi.html
Pete, hope you are recovering well. I found your site about two weeks ago from a video that Andreas Spiess had posted about your script and use of the ESP8266. I have read most of the posts regarding the script and flashing the ESP8266. Great information. Thank your for your work and sharing it with the community.
I am pretty much a novice to the electronics work. I have several ESP8266s and have tried flashing your code to a couple of them. If I can throw in a little humor, I don’t have an uncle named Bob (or the many years of experience in electronics you have) so I am having some issues with the flashing of your code.
I have downloaded the ESP8266Flasher.exe program and used that to flash the three files to the chip at the addresses specified. That appears to work ok. I see LED activity and the progress bar shows the activity. After this is complete I unplug the USB and plug it back in. I have used both the Serial Monitor from the Arduino IDE and Putty to look for serial activity and don’t see any. I have tried wiring the GPIO0 pin to ground, and GPIO to a LED. Any help/thoughts would be appreciated from anyone.
Thanks
esp8266flasher.exe appears to work well, I’ve been using that myself. Only wire GPIO to ground for programming.. ie before powering on or with reset.. with GPIO to grounde under those conditions it should program up, it should be high any time. GPIO12 is a good choice for testing with a LED.
When programming there is an additional complication that some WEMOS D1 boards need DIO mode, mostly, other boards need QIO mode, this can be set in ESP8266flasher.exe before you start programming.
Thanks for the reply. I should have said that I do intend to use BME280s. Is it possible to use Nano(s) to allow several BME280s per esp8266?
It is so good to see you making such progress in your recovery. I have had friends go through similar and strong early progress such as this is probably very heartening for you.
I am having our house’s central heating partially replaced and would really like to make the whole setup more efficient. I plan to use the excellent-looking Home Control script. I have a few questions about how to plan this. The problem is that we are having the plumbers in quite shortly and I need to have the valves in on the radiators, and some control of the boilers, but it’s all happening quite soon, before I can be certain of having a complete Home Control setup, so I am looking for a way to have some intermediate steps that will allow me to get past the plumber stage.
Eventually, I am looking forward to adding weather control, and extractor fans in various parts of the house that switch on & off depending on outside humidity. I am excited by the possibilities beyond that, but the ability to get on top of our heating & damp issues is most exciting. This project is exactly what I am after.
This is a large Victorian house. Sometimes we’re only interested in heating one room, sometimes the whole house. Gas boiler, supplying water to radiators on 3 floors.
The plan:
– 3 CH circuits, one per floor
– remote control valve on each radiator
– Bosch BME280 sensors throughout house to enable precise monitoring of temp/humidity
– The eventual aim is to have easy control of the CH system on a timer, with it being easy to override on a per-room basis, but considering that the house is basically running as two zones at the moment, I am happy to take the shortest path to having a working CH system and have fine-control sorted out at a later date.
I need to be able to get the valves installed such that the plumber is happy to sign off on them and work backwards from there.
Plan of attack (feel free to critique):
Milestone 1: all valves can be somehow set to a position that the plumbers will be able to test the system.
Milestone 2: boiler circuits can be controlled on a timer
Milestone 3: a radiator can be controlled.
Milestone 4: control of all valves, and the boiler, on a central timer per circuit. If possible, the ability to set radiators to an intermediate position.
Milestone 5: temperature sensors in each room, temperature targets in each room.
Questions:
1. What valves should I use?
1a. Normally open or closed?
2. Can the valves be partially open? This would seem to be more intuitive to me, as heating could be steadily supplying enough heat to a room to keep it at the desired temperature without cycling the boiler excessively.
3. Does the plan of attack above sound sensible?
4. Power supply considerations. This is the one that has been paralysing me slightly. How do I get power out to all these controllers & sensors?
Personally I’d be powering the valves with 220v (UK) though the alternative would be a low voltage supply for each valve, you really don’t want wires all over the place. Partially open? Well I guess that depends on what kind of valve. The alternative is manual valves (standard radiator) – definitely not per-radiator thermostatics – and zone by room or floor. I use an esp8266 in the area of zoning, I would have used a Dallas DS1802b in the past or a DHT22 (heaven forbid, don;t use a DHT11) – but today as my ESP8266 handles I2C, I plan to use BME280 boards. My Pi Node-Red software talks to the ESP8266 boards with manual override where needed.
For what its worth I used a combination of standard TRV bodies with a 220V thermal manifold https://www.aliexpress.com/item/230v-24v-normally-open-close-Electric-thermal-actuator-for-manifold-underfloor-heating-valve-radiante-actuador-M30X1/32842183873.html?spm=a2g0s.13010208.99999999.262.TpZhb0 connected to a Sonoff Basic or Sonoff TH and for those where mains power was not convenient I used battery powered bluetooth TRV’s. As all the radiators have standard TRV valves and its usually cheaper to buy complete TRV sets than just valves alone they can be installed and tested as standard TRV’s. Replacement is then a simple screw off screw on task. As for sensors you can use any combination talking to your control point but I would suggest the Xiamo Zigbee gateway and temperature sensors for the price you get a system that is cheaper than a “build your own”, good range and extensible with other sensors – all of which can then feed into Node Red for controlling the valves.
As to NO/NC it is just a matter o how you would like the valves default – I make sure mine open and close on a weekly basis to prevent sticking . Partial opening is not practical unless you cycle them rapidly which would not be good for the device and given the delay in room temperature reflecting radiator position probably not practical
Just took a look to remind myself. AnalogWrite (0-255) has nothing to do with analogRead (0-1023) and as my writing is deliberately slow for PWM, a readback function would sensibly have to read the INTENDED value not the actual value which limits its usefulness.
I want to read my PWM OUTPUT!
Can I send command nano:9,5,10?
Here is 10 an OUTPUT set by nano:9,12,10,
According to your command definition I will get back a message in JSON format on MQTT. However what is the topic of this message so I can subscribe to it?
Why I do this? Because I have some LED supplies with PWM level control. I however added a bit of logic to handle local control by touch switches to be able to set my LEDs also when MQTT is down or as an alternative operation mode without using the NodeRED dashboard. Now I only show the value sent to the LED, but I want to read the level from the output so I can see “local controlled” changes.
I’ve been using your great Script for over a year, without a hitch. Thank you !!
I just received a new Raspberry Pi 3 Model B.
I installed a hard drive as a boot drive, ran your Script, and watched as it installed without an issue.
Net result, the Script screams on a Raspberry Pi 3 Model B running off a hard drive.
I have 23 Tasmota MQTT devices on my network.
I’m so glad your back and best wishes with your recovery.
DennisB
Thank you Dennis. Glad the script works for you. Several people helped make this work reliably.
Interesting post as usual. Blogs like yours are better than many electronics magazines. I hope your recovery continues.
Great post Pete.
I hope you recover soon from your illness and that we treat you well in Spain.
I am using this for PWM controlled LED lamps via WiFi and the Nano Peripheral just works fine in contrary to the ESP8266 which has interruptions causing quite annoying light flashes. So use the nano for that purpose!
For ease of use I use Wemos D1mini with a supply from them as backbag in the same format as the D1 mini. Agreed for simple on/off Sonoff and Tasmota is very well packed in one case, no hustle to get your processors in a safe housing.
There are a few possibilities to attach a display to this and control them fom Node-Red.
So I am happy with this firmware, so keep it alive( and extend it)!
Pete did you ever tried to supply the Wemos D1 mini from the Nano Peripheral(it has a voltage regulator to 5V)?
Hi Leo
The D`1 Mini and Nano clones I use both have regulators so I can use either as the source.
So it should be possible to supply D1 mini from Nano 5V when I use 12V for Led dim and supply Nano? I cannot find current spec for this 5V, but it is AMS1117 5.0 with 0.9A current limit. I will have to try it and measure currents! Will report one of these days about my findings.
Great post as usual. Thanks Pete and welcome back.