Node Red Sunday

Peter ScargillESP8266, Node Red, node-red, node.js30 Comments on Node Red Sunday

Node-RedIt is Sunday, I’ve done my 4,000 steps down to the village for coffee and back and I’m having a play.

I use Node-Red on a Raspberry Pi to control just about everything in two houses – as my confidence in the setup has grown, I’m currently keeping the Pi up 24/7 with no sign of any issues (well, it could ALWAYS be faster). Normally I control the likes of ESP8266 boards with this to do everything from monitoring temperature, pressure and humidity to controlling heating, fans, lights and more.

It occurred to me, as I’ve also had success with my script in getting Node-Red, Sqlite, MQTT and other software running on other SBCs including the really cheap new ones coming out of the woodwork that it would be nice to be able to talk to them with the same software especially those running Armbian as I’ve had a lot of success with that personally.

So I’ve made a start. I thought you might find this preliminary effort interesting.

In my controllers, MQTT is at the heart of everything and facilitates utterly reliable message sending from device to Pi and Pi to device (indeed there is no reason I could not do device to device really). I thought it might be nice to start off with a little bit of code that simulates what the ESP8266s do constantly – turn things on and off.

So typically, given a device called fred, I might turn a relay on or off using MQTT messages:

topic: fred/toesp
payload: {out0:1}

Let me first explain why I do it this way. So MQTT typically works by sending simple messages back and forth – they have a topic (address) and a payload (data).  My devices, for example fred, listen for two topics:

fred/toesp
esp

That way I can send a message to a SPECIFIC device – or to all of them at once (OTA for example)

If the device needs to send messages back they will always be of the form:

fred/fromesp/out0

in this case the payload returned would be the current state of out0.

So that represents the basics. Here I will show how a cheap controller running Node-Red can accept commands – I won’t go into the actual mechanics of turning a light on and off because for example the GPIO commands used in a Raspberry Pi won’t work on a NanoPi NEO – alternative methods are used. I’ll simply cover how to get to that point.

So here is the simplest possible setup on the Pi, to turn a couple of outputs on and off.

MQTT

This is nothing more than 4 INJECT nodes dropped onto a flow along with an MQTT output. Typically the content of one of these nodes might be:

INJECT

Simple enough. When pressed the node outputs a topic of nanopineo2/toesp and a payload of {out0:1}

This would work with one of my ESP8266 units to turn output 0 on. “nanopineo2” is just the name of one of my NEO units running Node-Red.

Here’s what’s at the other end to make this work.

Nanopi Neo

Two MQTT inputs feeding a function. So the function will work if either the generic “toesp” is called or a device-specific version. In that first function I check out the command – really simple – and will be dramatically improved soon.

[pcsh lang=”js” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]

// An attempt over time to emulate my code in ESP8266 - taking commands from MQTT
// and processing I/O
// for example message {out26:1}

var m=msg.payload;

if ((m.charAt(m.length-1)=="}") && (m.charAt(0)=="{")) // ignore if not in braces
{
    m = m.substring(1, m.length-1);
    var breakup=m.split(":");
    var getname=breakup[0];
    var sendto=breakup[1].split(",")
    
    switch (getname)
    {
    case 'out0' :  msg.topic='out0'; msg.payload=sendto[0]; node.send(msg); break;
    case 'out1' :  msg.topic='out1'; msg.payload=sendto[0]; node.send(msg); break;
    default: break;
    }
}

[/pcsh]

The output is reduced to the actual command “out0” for example – and a value. So the next item is a switch in this case selecting “out0” or “out1” and ignoring everything else. The outputs from that switch go to functions which simulate actually controlling something – like a LED for example and in my demo the colour of the black dot changes from black to red when you send a command to turn that output on.

[pcsh lang=”js” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]

if (msg.payload==1)
  { 
      msg.payload="lamp1.val=100"; node.send(msg); 
      node.status({fill:"red",shape:"dot"});      
  }
else
  {
      msg.payload="lamp1.val=0"; node.send(msg); 
      node.status({fill:"black",shape:"dot"});
  }

[/pcsh]

And that is it for now. As you can see, this can RAPIDLY be expanded to cover every port bit available on your micro, it could control PWM etc. The next step will be to take {command:arg1,arg2….arg6} and parse the arguments into an array. That will then allow PWM and other kind of controls. If you really don’t care about pricing you could do the whole lot with a bunch of Raspberry Pis… but with new Orange Pi boards coming out at under a tenner, the options are endless.

30 thoughts on “Node Red Sunday”

    1. This is not a fair test. Here, the problem is not only related to the nrf2401: the RF module is sitting on top of the MCU, which is really a bad idea. Bad design is the main problem.

      This is not to say that the nrf2401 is good, though!

      But even if the nrf2401 is using the same 2.4 GHz band as Wi-Fi, their simpler (G)FSK modulation scheme and non-existent redundancy are adding to the physical high RF absorption at these relatively short wavelengths.

      Wi-Fi OTOH is using sophisticated (and of course, more power-hungry) modulation methods including redundancy, thus its better performance at given antenna size and physical environment.

      However, the 2.4 GHz band is quite crowded in urban conditions, and with is longer wavelength and lower absorption, a 868 (or 902-928) MHz RF device with some simple FEC (Forward Error Correction, like BCH + CRC) could prove superior for low bandwidth links.

      At their relatively low price ($2.70, http://www.ebay.com/itm/311615575571), the RFM69 modules may be worth testing when power is important and/or the Wi-Fi band is too crowded.

      Then is the problem that there is no standard protocol for the ISM band, and all OS implementations I have seen so far are too simplistic.

      1. The RFM69 radios are excellent, and sip power. I don’t use the HW variety to save on power and I can pick up the signal throughout the house.

        The problem with the ESP is they are power hogs. Not everyone wants to have every sensor near a power socket, or run cables through walls. I have a temperature sensor in each room of the house, they run on a couple of AA’s for a year and no cables running to them. They are wife friendly, and look like part of the furniture.

        It is simple enough the write a bit of firmware to bridge the radio’s to mqtt.

        Glen.

          2. Mentioned it earlier……

            – Go to the Arduino IDE
            – Install via “Sketch->Include Library->Manage Libraries” and filter on “mysensors” to install (its all part of the Arduino core now)
            – Select sketch “File->Examples->MySensors->GatewayESP8266MQTTClient” and uncomment line
            //#define MY_RADIO_RFM69
            (Also comment out the line #define MY_RADIO_NRF24)
            Wire up your radios with the sensor “node” of choice and that’s it!
            Works a treat TBH

        2. I remember the RFM69s now… yes and cheap from China of course. An interesting project would be using an ESP8266 to control a small network of them… and pass the info back via MQTT – would not want to go back to using Arduinos…

            1. Silicon labs – THAT’s the one – I still have some here attached to Arduino-like 1284-based jobs – that was if you like at the tale-end of my research before I switched lock-stock and barrel over to ESPs – they had a decent range – the spring-like aerial wasn’t the best thing in the world to look at but they seemed quite good.

              I did note that the ESP to radio software someone linked to here – with MQTT – did only seem to be one way – reading incoming packages – and I was thinking… but what if the package doesn’t get through – really need a transparent system that retries until it gets through.

              1. As mentioned earlier, the difficulty with the ISM band RF is that there is no standard protocol. This means that you are free to do what you want, but then, you loose all the pre-crunched work that is done under the hood by these standards like Wi-Fi, Bluetooth, Zigbee, etc.

                I mean, it is easy to send / receive a few bytes over the air, it is much more involved to get it working reliably.

                I mentioned FEC (Forward Error Correction) in a previous post. If you don’t see the point, check quickly what it can bring to you on slide 21 here (the whole document is worth reading though!):
                http://atlantarf.com/Error_Control.php

  4. Coming back to node-red 😉
    I’ve got imperihome working with node-red as a controller (as per a previous thread here). Node-red is great as a simple controller but I would like more as follows:
    – A better onboarding process e.g. press a button instead of coding Javascript . This could even include OTA programming to personalise the device via UDP perhaps e.g.
    1) Switch on device and connect to wifi (wifi manager?)
    2) Go to controller and it sees “device requesting connection”, use dropdown to say what it is to be and press “go” to program it -the device is then “claimed”
    3) device now behaves and communicates as you want, and is automatically seen in Imperihome etc
    – Ability to easily create “scenes” ideally with a drag and drop editor (NOT CODING!!!).
    e.g. input sensor – turns on light device (a bit like node red – but at the node level (i.e. not a generic MQTT node). i.e. each node is a picture of the sensor(!)
    – Have the display part (e.g. Android app) locked down with different views for guests/master.

    OK, unless mistaken, one of the main “concept” difference between Pete’s code ideas and Mysensors is:
    – Mysensors standardise at a node level and allow the user to code at the node.
    – Pete’s code goes deeper than that and MQTT codes at the device level (without the user being able to tinker at the node)

    I’m thinking that the ulitmate would be a combination of the above i.e. no programming at the node, but further standardisation on the MQTT at node level, so that an “off the shelf” controller can easily program the sensor device and talk to it thereafter. The goal is to be no programming needed
    Just some thoughts….

    1. While I would agree Node-Red is simple to use, it is FAR from simple… I’ve done some pretty complicated stuff spread over many pages and still the little Raspberry Pi copes.

      I took a good look through the MySensor stuff – I didn’t really see anything special. I think I’ve taken enough away to be able to add the more useful ideas to my own stuff. Erm, press a button – an MQTT message is published – doesn’t get any simpler. In my case that’s on GPIO2 or 14. I’ve seen the systems with scenes – and they were so complicated for those who didn’t design them – that’s what led me to Node-Red in the first place 🙂 If you go to http://flows.nodered.org/ you will see one hell of a lot of both nodes – and already written flows to save you having to do much if any coding.

      When a device logs into my node-red-contrib-esplogin node, it gets stuff like time and date etc automatically from that point onwards – and that is logged in a database – so it would be trivial to use that info to make things happen automatically – especially as the web/serial info that the ESP can send includes some currently unused status space.

      But then as this is all available as source code – I for one would be happy to see merge requests with new stuff… I have some improvements to look at to my script today thanks to one of the readers in here. If anyone has ideas for widely uses sensors I’ve not included – by all means fire suggestions – there’s a long winter coming up..

  5. My thoughts regarding the mysensor.org project and following discussion:
    – the recommended NRF modules are really bad; however, it does also support the better RF69 ones (Semtech SX1231 chips)
    – the ASCII protocol is a joke: at the price of human readability, you could favorably replace this with a binary protocol and improve time-on-air by a factor of at least 5x. This means much better resilience to errors and/or spurious emissions, less collisions, etc.
    – there still may be a need for under-powered nodes, where only a small battery is desirable, and where a sub-GHz RF would prove useful. But at home, most of the controlled/monitored devices are near a wall socket. The problem is more to find a small AC/DC module to power the nodes from mains (HLK-PM01 modules may prove useful)
    – magnetometer chips (HMC5883L, MAG3110 or even 9-DoF MPU9250) may be used to monitor door / window opening / closing, much better than orientation sensors
    – regarding Cloud-based voice recognition services: I feel really nervous about letting any of them listening to my conversations with an always open microphone…

  6. Hi

    Another thing you should check out (if you aren’t already) is voice control via the Amazon Echo. One of those devices that I wasn’t sure I would use but now find invaluable. Using HA-Bridge I can emulate Hue and WeMo devices using their inbuilt skills to send MQTT messages to my Sonoff switches for on and off and dimming commands to other devices. You can use a variety of commands such as arm,lock,on etc. and any name you like for the device. If that is too restrictive you can build your own skills on the AWS platform. Integration into your current software should be very straightforward and she even says “Sleep tight” to you when you go to bed.

    Steve

    1. Hi Steve

      I have one of those SEEED gadgets just waiting to be populated with Amazon Echo info. I did the other day do some comparison between Amazon Echo speech recognition and Google – and I’m afraid the latter – at least with my relatively accent-free voice, wins hands down. I’m hoping Google are going to get rid of the restrictions they put on using their API so that perhaps a PI can use Google voice again… I said “weather in Galera, Spain” and Google immediately figured out that the most likely name of a place in Spain that sounds like that is “Galera” – Amazon on the other hand consistently got it wrong. What I’d like is a simple voice – to Node-Red interface – sending the text in a payload to Node-Red via whatever means (mqtt or if already on the Pi, another means). I could then do what I want with the speech. Back home in the UK I do have a kind of solution – an old Android phone using TASKER picks up Google speech input and diverts it – but it is not perfect (someone night have addressed that now) and the over-ride, stopping Google from further interpreting the data doesn’t always work. But generally speaking yes I am interested in reliable, accurate voice control that I can feed into Node-Red. I had 90-odd percent accuracy with a little parser I built so I could say “ok google turn the temperature down 1 degree”. Once filtered that came to “temperature down 1” – which is enough to then control the heat….

  7. OK Last post – This video to see info on their MQTT message structure.
    (N.B. on arduino – more pins)

    The main thing is this – these guys have done all the hard work of standardising things as per here
    https://www.mysensors.org/download/serial_api_20
    e.g. a “temp sensor” is S_TEMP (value 6)
    a “RGB light” is S_RGB_LIGHT (value 26)
    The video expains how the MQTT messaging can be deciphered (look at 8 mins in)
    I’m just thinking “Why re-invent the wheel” – also, all the people on that site will be able to use your stuff without re-training on the standards (if you see what I mean)
    Perhaps another way to do things is to look at their node in node-red.
    http://flows.nodered.org/node/node-red-contrib-mysensors
    I mean this genuinely Pete – If you want to keep existing structures then thats fine of course

    1. Thanks for this Pete- I’m sure people will find the sensor info interesting – but I have to say – I would not now touch those NRF radios with a bargepole – for months I was lured by the low cost.

      They are now hardly any cheaper than the infinitely better ESP-12 modules which are powerful enough not only to handle the WIFI connection and MQTT but also just about anything an Arduino can do – I wonder if any of those libraries have been tested on the ESP8266 devices. I do of course understand that WIFI control is not the only way to do things – but it’s the route I’ve taken in all my work in here.

      Eventually I came to the conclusion that with no ability to read and write at the same time – and no ability to read their own signal strength, the NRFs are utterly ill-suited to reliable networking – and trust me I did not come to that conclusion lightly – my old and now defunct blog at scargill.wordpress.com is a testament to the amount of work I put into them.

      If you want to see some of the struggles I had – go to http://www.scargill.net – on the left after the archives is a search box – put in NRF24L01.

      I used at that time the NRF24NETWORK which had extremely limited ability to recover from lost signals – then moved onto much better mesh systems (RADIOHEAD) but the basic flaws in the NRF24L01 remain and I note there has not been much updating of the libraries of late.

      After much wasted time and after gaining a good understanding of these radios – I abandoned them. Through certain types of wall their range can become as low as several feet – which makes anything but PROPER mesh somewhat impractical. I’ll grant you that battery operation, where needed might put a different slant on this but then I’ve seen people get good battery life out of ESPs.

      For those who simply must use radios, the best mesh system I’ve come across is RADIOHEAD – if you check out this link…

      http://www.airspayce.com/mikem/arduino/RadioHead/index.html

      You’ll see that a wide range of radio systems is supported many of them being FAR, FAR better than the NRFs. The American boards tend to be expensive when you take tax and postage into account but there are one or two from China which are very good. the code is intended for Arduinos but I guess with a little work could be modified to work with an ESP8266 hence providing a WIFI to RF gateway.

      I’m not sure how many people are aware of this but the ESP chip HAS a mesh network available in the API. I’ve not used it myself but it does follow a similar design to the NRFs, the difference being of course that the ESPs have full signal strength detection and FAR better range.

      As a result of this – my own view is that overall, a WIFI ESP solution for most purposes beats a radio/Arduino solution hands down – the only shortfall being the number of pins available on the ESP – which is why I’ve started down the route of using I2c.

      Port expansion and temperature/pressure/humidity/ADC/PWM are all covered in other blog entries in here and with help from the community I have drivers in the code for many such devices.

      Looking at the list of items in mysensors there are one of two ideas I will incorporate into my own software as you’ll see below. I am hoping that the ESP32 units once out there and competitive drop in price dramatically as it should be possible to port much of the work I’ve done and the extra FLASH, RAM and port capability will open up fantastic new opportunities not that I’m anywhere near out of space yet but I am a little short of program RAM.

      Looking at the MySensors “Sensors and Actuators”… some comments.. there will be readers in here who’ve not yet gone through the home control work that Aidan and I have done and may not be aware of everything that the software we’ve put up on the web can do or why we did it a particular way – so here goes…

      Air humidity sensor. Bad choice of device – that blue DHT11 is rubbish – put several together and compare values and you’ll see that both temperature and humidity vary WILDLY – this is easily fixed – use slightly more expensive DHT22 which are almost code compatible and MUCH more accurate. The Home Control 2016 software handles both and the Dallas chip – and via I2c, several others.

      Atmospheric pressure – already handling that – but I prefer the BME085 which also does temperature and humidity all in one package. But I can handle all of them via I2c.

      Bed occupancy – simple enough to implement on any system – but a good idea. I would have thought another way to handle this would be a simple, cheap IR movement sensor pointing down?

      LED dimming – one of our boards has three MOSFETS and the ESP is able to handle 14 bit+ PWM. The normal 8-bit PWM used in the MySensors system is AWFUL at low levels – you can CLEARLY see each step. A better way to do it would be at the cost of a timer, use the advanced PWM code I described in an earlier blog which gives the Arduino the ability to do TWO PWM outputs at 16 bits (a real shame 3 isn’t possible).

      Of course you would not use that many levels – what I do is put that through lookup tables to get, say 256 levels, weighted so that at the low end you’re looking at just a couple of values change per step and at the higher end, changes of thousands per step. This gives a smooth looking PWM, important for example if you are implementing a slowly changing SAD light. I’ve also implemented code for RGB lights to give control over colour temperature (white). All of that is detailed in the Home Control 2016 manual.

      Display and time – I recently implemented various LCD and OLED drivers and the 2 line version is one of them. The boards in my setup are all aware of time, date, dusk and dawn so that’s done.

      Distance – I’ve not implemented that yet and that would make a nice addition.

      Gas detection – as the boards generally available for that feature as well as analog output – a comparator output – that’s effectively available.

      Gesture control – those devices are easy to use – something to think about over the winter.

      IR send/receive – I got into that a while ago – IR send is trivial – running IR receive and doing anything else is quite difficult – even on the Arduinos. Eventually as the number of good phone apps started to increase (Imperihome, Blynk and others) I decided to concentrate on controlling everything by mobile phone as most of us have these anyway. Something I’ve been thinking of for a while – when discussing my I2c universal peripheral – is to think about taking the best playback library, putting it on one of those £1.80 mini-Arduinos and talking to it via I2c. Another winter job I think.

      Light level – well, I’d not actually noticed that sensor before but as it is I2c, I have now ordered one from AliExpress – at 98 pence would you believe – it should be with me in 3 weeks and I’ll add that to the I2c sensors the system handles. Nice one.

      Motion sensing: There are inputs on the ESP which in my software can trigger MQTT messages – I already use this to control room lighting.

      Orientation sensing – in a home control environment I’m not sure I see a use for this. Same with parking sensors.

      Pulse power meter. The reason I set up input counters in the first place was for this very reason – to count the pulses from the electricity meter. Reading this into Node-Red at regular intervals and a quick calculation would easily allow for monitoring electricity use. The difficult bit is sticking the sensor onto the meter – not sure how the electricity reader guy would feel about that 🙂

      Looking through the rest:

      Relay control – obviously done including timeouts.

      Soil moisture – implementing that is trivial – choosing the right sensor is the trick. The sensors described in the mysensors page would not last five minutes in a real soil environment – blogged about that ages ago and readers generally came to the conclusion that you need a more expensive capacitive system where the actual material is shielded from the corrosive soil.

      Range gauge – I initially had this on the counting inputs I’ve referred to above – but at the end after telling everyone how much rain had fallen only to realise no-one really cared and could get that from an App on the phone, I abandoned that.

      RFID – useful for access of course – but then complete access systems are so cheap.

      Scene controller – some time ago I added in the use of NEXTION displays to the setup and already use them for controlling lights, heating etc. A scene controller is merely a variation on that…

      Temperature and humidity covered as mentioned above.

      UV – I’m spending the winter in the UK – there won’t be any UV – but here in Spain – yes I could see that. Even from AliExpress that ~MySensor sensor is expensive…. I wonder if this (see link) might be a better bet… As this gives an analog output – merely connecting it to the analog input on the ESP and reading that by the standard analog input command – would give you UV detection. Accuracy on this chip looks good and it is cheap… https://goo.gl/81MM4Y.

      Well of nothing else I hope this conversation has triggered a few ideas.

      1. i’ve seen orientation sensors used for garage doors, to read their inclination instead of just open/closed… and parking sensors in the same situation… imagine a semaphore with standard colors to tell you when you reached the wall… but, hey, in more than 20 years of driving, i know the “bounderies” of my car and it’s veeeery rare that i touch something, on sides, front or back 🙂

      2. typo: BME085 doesn’t exist, i think you refer to BME280, as BMP085 is old and superseded by BMP180… but the most complete is BME280, right now

        1. Sorry – yes absolutely I meant the BME180 or 280 (I don’t have one handy but it is documented in my doc file) – works a treat).

  9. Here’s a thought for you – One of the “Mysensors” programs is an “ESP8266 MQTT gateway”.
    Is it possible you could adopt this program to use your MQTT? (or other way round)
    https://www.mysensors.org/download/serial_api_20
    If you could do this, your work would be opened up to a wider audience and you’d gain the benefit of the Mysensors “platform” e.g. their agreement with various home controllers (over 20 now).

    1. The first thing I noticed when I went into that website was something about using NRF24L01 radios – I would never want to get into a conversation with anyone about issues with them – they are horrible and I wasted a lot of time on them before ultimately realising WHY they are horrible. I noted also the software is designed to handle sleeping devices and they talk about having radio meshes – check through the blog – been through all of that and come to the realisation that, at least in the direction I want to follow, for reliability – WIFI – not meshes – and always on – I know that doesn’t work for everyone and by using MQTT storage I could actually handle sleeping devices – but at this point I’d like to concentrate on getting more functionality out of the ESPs (at some point I’ll start tackling ESP-32) and low-cost SBCs all over either wired or wireless Ethernet.

      Erm, where is the bit about MQTT??

      Actually as my setup is based not only on MQTT but also Node-Red – an interface between most systems and MQTT would be a matter of connecting two nodes together. In this blog I’ve just looked at a very simple interface between two SBCs, in fact, the command set and the number of peripherals is very enlarged compared to the simple example in here. See the Home Control 2016 article. I talk mainly via MQTT to devices but think about it – mqtt input and output nodes, i/o nodes for some kind of radio devices and two DIY functions in between offers pretty much unlimited expansion.

      1. I’m thinking with relatively few changes, you could leverage all the hard work they’ve done to benefit your solution. In partcular, the work they’ve done for MQTT with controllers like OpenHab (and also their version of a NODE -RED controller)

        To check it out, all you have to do In the Arduino IDE is install via
        “Sketch->Include Library->Manage Libraries”
        and filter on “mysensors” to install (its all part of the Arduino core now)

        The MQTT sketch to look at is “File->Examples->MySensors->GatewayESP8266MQTTClient”
        What this does – is use MQTT to communicate to their selectino of controllers.
        What I’m thinking is that if you mimiced this format, you could utilise all the hard work they’ve done with the APIs i.e. you could use the MQTT controllers like OpenHab 🙂
        Why do all this? – well some of the the home automation controllers are able to create “scenes” with a very nice drag and drop editor.
        In the end, all you have to do is mimic their idea of sending MQTT messages!

      2. One of the advantages of the nrf24 in the MySensors ecosystem is low power sensors (long life battery powered sensors). I have built several 4 way sensors (light, motion, temp & hum) powered from a single 3V battery and they have lasted 20 months and still going strong. The MySensors project allows for the use of different radios and the ESP8266 using the same protocol. I use sonoffs as well as arduinos within my system.

        1. There are many low power transceivers out there, not much more expensive than the NRFs and WAY better range and reliability. I spent a LOT of time on those things – never again. Very little buffering, no simultaneous TX/RX, no means to measure signal strength hence no way to decide which is the best in a mesh. Each to their own and if people have had success with them – that’s good to hear. I could not honestly recommend anyone to go down that route. If lasting months on a battery is important then some of the other radios out there might be well worth looking at – for me, reliably returning or accepting data in a crowded radio environment is important. I did a lot of tests with the NRFs sending and receiving data constantly – I think the longest I ever got without an intermittent failure was 24 hours. Anyway, expect to see some sensors from MySensors added to Home Control 2016 soon. I think light and UV would make useful additions to pressure, temperature and humidity.

Comments are closed.