Category Archives: Node Red

Big Timer

tmp17DBBig Timer is (probably) the best-ever timing node for Node-Red, providing a general purpose timer as well as  handling summer/winter correctly as well as (importantly) lighting up time (for which you should provide longitude and latitude). After all you probably don’t turn the outside lights on at 6pm!! You turn them on when it gets DARK.

As always - available at node-red-flows and at npmjs.com

Latest (Dec 16, 2018) additions/changes include timestamp on outputs, additional on/off inputs, improvements to "timer" input option (default secs) example:

timer X (mins)

timeoff X (mins)

Short extract from the readme file:

The ultimate Timing node for Node-Red

Purpose

BigTimer is a powerful timing node offering a range of timing facilities for Node-Red. BigTimers can be used singly or in groups. Full support for dusk/sunset dawn/sunrise and variations also day/week/month (and special days) control. The node offers outputs suitable for MQTT, speech and databases. You can also manually over-ride the UTC time setting on the host computer if required.

Usage

Suitable for general use and very powerful (yet simple), BigTimer has 3 outputs, the first of which triggers when there is a change of state and presents one of two messages (for, for example, MQTT or other control mechanism), the second output is a simple 1 or 0 every minute in the payload and also has additional outputs reflecting the status message in msg.state and message time and others. The third output presents a message which could be used for speech or debugging.

Continue reading Big Timer

Facebooktwittergoogle_pluspinterestlinkedin

The Wall Display

ESP8266 Wall DisplayThis blog has been updated as I've discovered that these displays are, after all, still freely available from AliExpress. I’ve recently spent time working on my hallway wall display, an amount of time which would be considered commercial suicide. Good job it isn’t commercial.

Of course I’ve not just been working on the pretty colours – I’ve revamped the control codes for the ESP8266-driven controller twice, discovered and fixed an OTA flaw in the code, re-hashed the Node-Red driving code, found and fixed countless other bits and pieces… you know – in other words – pretty much revamped everything just to improve a simple display.

Continue reading The Wall Display

Facebooktwittergoogle_pluspinterestlinkedin

The Script

If you are running a Raspberry PI and the script gives problems with Node-Red, note that 22/09/2018 I've updated script to use Node6.x and not Node 8.x as that currently causes issues, even the IBM script uses Node 6.x

Regular reader will know about the script that Aidan Ruff and I originally developed to put Node-Red and several other packages onto the Raspberry Pi for our own home control purposes. This has been developed with help from several people and in particular my friend Antonio.

The Script, ESP-GO and Nano Peripheral

WELL – here is the script which is intended to help set up certain Raspbian, Debian or similarly-based SBCs which now includes logging and handling Raspbian Stretch (tested on Raspberry Pi  2 and 3, 3B+). As well as it's original purpose of setting up a Raspberry Pi, the script also runs well with several other boards.  See right hand side of the above image for what the script does, given a basic operating system install. Read on…

Continue reading The Script

Facebooktwittergoogle_pluspinterestlinkedin

Waiting for Peter, closing out 2017

MrShark at the console, while we're all waiting for Peter to come back, let's try to add some new content to the blog 🙂

I've a couple of new boards to test, unfortunately they've to wait I receive their PSUs... material for an other time 🙂

So, I'll add some valuable resources found on the net in recent times, about Node-Red development...

I wish a Happy New Year to Peter and his wife, and to all of you readers... please keep visiting this blog and comment, let's leave some new material for him when he's back 🙂

Continue reading Waiting for Peter, closing out 2017

Facebooktwittergoogle_pluspinterestlinkedin

Node-Red and Alexa with Big Timer

tmp96D4I’ve recently updated my setup at home, finally doing away with HA-Bridge and I thought you might like to see this simple setup.

So for beginners, HA-Bridge is a piece of software (a very nice piece of software) that runs on for example a Raspberry Pi, who’s job (in my case) is to talk to Amazon’s Alexa and send off MQTT commands to control things.  In reality I’ve always had it return commands to Node-Red – because many of my controlled items are on timers and I need the two put together.

Continue reading Node-Red and Alexa with Big Timer

Facebooktwittergoogle_pluspinterestlinkedin

Graphing System Info in Node Red and Grafana

We were just talking about getting system info into graphs in the comments in a previous blog entry – and I realised I already had some code for getting that info in Python – and using a library that can provide a LOT more.

So – as it was there already  - I thought I’d put this quick entry together for anyone interested.

Continue reading Graphing System Info in Node Red and Grafana

Facebooktwittergoogle_pluspinterestlinkedin

More Alexa Marvels

Some of you may recall that in the dim and distant past, Aidan and I wrote a SKILL for Amazon’s Alexa – which let us do anything by simply passing the speech back to Node-Red… and we could sent text as speech to Alexa. Well, that relied on SSL port access and writing skills and… we’re still waiting for something easier and so without effort, that arbitrary response is still  a little way off, but..

If you want REAL EASY  - read on. (updated 13/10/2017)

Continue reading More Alexa Marvels

Facebooktwittergoogle_pluspinterestlinkedin

Garden Data Collection

MiFloraThis entry is all about InfluxDB, Grafana and a cute little Bluetooth garden sensor!

Some time ago a pal of mine and I did some swaps of spare kit and I ended up with a Mi Flora sensor.  A rather pleasing looking device,  the MI Flora has a 3v lithium button cell and can transmit by Bluetooth LE, the light level, temperature, battery level and moisture.

Battery is supposed to last for months. I’ve provided an AliExpress link above which puts the units at under £8 Inc. postage. No doubt that price will vary dramatically depending on where you chose to spend your money. Amazon charge more like £13.

So having pulled off the protective tab which meant the battery would work, I put the unit to one side as I had no idea how to read it.

A week or so later in Spain, with poor weather my pal Jay came on to ask me if I’d used it. I had not – and so we decided to give it a go. As it turned out I had some cheap Bluetooth 4 dongles I’d bought on a whim last year. I plugged one into my Raspberry Pi 2. The program “hcitool” lets you scan for Bluetooth devices and as it happens it was already installed on the Pi. From the command prompt:

sudo hcitool lescan

Instant success as the Mi Flora device with it’s MAC address was revealed. I made a note of the MAC address.

I would need Python 3 to experiment – but this was already installed on the Pi and some code to get the information from the unit. This code turned out to be ok with a little modification. I ended up creating a folder called /home/pi/myflora and dumping the lot in there.

Jay had modified the program to send MQTT. That of course needed the PAHO client. In order to do that I needed something called PIP.

sudo apt-get install python3-pip

sudo pip3 install paho-mqtt

Note NOT pip (which works for Python 2.7) but pip3.

I then added to the MI flora directory a test program which Jay sent me called bathroom.py – presumably because he’d put the sensor in his bathroom – I adjusted the output slightly as I was putting my unit in the garden

import sys
import paho.mqtt.client as mqtt

from miflora.miflora_poller import MiFloraPoller, \
MI_CONDUCTIVITY, MI_MOISTURE, MI_LIGHT, MI_TEMPERATURE, MI_BATTERY

poller = MiFloraPoller("C4:7C:8D:32:08:63")
print("Getting data from Mi Flora")
print("FW: {}".format(poller.firmware_version()))
print("Name: {}".format(poller.name()))
print("Temperature: {}".format(poller.parameter_value(MI_TEMPERATURE)))
print("Moisture: {}".format(poller.parameter_value(MI_MOISTURE)))
print("Light: {}".format(poller.parameter_value(MI_LIGHT)))
print("Conductivity: {}".format(poller.parameter_value(MI_CONDUCTIVITY)))
print("Battery: {}".format(poller.parameter_value(MI_BATTERY)))

# Publishing the results to MQTT
mqttc = mqtt.Client("miflorabathroom")
mqttc.username_pw_set("admin","xxxx")
mqttc.connect("192.168.1.19", 1883)
mqttc.publish("miflora/garden", "{ \"battery\" : " + str(poller.parameter_value(MI_BATTERY)) + ", \"version\" : \"" + str(poller.firmware_version()) + "\", \"sunlight\" : " + str(poller.parameter_value(MI_LIGHT)) + ", \"temperature\" : " + str(poller.parameter_value(MI_TEMPERATURE)) + ", \"moisture\" : " + str(poller.parameter_value(MI_MOISTURE)) + ", \"fertility\" : " + str(poller.parameter_value(MI_CONDUCTIVITY)) + " }")
mqttc.loop(2)
# End of MQTT section

So the idea was that running “python3 garden.py” would send some MQTT to my Mosquitto MQTT broker – and indeed it did (change MAC address for the MI Flora and MQTT details in the above).  Subscribing to “miflora/garden” did the trick – I could fire off the program and with in seconds I’d see the resulting JSON package coming out into my MQTT-Spy client.

Next job was to get Node-Red talking to the MI Flora – I used an INJECT node, set to retrigger every 15 minutes – firing off to an EXEC node which contained nothing more than “python3 /home/pi/miflora/bathroom.py”

That worked – so now I had a regular supply of data in the form of an easy to use JSON package.

I split the package up – and fed it into a Node-Red Dashboard graph – and that was my first disappointment. If you fire more than 4 items into those graphs – the popup fails to show all of them – and the graphs are not very good anyway. I think a lot more work needs to go into that.

I do use GROVESTREAMS on-line (which is REALLY flexible) to store data remotely but there’s a limit as to how far back you can read for free – that was not the case when I first started using them and wrote the Node-Red-Contrib-Grove node to make it all easy.

So now I was on a search for a decent graphics package that would run on the Raspberry Pi (or similar). NOT as easy as it sound,  unless you’re very easily pleased when it comes to graphics.

After a lot of experimenting I decided to give Grafana a shot. I think if I started writing this blog again, I would would have Grafana in the title because this really, as it turns out, is the star of the show.

Having seen a few screenshots, I had visions of populating  a SQLITE 3 database and firing that data into Grafana.

Erm, no. Despite (wait for it) using SQLITE internally to store users and dashboards, the package does not actually support SQLITE for data !!!  I've explained that now, to save you some confusion later. When I did the install, it kept referring to SQLITE and could I HELL figure out how to set up a database for my data – that’s why.

As a Pi user (that means SD use so I’m not keen on MySQL as SD has limited WRITE capability and I worry about databases shortening the life of the SD - see the later article on SD life - very informative) the options for data sources did not look appealing to me at all, the least offensive being INFLUX and that is the one I settled on – partly as there is a node for it available for Node-Red.  Despite the pretty website this is still very preliminary at version 1.02. Influx doesn’t use terms like tables and fields which kept me going for a couple of hours – but it is good at storing time-stamped data – indeed you don’t even do the time-stamping. You create an empty database and start firing data at the DB in pairs – stream name – and data value – it really is very simple once you get started. I guess you could think of it as a single table with pairs of data-name and data-value.  I went down a few dead ends and discovered if you got your stream names wrong – good luck renaming them as INFLUX does not support renaming!

Anyway, it ‘s all very easy once you get started. I did things backwards and installed Grafana first. Here’s what I did – no guarantee it will work for you.

You might want to consider reading this before using InfluxDB on the Pi for long-term use - there are very few build options for small SBCs and this article gives the impression it likes more RAM than we would normally have on the likes of the Pi. I do have to say though that it is working just fine on mine after several days of operation. Comments welcome.

I grabbed the Grafana file for my Pi from here. https://github.com/fg2it/grafana-on-raspberry

sudo dpkg -i grafana_4.1.2-1487023783_armhf.deb
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server

That was Grafana up and running on port 3000 – user name admin, password admin to actually DO anything – and quickly change that password!!! I did that, went into the ini file and disabled user registration as the admin can do that and opened up port 3000 redirect in my router so I could access this from the outside world. That all worked well.

wget http://ftp.us.debian.org/debian/pool/main/i/influxdb/influxdb_1.0.2+dfsg1-1_armhf.deb

sudo dpkg -i influxdb_1.0.2+dfsg1-1_armhf.deb

That set up influx and that ran without further ado at port 8083. I did NOT open that up to the outside world. In there I created a new empty database called “logger”

logger

In Grafana – which supports influx without any changes, I followed Engineer John’s instructions to set up the data-source and a new Dashboard.

So the next thing – would be to get some data.  I played with this for some time – should I keep incoming data source names simple – as you can’t rename them? Then Jay pointed out that in Grafana you can ALIAS names. That did it. So in influxDB node for Node-Red, you fire in a payload – and depending where you want to put your data – you use msg.measurement.. Yes, exactly, what’s wrong with msg.topic you might ask. So here is my little MI Flora MQTT node firing values into InfluxDB

MiFlora[6]

And that yellow function block…

node.status({fill:"blue",shape:"dot",text: msg.payload});

tmp=JSON.parse(msg.payload);

msg.measurement="battery"; msg.payload=tmp.battery; node.send(msg);
msg.measurement="sunlight"; msg.payload=tmp.sunlight; node.send(msg);
msg.measurement="temperature"; msg.payload=tmp.temperature; node.send(msg);
msg.measurement="moisture"; msg.payload=tmp.moisture; node.send(msg);
msg.measurement="fertility"; msg.payload=tmp.fertility; node.send(msg);

Yes, I know – far better ways to do this – but it was my first stab at it and it worked. For general use however I added an incoming function that merely copies msg.topic into msg.measurement and that means any single data-source coming in via MQTT can be blasted straight into InFluxDB.

Yes, that’s write – you don’t create tables or fields – you just send in field names and data and that’s it!

I also fed in some data from my general sensors – with horrible topics like pergola/lighting but of course thanks to the Grafana alias ability – names like that would not show up in Dashboards.

Having stored up some data it was time to get it into graphs – I took the VERY simplest approach of showing lines as I have at this point no idea what some of the Grafana functions do – I DO know that smoothing the data might be fun as Grafana relies on the data-source to do that and Influx doesn’t do that.  What IS nice is that you don’t have to worry about things like “I want the last 5 minute’s worth” or “show me the last month’s worth” as Grafana does all of that for you – from here  it gets easy.

New dashboard and under metrics ADD ROW – takes an hour or so to get to grips with this..

Grafana setup

I’ve expanded the top one here – I have 3 lines in that one graph – see the blank ALIAS BY  - if you want to change the source name (in this case battery) put something in there.

And that  is how I got to this point…

Grafana imagery - my stats

So top left you see battery status from my deep discharge battery and  the two MI Flora units, then moisture levels from the two units – light levels – and on the right centre – a whole host of temperature readings, overall humidity and finally just to play with – the UK temperature split off and displayed differently.

There’s a lot to take in – and I am as you can see NOT an expert at this yet so before firing questions back – you might want to spend some time looking at the links on this blog entry and using Google. If you think another database is better by all means write in to say why – and if you think there’s a better easy-to-fit-in-Pi graphing system, preferable one that works with SQLITE – again do let us know. I started all of this with zero information late afternoon yesterday and had it all working by the end of last night – so it’s not THAT big a deal even if my clumsy writing makes it look that way.

Not entirely sure what’s going on the this battery level on my first day – but it did occur to me that the unit is intended for putting into plant-pots – not sitting outside in the sopping rain so I’ve covered it in cling-film!! As we’re off back to the UK for a few weeks,  I don’t want to come back to a rusted heap.  Note the Mi Flora battery at the top – seems to be self-recovering!!

Update: the little Bluetooth sensors have now been running for months and the battery levels remain fine – the Bluetooth range however is crap – best to have a Bluetooth dongle nearby to talk to them – don’t even think about two thicknesses of wall!

Grafana output

Update 27/03/2017: I've been hitting issues with out of bounds temperature data – i.e. a couple of duff readings crept in - with values like 2000 and -64  which of course flattened the data completely. I realised quickly that InfluxDB has no DELETE function and as far as I could tell - things like comparing data with a value made the graph line disappear.

I put something up on the web and someone came back and asked if I was using InfluxDB 1.2 which apparently has a DELETE function. Well, I wasn't - I was way back at 1.01 - worse - the same folder where that came from only went up to 1.1

However, thanks to THIS helpful chap - I easily updated to Influx 1.2 without the issue he described. I STILL can't add something like "where value < 100" but maybe I can delete ??? Anyone with more info on this please by all means chip in...

Update 18/04/2017: I've just fitted the second of these units and also fixed a minor error in the code above.  All that is needed for a second unit is a copy of the program with different MAC address and different MQTT publish address. Sure you could use the same one twice with command line parameters but I took the easy option.

Bluetooth RANGE is becoming an issue. The Pi is in my office - breezeblock wall then the garden - the second unit I tried maybe 15ft or so from the office wall - not a chance.  Clearly BT4 is good for power, not so good for range.

So - there's a nice job for an ESP32... BT in - WIFI out.  A relay!

Update 30/04/2017: At the time of writing, InfluxDB is up to version 1.2.2 and if you follow the link I gave above but instead of 1.2.0  use 1.2.1 in the two relevant links - you will get the latest version of InfluxDB. Similarly I found Grafana 4.20 - same place as before but called https://bintray.com/fg2it/deb/download_file?file_path=main%2Fg%2Fgrafana_4.2.0_armhf.deb

I just repeated the Grafana install routine to update it using the new simpler name grafana_4.2.0_armhf.deb - all worked perfectly.

I'm pretty happy with Grafana and I now know how to delete data points (by time) but if you really want to use SQLITE and do your own thing - take a look at the excellent work that Csongor Varga has done - https://www.youtube.com/watch?v=nkKf26oKzhQ


If you like this post – please share a link to it by social media, by email with friends or on your website.
More readers means more feedback means more answers for all of us. Thank you!

Facebooktwittergoogle_pluspinterestlinkedin

Android Phone as Server

Why am I showing you a picture of a cracked-screen phone?

Linux on Android PhoneWell because this particular phone is a bust Oukitel K10000, the phone with the world’s longest-lasting battery and an excellent workhorse. My friend Aidan gave me this (rather than chucking it away) some time ago and it has been sitting doing nothing. All that is wrong with it is a cracked (and exceedingly dangerous on the fingers) screen. I’ll bet I’m not the only one with such a phone lying around wasting space.

Well, as of yesterday, it is a Debian server with all my usual stuff going on quietly in the background – with the screen normally off – running freezing cold and hopefully being super reliable.

This is an experiment only – beware – if your phone or tablet dies it is your problem….  oh and your Android phone/tablet needs to be ROOTED. 

Imagine turning your old, dust-covered phone into a sleek, battery backed-up server with unfeasibly long backup time, immunity to any mains spikes, a silent, fast Debian base for all the stuff in my script – which includes Node-Red, Apache/PHP, SQLITE (+ PHPLiteAdmin), MQTT, MC, Ha-Bridge and more!  If you’ve been following this blog you’ll know about the script.

So this article applies to ROOTED Android phones and we’ve only tested around Android 5.0 onwards.  In my case I rooted the phone with KingRoot (note, NOT the one in the PlayStore which is a con – avoid it - but the one at kingroot.net ) – please note that rooting phones is beyond the scope of this article and if you are not confortable with this you should not do it. There are a lot of links out there on the subject and many of them are fraudulent.

tmpCD52There is an APP in the Play Store called Linux Deploy. It is also on GitHub. Beware that this is essentially undocumented unless you are Russian – so please don’t ask how you can use your phone’s GPS or Sound from Debian – not a clue!

You should have a modern-ish WiFi enabled (or hardwired if you like) Android phone or tablet with at least 16GB of internal FLASH (which should give you 10GB working space).   If you only have 8GB that will leave you only 2GB which - really – isn’t enough.

Getting Debian 8 on the phone:  After ensuring your phone/tablet is rooted, Install the App.

Linux on Android PhoneIn the app, on the top LEFT menu – find  REPOSITORIES and select Debian 8 for ARM.

On the bottom right is a drop down – you should go in there and select INSTALLATION TYPE – directory (OR FILE with a reasonable file size limit – say 5GB – the 2GB default will NOT work). Change the user to “pi” with suitable password in the same menu. TICK for allowing initialisation – and use of SSH server. Also where you see INIT SYSTEM change that to sysv. 

Then TOP RIGHT menu  - INSTALL – that might take some time – top right again CONFIGURE – then bottom menu START – and you should have a working Linux you can get into with for example WINSCP for that “pi” user. The IP address is clearly shown in the App.

I suggest not going further until you are comfortable with the above – took me several attempts because I didn’t follow the above exactly (well, and Antonio and I were learning at the same time).

Running the script: Via WinSCP or similar, put the script into the pi directory – changing permissions as normal - run the script – and ensure PHONE is selected – it isn’t by default. Come back after lunch. The script will no doubt suggest a reboot. Instead, hit the STOP button on the bottom of the phone screen – wait for it complete, hit the START button – wait – and you should now have everything in the script running!

Now – I’m using this to run my script – but I’m thinking you COULD use it to serve files etc. – I’ve not tried this but I’m guessing it would be easy to access an SD card…. and make that a folder…. hmmm.

Anyway, it is now the day after I installed all this – the phone is sitting there “off” and unlike my FriendlyArm board with it’s whirling fan, is sitting silently and freezing cold yet ran the script much faster than any of my SBCs – around 40 minutes.

K10000 running Debian 8No guarantees as there just isn’t enough info about Linux Deploy out there (unless you find/translate some) – but it does seem to work well now that we’ve made sufficient alterations to the script to take this particular setup into account. A fun project, to be sure. Now, I know this is a not a fair comparison and tomorrow we might come back and say … no good (for whatever reason).. but at £107 for that particular phone… compare – how much would it costs for a Raspberry Pi 3, uninterruptable power supply able to keep the phone going for something like a couple of days with no power, a high-def touch screen, a solid case for the whole thing.. indeed ,it might be possible to use a cheap tablet…  I was looking on Ebay – 16GB Android tablet – perfectly working except for missing buttons and cracked digitiser – around £10

One thing to note – I’ve turned rsyslog messages off – it was spitting out all sorts of unwanted helpful messages when you changed brilliance on the phone or disconnected the USB etc –REALLY annoying.. If you feel you need logging on -

sudo service rsyslog start

That will stay on until the next stop-start…

Node-Red running on a K10000 phoneSuch a shame it isn’t clear how to access some of the peripherals like audio. But who knows – someone reading this article may already have solved such challenges.

Please note: the pretty colours on the right do not come as standard. Click images to see larger versions.

This is really starting to look GOOD!!!!

Revelation: I’ve now taken on-board ideas from others and thanks for that – both people in here and on Google+ – most of the other solutions are longwinded and need other packaged  so up to now Linux Deploy – I’m now installing on my old Nexus 7 2012 UTTERLY SUCCESSFULLY (though not the fastest kid on the block) ( after rooting it with the Nexus toolkit ) - using Linux Deploy (which is completely self-contained and offers full root access – is looking the best). The ONLY thing you can’t do is use the Android peripherals – because of lack of UK info but this morning I figured it all out. 

We’ve also tested this one OnePlus One (model BACON) and a Xiaomi Redmi 3 model IDO). The K10000 has now been up for several days.

Ok, bear with me – you have Node-Red on Linux – and MQTT. So, you run Tasker on the phone (in the Android section) with MQTT – and now you have access to and control of all of the Android facilities that TASKER can handle (i.e. just about the lot) from within the Debian environment. Doddle.. now all I need is some time!!

Facebooktwittergoogle_pluspinterestlinkedin