This is a major re-write of a year-old blog. At the time, I simply wanted to store a global object in a file and retrieve it. Things have changed, I realise the way we did this was awful – hence the re-write.
One way is to save the object as a persistent message in MQTT – another to a database, another to a file. I like the idea of a file as somehow it just seems that this will lead to the least WRITES – and that might be important for SD.
Why would you want to do this?
Before I forget – you MUST alter the Node-Red settings.js file to REQUIRE fs as in..
You only need to do this if you are NOT using “the-script” as that does it for you.
Well, specifically my own requirement is to save the state of a home control object which has all sorts of info about a heading system. Let’s call the object “thermostat” and it may have all sorts of innards -“thermostat.temperature”, “thermostat.set”, “thermostat.fallback” etc. In my case about 20 items. Not something you want to split up into individual variables. I used this as I have 2 parts of the house with separate heating and so I could duplicate all the code simply using two separate objects. I need to store them on disk when changes are made – and retrieve them on power up.
And this is where the original article went wrong…
The easy and obvious way is to use an inject node which runs once only on startup – to feed a file function – which gets the contents of the file – and stores them into an object. Now – you need to do this in the LEFT-MOST tab in Node-Red as this is executed first – before any other tabs – so you get to define the object BEFORE it gets used.
That’s not the end of it of course – what happens if this is a new installation and the file does not exist? Well, in that case, you check for the “undefined” object coming out of the file node – create the object with defaults – and store the result in the (new) file. Easy. And that’s what this article was originally about.
Except that on close inspection – it didn’t work. Originally, Node-Red didn’t guarantee that the left-most tab would work first – now it does run first. But that still didn’t get it – when I started looking – there were (non-fatal) error messages showing that clearly the object was being used before it had been defined!
Now, why would that be? The answer lies in the fact that (just about) everything in Node-Red is asynchronous – i.e. it does not hang around to wait. So here I was using a file input node assuming the file would be opened – and my function would be executed before anything else. Nooooo. The file node accepts the command to open a file – but then while it is waiting for this – other things can happen – that is, other flows can run which is EXACTLY what I did NOT want to happen. My thanks to the guys at the Node-Red Google forums for that insight.
The solution is simple once you now what the problem is. Firstly we need a function (which I made into a sub-flow) which takes in a file path, opens up the file and returns either the contents of the file in the payload or returns “undefined” if the file is not there – just as the normal file-in node would do… but synchronously.
There is also the matter of SAVING an updated object – I’ve used a second timer for that – one which checks every second – but NOT at the start. It simply outputs the text “check. So – at the start, the file is read, if it is not there, “undefined” is returned.
If “undefined” then create the object and store it. If defined, read it. If “check” – check if object.counter is true and if so decrement it – if now zero, update the object to disk. By doing this all in the same place, a lof of code duplication is avoided. Indeed as I have TWO thermostats I can use the same code with a different object and re-use one of the timers.
Read file sync is a sub-flow – could just as easily be a function – but I put it in a sub-flow as the content is the same every time – it is the initial inject that is passing the file name. The two (orange functions are the same other than the actual object name (thermostat and office_thermostat).
Here’s the content of the function at the heart of Read File Sync
[pcsh lang=”js” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]
var fs=global.get("fs");
try {
msg.payload = fs.readFileSync(msg.payload).toString();
}
catch (ex)
{
msg.payload=undefined;
}
return msg;
[/pcsh]
And here is my code in the orange top function – the brown item on the right is just a file write node. Clearly this is for my particular application but you should get the general idea..
[pcsh lang=”js” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]
if (msg.payload===undefined) // if the file was not valid - create the object, predefine, store
{
var stat = {
max_humidity : 100,
display_temperature : 22,
desired : 22,
display_humidity : 45,
heating : 0,
display_external_temperature : 14,
display_external_humidity : 54,
display_status : "Normal",
display_enabled : 0,
display_last_page : "home",
display_last_colour : 0,
pass : "",
display_dim : 60,
frost : 14,
hold : 0,
p1 : 180,
p2 : 480,
p3 : 600,
p4 : 1080,
p5 : 1380,
pw1 : 180,
pw2 : 480,
pw3 : 600,
pw4 : 1080,
pw5 : 1380,
t1 : 16,
t2 : 21,
t3 : 19,
t4 : 21,
t5 : 18,
tw1 : 16,
tw2 : 21,
tw3 : 19,
tw4 : 21,
tw5 : 18,
temperature_offset : 0,
last_state : -1,
current_state : 0,
update : 0,
daynow: 1
};
global.set("thermostat",stat);
msg.payload=JSON.stringify(stat);
return msg;
}
else if (msg.payload=="check") // this is the every second check - assuming file exists
{
if (global.get("thermostat.update")!==0)
{
global.set("thermostat.update",global.get("thermostat.update")-1);
if (global.get("thermostat.update")===0)
{
msg.payload=JSON.stringify(global.get("thermostat"));
return msg;
}
}
}
else // read in the file if it was valid - don't output anything
{
global.set("thermostat",JSON.parse(msg.payload));
global.set("thermostat.laststate",-1); // detecting change of timezone
global.set("thermostat.temperature_offset",0);
}
[/pcsh]
Not sure how useful you feel this is for your projects compared to a database – but it is simple and fast. The point of the global object, of course, is that you can use it on any page in your node-red setup. Certainly, by careful use of the timer when updating, you can vastly decrease the number of WRITES compared to a normal database setup. Handy for those of us on a Raspberry Pi with limited SD writes.
There are those who put their hands up in horror at the idea of globals – I do not. I like the idea of having the ability to view or update things from any tab (flow) with ease. Real world apps? I’ve been using this (bearing in mind the initial issue with start up values just resolved) in a holiday rental since April – with people in all the time – running the heating and some lighting – not one issue.
Hi Peter,
Great post. Please edit your post to provide a warning about the required settings.js modification to require fs. Instead of immediately reading the comments I started debugging why it wasn’t working myself, lost some time but gained some knowledge.
Done.
in case you use THE SCRIPT, this mod is already there
What is THE SCRIPT?
link.click(right_column);
while (!understood) {
read();
}
Or… https://tech.scargill.net/a-christmas-script/
But I suggest you also look at….
https://tech.scargill.net/home-control-2016/
Firstly thanks for all the brilliant posts, they’ve helped a complete beginner like me to get pretty far with my Raspi heating controller. However I’m struggling to get your variable storage flows to fully work. The Read File Sync node (sub-flow in yours, function node in mine) always outputs “undefined” as it can’t get the global variable “fs”. Is there somewhere else in your flows where the global variable “fs” is set?
Sorry I kind of assume every one has that.
In your /home/pi/.node-red directory – the settings.js file…
at the start
var fs = require(“fs”);
then further down here is my functionGlobalContext section
functionGlobalContext: {
os:require(‘os’),
moment:require(‘moment’),
fs:require(‘fs’),
mySettings:mySettings
},
and that is how I am able to access FS and a load of other things – from Node Red !!
Thanks for that amazingly quick response!!! The Read file sync node is working!! Sorry for such a basic question, but I’m completely new to programming of any kind.
The function Read File Sync does not work for me. It is only returning undefined. Tested on Linux and Windows.
Reading the same file with the File node works fine.
Any idea or comments?
V0.16.2 npm
>> Now – you need to do this in the LEFT-MOST tab in Node-Red as this is executed first – before any other tabs – so you get to define the object BEFORE it gets used.
Quoted part of post together with using “readFileSync()” helped me lot. I would like to share my findings for case that the same left-most tab contains multiple “inject” nodes:
– the “inject” node that appears first in flow definition, that is within flows file – not topologically, will be executed first.
I’m currently reading “The Node Craftsman book” – very interesting seeing the differing ways you can handle objects in Javascript, though right now having cracked simple objects, my big bottleneck is image and font loading. I have both a custom font and images in my current gauge and ensuring they are both fully loaded is proving somewhat challenging. This didn’t show up at home, using the locally but as I’m now sitting in (sometimes) sunny Spain with the attendant delays in accessing files from my home server, the initial load-up issue is becoming apparent. I’ve tried some obvious techniques and they’ve not done me any good up to now.
maybe you should do a post on recommended readings, just to have a central place to list them, instead of comments here?
Hmm, a page might be better – but I’m running out of room on the menu for logged-in members…
maybe incorporate the ABOUT page as a link MORE DETAILS or similar, under the already present mini description on the right? Or enable submenus?
By the way, do you have some links that are helpful for learning node-red. The official site is great as a reference but not as useful to me for learning. Be great if someone had a video course (for a cost or for free would be ok)
That is possibly something we can fire back at others – guys – those of you who have learned Node-Red – any recommendations for good learning sites?
video, don’t know… node-red itself is quite easy, once you learn how to install and connect nodes, what is really needed is programming in javascript, the way NODEJS is used to be programmed… the more in depth you go, the more you need programming… so, to start, free:
https://leanpub.com/nodecraftsman (it’s a long free following to the commercial “The Node Beginner Book”)
http://www.hivemq.com/mqtt-essentials/ (best MQTT intro)
javascript, well, just use google, it’s FULL of resources…
then pay:
the book by Rui Santos: https://randomnerdtutorials.com/getting-started-with-node-red-on-raspberry-pi/ (link in article)
https://www.amazon.com/Wiring-IoT-Connecting-Hardware-Raspberry/dp/1491953330/ (still to be released)
https://www.amazon.com/s/field-keywords=a+smarter+way+to+learn (javascript books by Mark Myers)
and, youtube…
I’d also recommend Javascript Principles https://www.amazon.co.uk/gp/product/B00JFN7MD6/ref=oh_aui_d_detailpage_o03_?ie=UTF8&psc=1
The English can be a little clumsy and there are a few annoying typos but it takes Javascript from the aspect of Jquery and it explains what is going on under the hood – which makes it easier to grasp context and the structure of the environment. Also it is only 99p on the KIndle so no great loss if its not right for you.
Thanks for your prompt response Pete.
You make a good point about project motivation. Most are probably for their own home projects and they can easily make changes themselves. If I set up some way to access the client’s system externally, I could log in and make changes for them (I don’t want them fiddling with the editor as they are farmers and blue-collar workers typically). The system will have a 3G dongle, I suppose if I could set up a static IP on the cellular then I could access the system to make changes?
1) having a constant access to some one other system could be very “unhealthy” for yourself… what about if they accuse you of something?
2) most 3G networks are NATted, so you don’t have a public ip, or, best, you have one shared with many others, and worst of all, it changes… imagine a large private network like an office, or your home, that’s what you get, your 3G ip is as your router ip, but you can’t do port forward…
3) possible solution, if you want to go this way: https://www.dataplicity.com/
I use a Telegram Bot to query my Raspberry Pi for it’s external IP address.
no need for that in this case, just use a dyndns service provider, like noip…
Thank you for your post, this is very helpful.
I am also new to node-red. Current I am working on a network of temperature sensors and looking for some help.
Firstly, I want the user to be able to make changes to the system parameters without giving them access to editor (I password protect editor and only give access to dashboard). How would I let the user change things like High alarm and low alarm values and title for the sensor. Can they fill in these details and use the File node to save them?
Also, what about the situation they want to add another sensor to their system? Is there any way the system can create a new flow from an action in the dashboard or maybe have flows ready to go but not “enabled” or made visible when they are added?
I have one password on the editor another on the UI. I also use HTTPS for external access. If you look at my recent thermostat project, setting of high and low values for the entire week along with temperatures is done via the UI on a phone. For adding new stuff, just go into Node-Red and add nodes etc, really easy. My projects are oriented to people who do their own stuff and maybe want the wife/husband/boyfriend/girlfriend to control stuff like lights and heating on a nice interface. I don’t concentrate on making drop-in products for sale as that’s not that interesting. Of course – once you get used to Node-Red and spend the time to get skilled in it, the sky is the limit – expecially thanks to the UI which allows for templates which can contain whatever you want within reason.
Hi Peter,
I hope you can help me. I’m a novice user of Node-Red (migrating from EventGhost) and am struggling to figure out how I can persist the states of my various lights and sockets into a JSON formatted string in a file. I’ve been reading your various blog articles for hours, and searched documentation on global.set and global.get functions, plus associative arrays in js and I’m getting knowhere fast. What I’m aiming to do is have the msg.topic as the key and the msg.payload as the value for each entry in the array, then JSON.stringify() it and put it in a file that AutoTools on my Android phone can subsequently read. I’d be very grateful of any help and advice you can offer.
Seems to me you are already there – so for example in my heating controls I stringify the object and simply save it to a file using the file node. On power up using an inject node set to inject only once on power up – I read back and json.parse the string back into an object. As for getting that onto your phone – my scenario keeps node-red up to date – not the phone.
For what I’m doing, it doesn’t even need to be as complicated as that. To keep track of the status of 4 lamps and report those via LED controls on the Blynk dashboard, it looks like all I need to do is use the File node on its own to write either 0 or 1 to a file called lamp1status (or lamp2status, lamp3status, etc).
I can then have a button on the Blynk dashboard called “Get Status” which tells Node Red to read each of the 4 files and inject the 0 or 1 into the Blynk node that controls each LED on the dashboard.
Yes, that would work – but as it’s now written – may as well use it – you never know, you might add another lamp and a display and….