Node-Red Objects to File and Back

This is a major re-write of a year-old blog. At the time, I simply wanted to store a global object in a file and retrieve it. Things have changed, I realise the way we did this was awful – hence the re-write.

One way is to save the object as a persistent message in MQTT - another to a database, another to a file. I like the idea of a file as somehow it just seems that this will lead to the least WRITES – and that might be important for SD.

Why would you want to do this? Well, specifically my own requirement is to save the state of a home control object which has all sorts of info about a heading system. Let's call the object "thermostat" and it may have all sorts of innards -"thermostat.temperature", "thermostat.set", "thermostat.fallback" etc. In my case about 20 items. Not something you want to split up into individual variables.  I used this as I have 2 parts of the house with separate heating and so I could duplicate all the code simply using two separate objects. I need to store them on disk when changes are made - and retrieve them on power up.

And this is where the original article went wrong...

The easy and obvious way is to use an inject node which runs once only on startup - to feed a file function - which gets the contents of the file - and stores them into an object. Now - you need to do this in the LEFT-MOST tab in Node-Red as this is executed first - before any other tabs - so you get to define the object BEFORE it gets used.

That's not the end of it of course - what happens if this is a new installation and the file does not exist? Well, in that case, you check for the "undefined" object coming out of the file node - create the object with defaults - and store the result in the (new) file. Easy. And that's what this article was originally about.

Except that on close inspection - it didn't work. Originally, Node-Red didn't guarantee that the left-most tab would work first - now it does run first. But that still didn't get it - when I started looking - there were (non-fatal) error messages showing that clearly the object was being used before it had been defined!

Now, why would that be? The answer lies in the fact that (just about) everything in Node-Red is asynchronous - i.e. it does not hang around to wait. So here I was using a file input node assuming the file would be opened - and my function would be executed before anything else. Nooooo.  The file node accepts the command to open a file - but then while it is waiting for this - other things can happen - that is, other flows can run which is EXACTLY what I did NOT want to happen. My thanks to the guys at the Node-Red Google forums for that insight.

The solution is simple once you now what the problem is. Firstly we need a function (which I made into a sub-flow) which takes in a file path, opens up the file and returns either the contents of the file in the payload or returns "undefined" if the file is not there - just as the normal file-in node would do... but synchronously.

There is also the matter of SAVING an updated object – I’ve used a second timer for that – one which checks every second – but NOT at the start. It simply outputs the text “check. So – at the start, the file is read, if it is not there, “undefined” is returned.

If “undefined” then create the object and store it. If defined, read it. If “check” – check if object.counter is true and if so decrement it – if now zero, update the object to disk. By doing this all in the same place, a lof of code duplication is avoided. Indeed as I have TWO thermostats I can use the same code with a different object and re-use one of the timers.

update objects to and from disk

Read file sync is a sub-flow – could just as easily be a function – but I put it in a sub-flow as the content is the same every time – it is the initial inject that is passing the file name.  The two (orange functions are the same other than the actual object name (thermostat and office_thermostat).

Here’s the content of the function at the heart of Read File Sync

var fs=global.get("fs");
try {
msg.payload = fs.readFileSync(msg.payload).toString();
}
catch (ex)
{
msg.payload=undefined;    
}
return msg;

And here is my code in the orange top function – the brown item on the right is just a file write node. Clearly this is for my particular application but you should get the general idea..

if (msg.payload===undefined) // if the file was not valid - create the object, predefine, store
    {
       var stat = {
        max_humidity : 100,
        display_temperature : 22,
        desired : 22,
        display_humidity : 45,
        heating : 0,
        display_external_temperature : 14,
        display_external_humidity : 54,
        display_status : "Normal",
        display_enabled : 0,
        display_last_page : "home",
        display_last_colour : 0,
        pass : "",
        display_dim : 60,
        
        frost : 14,
        hold : 0,
    
        p1 : 180,
        p2 : 480,
        p3 : 600,
        p4 : 1080,
        p5 : 1380,

        pw1 : 180,
        pw2 : 480,
        pw3 : 600,
        pw4 : 1080,
        pw5 : 1380,
        
        t1 : 16,
        t2 : 21,
        t3 : 19,
        t4 : 21,
        t5 : 18,
    
        tw1 : 16,
        tw2 : 21,
        tw3 : 19,
        tw4 : 21,
        tw5 : 18, 
    
        temperature_offset : 0,
        last_state : -1,
        current_state : 0,
        update : 0,
        daynow: 1
    };
    global.set("thermostat",stat);
    msg.payload=JSON.stringify(stat);
    return msg; 
    }
else if (msg.payload=="check") // this is the every second check - assuming file exists
    {
    if (global.get("thermostat.update")!==0)
        {
        global.set("thermostat.update",global.get("thermostat.update")-1);
        if (global.get("thermostat.update")===0)
            {
            msg.payload=JSON.stringify(global.get("thermostat"));
            return msg;
            }
        }
    }
else // read in the file if it was valid - don't output anything
    {
    global.set("thermostat",JSON.parse(msg.payload));
    global.set("thermostat.laststate",-1); // detecting change of timezone
    global.set("thermostat.temperature_offset",0);
    }


Not sure how useful you feel this is for your projects compared to a database – but it is simple and fast. The point of the global object, of course, is that you can use it on any page in your node-red setup. Certainly, by careful use of the timer when updating, you can vastly decrease the number of WRITES compared to a normal database setup. Handy for those of us on a Raspberry Pi with limited SD writes.

There are those who put their hands up in horror at the idea of globals - I do not. I like the idea of having the ability to view or update things from any tab (flow) with ease.  Real world apps? I've been using this (bearing in mind the initial issue with start up values just resolved) in a holiday rental since April - with people in all the time - running the heating and some lighting - not one issue.

Facebooktwittergoogle_pluspinterestlinkedin

17 thoughts on “Node-Red Objects to File and Back

  1. For what I'm doing, it doesn't even need to be as complicated as that. To keep track of the status of 4 lamps and report those via LED controls on the Blynk dashboard, it looks like all I need to do is use the File node on its own to write either 0 or 1 to a file called lamp1status (or lamp2status, lamp3status, etc).
    I can then have a button on the Blynk dashboard called "Get Status" which tells Node Red to read each of the 4 files and inject the 0 or 1 into the Blynk node that controls each LED on the dashboard.

  2. Hi Peter,
    I hope you can help me. I'm a novice user of Node-Red (migrating from EventGhost) and am struggling to figure out how I can persist the states of my various lights and sockets into a JSON formatted string in a file. I've been reading your various blog articles for hours, and searched documentation on global.set and global.get functions, plus associative arrays in js and I'm getting knowhere fast. What I'm aiming to do is have the msg.topic as the key and the msg.payload as the value for each entry in the array, then JSON.stringify() it and put it in a file that AutoTools on my Android phone can subsequently read. I'd be very grateful of any help and advice you can offer.

    1. Seems to me you are already there - so for example in my heating controls I stringify the object and simply save it to a file using the file node. On power up using an inject node set to inject only once on power up - I read back and json.parse the string back into an object. As for getting that onto your phone - my scenario keeps node-red up to date - not the phone.

  3. Thank you for your post, this is very helpful.
    I am also new to node-red. Current I am working on a network of temperature sensors and looking for some help.
    Firstly, I want the user to be able to make changes to the system parameters without giving them access to editor (I password protect editor and only give access to dashboard). How would I let the user change things like High alarm and low alarm values and title for the sensor. Can they fill in these details and use the File node to save them?
    Also, what about the situation they want to add another sensor to their system? Is there any way the system can create a new flow from an action in the dashboard or maybe have flows ready to go but not "enabled" or made visible when they are added?

    1. I have one password on the editor another on the UI. I also use HTTPS for external access. If you look at my recent thermostat project, setting of high and low values for the entire week along with temperatures is done via the UI on a phone. For adding new stuff, just go into Node-Red and add nodes etc, really easy. My projects are oriented to people who do their own stuff and maybe want the wife/husband/boyfriend/girlfriend to control stuff like lights and heating on a nice interface. I don't concentrate on making drop-in products for sale as that's not that interesting. Of course - once you get used to Node-Red and spend the time to get skilled in it, the sky is the limit - expecially thanks to the UI which allows for templates which can contain whatever you want within reason.

  4. Thanks for your prompt response Pete.
    You make a good point about project motivation. Most are probably for their own home projects and they can easily make changes themselves. If I set up some way to access the client's system externally, I could log in and make changes for them (I don't want them fiddling with the editor as they are farmers and blue-collar workers typically). The system will have a 3G dongle, I suppose if I could set up a static IP on the cellular then I could access the system to make changes?

    1. 1) having a constant access to some one other system could be very "unhealthy" for yourself... what about if they accuse you of something?
      2) most 3G networks are NATted, so you don't have a public ip, or, best, you have one shared with many others, and worst of all, it changes... imagine a large private network like an office, or your home, that's what you get, your 3G ip is as your router ip, but you can't do port forward...
      3) possible solution, if you want to go this way: https://www.dataplicity.com/

  5. By the way, do you have some links that are helpful for learning node-red. The official site is great as a reference but not as useful to me for learning. Be great if someone had a video course (for a cost or for free would be ok)

    1. video, don't know... node-red itself is quite easy, once you learn how to install and connect nodes, what is really needed is programming in javascript, the way NODEJS is used to be programmed... the more in depth you go, the more you need programming... so, to start, free:
      http://noderedguide.com/
      https://leanpub.com/nodecraftsman (it's a long free following to the commercial "The Node Beginner Book")
      http://www.hivemq.com/mqtt-essentials/ (best MQTT intro)
      javascript, well, just use google, it's FULL of resources...

      then pay:
      the book by Rui Santos: https://randomnerdtutorials.com/getting-started-with-node-red-on-raspberry-pi/ (link in article)
      https://www.amazon.com/Building-Web-Things-examples-Raspberry/dp/1617292680/
      https://www.amazon.com/Wiring-IoT-Connecting-Hardware-Raspberry/dp/1491953330/ (still to be released)
      https://www.amazon.com/s/field-keywords=a+smarter+way+to+learn (javascript books by Mark Myers)

      and, youtube...

      1. I'd also recommend Javascript Principles https://www.amazon.co.uk/gp/product/B00JFN7MD6/ref=oh_aui_d_detailpage_o03_?ie=UTF8&psc=1
        The English can be a little clumsy and there are a few annoying typos but it takes Javascript from the aspect of Jquery and it explains what is going on under the hood - which makes it easier to grasp context and the structure of the environment. Also it is only 99p on the KIndle so no great loss if its not right for you.

  6. I'm currently reading "The Node Craftsman book" - very interesting seeing the differing ways you can handle objects in Javascript, though right now having cracked simple objects, my big bottleneck is image and font loading. I have both a custom font and images in my current gauge and ensuring they are both fully loaded is proving somewhat challenging. This didn't show up at home, using the locally but as I'm now sitting in (sometimes) sunny Spain with the attendant delays in accessing files from my home server, the initial load-up issue is becoming apparent. I've tried some obvious techniques and they've not done me any good up to now.

  7. >> Now - you need to do this in the LEFT-MOST tab in Node-Red as this is executed first - before any other tabs - so you get to define the object BEFORE it gets used.

    Quoted part of post together with using "readFileSync()" helped me lot. I would like to share my findings for case that the same left-most tab contains multiple "inject" nodes:
    - the "inject" node that appears first in flow definition, that is within flows file - not topologically, will be executed first.

Leave a Reply

Your email address will not be published. Required fields are marked *