Garden Data Collection

MiFloraSome time ago a pal of mine and I did some swaps of spare kit and I ended up with a Mi Flora sensor.  A rather pleasing looking device,  the MI Flora has a 3v lithium button cell and can transmit by Bluetooth LE, the light level, temperature, battery level and moisture.

Battery is supposed to last for months. I’ve provided an AliExpress link above which puts the units at under £8 Inc. postage. No doubt that price will vary dramatically depending on where you chose to spend your money. Amazon charge more like £13.

So having pulled off the protective tab which meant the battery would work, I put the unit to one side as I had no idea how to read it.

A week or so later in Spain, with poor weather my pal Jay came on to ask me if I’d used it. I had not – and so we decided to give it a go. As it turned out I had some cheap Bluetooth 4 dongles I’d bought on a whim last year. I plugged one into my Raspberry Pi 2. The program “hcitool” lets you scan for Bluetooth devices and as it happens it was already installed on the Pi. From the command prompt:

sudo hcitool lescan

Instant success as the Mi Flora device with it’s MAC address was revealed. I made a note of the MAC address.

I would need Python 3 to experiment – but this was already installed on the Pi and some code to get the information from the unit. This code turned out to be ok with a little modification. I ended up creating a folder called /home/pi/myflora and dumping the lot in there.

Jay had modified the program to send MQTT. That of course needed the PAHO client. In order to do that I needed something called PIP.

sudo apt-get install python3-pip

sudo pip3 install paho-mqtt

Note NOT pip (which works for Python 2.7) but pip3.

I then added to the MI flora directory a test program which Jay sent me called bathroom.py – presumably because he’d put the sensor in his bathroom – I adjusted the output slightly as I was putting my unit in the garden

import sys
import paho.mqtt.client as mqtt

from miflora.miflora_poller import MiFloraPoller, \
MI_CONDUCTIVITY, MI_MOISTURE, MI_LIGHT, MI_TEMPERATURE, MI_BATTERY

poller = MiFloraPoller("C4:7C:8D:32:08:63")
print("Getting data from Mi Flora")
print("FW: {}".format(poller.firmware_version()))
print("Name: {}".format(poller.name()))
print("Temperature: {}".format(poller.parameter_value("temperature")))
print("Moisture: {}".format(poller.parameter_value(MI_MOISTURE)))
print("Light: {}".format(poller.parameter_value(MI_LIGHT)))
print("Conductivity: {}".format(poller.parameter_value(MI_CONDUCTIVITY)))
print("Battery: {}".format(poller.parameter_value(MI_BATTERY)))

# Publishing the results to MQTT
mqttc = mqtt.Client("miflorabathroom")
mqttc.username_pw_set("admin","xxxx")
mqttc.connect("192.168.1.19", 1883)
mqttc.publish("miflora/garden", "{ \"battery\" : " + str(poller.parameter_value(MI_BATTERY)) + ", \"version\" : \"" + str(poller.firmware_version()) + "\", \"sunlight\" : " + str(poller.parameter_value(MI_LIGHT)) + ", \"temperature\" : " + str(poller.parameter_value("temperature")) + ", \"moisture\" : " + str(poller.parameter_value(MI_MOISTURE)) + ", \"fertility\" : " + str(poller.parameter_value(MI_CONDUCTIVITY)) + " }")
mqttc.loop(2)
# End of MQTT section

So the idea was that running “python3 garden.py” would send some MQTT to my Mosquitto MQTT broker – and indeed it did (change MAC address for the MI Flora and MQTT details in the above).  Subscribing to “miflora/garden” did the trick – I could fire off the program and with in seconds I’d see the resulting JSON package coming out into my MQTT-Spy client.

Next job was to get Node-Red talking to the MI Flora – I used an INJECT node, set to retrigger every 15 minutes – firing off to an EXEC node which contained nothing more than “python3 /home/pi/miflora/bathroom.py”

That worked – so now I had a regular supply of data in the form of an easy to use JSON package.

I split the package up – and fed it into a Node-Red Dashboard graph – and that was my first disappointment. If you fire more than 4 items into those graphs – the popup fails to show all of them – and the graphs are not very good anyway. I think a lot more work needs to go into that.

I do use GROVESTREAMS on-line (which is REALLY flexible) to store data remotely but there’s a limit as to how far back you can read for free – that was not the case when I first started using them and wrote the Node-Red-Contrib-Grove node to make it all easy.

So now I was on a search for a decent graphics package that would run on the Raspberry Pi (or similar). NOT as easy as it sound,  unless you’re very easily pleased when it comes to graphics.

After a lot of experimenting I decided to give Grafana a shot. I think if I started writing this blog again, I would would have Grafana in the title because this really, as it turns out, is the star of the show.

Having seen a few screenshots, I had visions of populating  a SQLITE 3 database and firing that data into Grafana.

Erm, no. Despite (wait for it) using SQLITE internally to store users and dashboards, the package does not actually support SQLITE for data !!!  I've explained that now, to save you some confusion later. When I did the install, it kept referring to SQLITE and could I HELL figure out how to set up a database for my data – that’s why.

As a Pi user (that means SD use so I’m not keen on MySQL as SD has limited WRITE capability and I worry about databases shortening the life of the SD) the options for data sources did not look appealing to me at all, the least offensive being INFLUX and that is the one I settled on – partly as there is a node for it available for Node-Red.  Despite the pretty website this is still very preliminary at version 1.02. Influx doesn’t use terms like tables and fields which kept me going for a couple of hours – but it is good at storing time-stamped data – indeed you don’t even do the time-stamping. You create an empty database and start firing data at the DB in pairs – stream name – and data value – it really is very simple once you get started. I guess you could think of it as a single table with pairs of data-name and data-value.  I went down a few dead ends and discovered if you got your stream names wrong – good luck renaming them as INFLUX does not support renaming!

Anyway, it ‘s all very easy once you get started. I did things backwards and installed Grafana first. Here’s what I did – no guarantee it will work for you.

You might want to consider reading this before using InfluxDB on the Pi for long-term use - there are very few build options for small SBCs and this article gives the impression it likes more RAM than we would normally have on the likes of the Pi. I do have to say though that it is working just fine on mine after several days of operation. Comments welcome.

I grabbed the Grafana file for my Pi from here. https://github.com/fg2it/grafana-on-raspberry

sudo dpkg -i grafana_4.1.2-1487023783_armhf.deb
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server

That was Grafana up and running on port 3000 – user name admin, password admin to actually DO anything – and quickly change that password!!! I did that, went into the ini file and disabled user registration as the admin can do that and opened up port 3000 redirect in my router so I could access this from the outside world. That all worked well.

wget http://ftp.us.debian.org/debian/pool/main/i/influxdb/influxdb_1.0.2+dfsg1-1_armhf.deb

sudo dpkg -i influxdb_1.0.2+dfsg1-1_armhf.deb

That set up influx and that ran without further ado at port 8083. I did NOT open that up to the outside world. In there I created a new empty database called “logger”

logger

In Grafana – which supports influx without any changes, I followed Engineer John’s instructions to set up the data-source and a new Dashboard.

So the next thing – would be to get some data.  I played with this for some time – should I keep incoming data source names simple – as you can’t rename them? Then Jay pointed out that in Grafana you can ALIAS names. That did it. So in influxDB node for Node-Red, you fire in a payload – and depending where you want to put your data – you use msg.measurement.. Yes, exactly, what’s wrong with msg.topic you might ask. So here is my little MI Flora MQTT node firing values into InfluxDB

MiFlora[6]

And that yellow function block…

node.status({fill:"blue",shape:"dot",text: msg.payload});

tmp=JSON.parse(msg.payload);

msg.measurement="battery"; msg.payload=tmp.battery; node.send(msg);
msg.measurement="sunlight"; msg.payload=tmp.sunlight; node.send(msg);
msg.measurement="temperature"; msg.payload=tmp.temperature; node.send(msg);
msg.measurement="moisture"; msg.payload=tmp.moisture; node.send(msg);
msg.measurement="fertility"; msg.payload=tmp.fertility; node.send(msg);

Yes, I know – far better ways to do this – but it was my first stab at it and it worked. For general use however I added an incoming function that merely copies msg.topic into msg.measurement and that means any single data-source coming in via MQTT can be blasted straight into InFluxDB.

Yes, that’s write – you don’t create tables or fields – you just send in field names and data and that’s it!

I also fed in some data from my general sensors – with horrible topics like pergola/lighting but of course thanks to the Grafana alias ability – names like that would not show up in Dashboards.

Having stored up some data it was time to get it into graphs – I took the VERY simplest approach of showing lines as I have at this point no idea what some of the Grafana functions do – I DO know that smoothing the data might be fun as Grafana relies on the data-source to do that and Influx doesn’t do that.  What IS nice is that you don’t have to worry about things like “I want the last 5 minute’s worth” or “show me the last month’s worth” as Grafana does all of that for you – from here  it gets easy.

New dashboard and under metrics ADD ROW – takes an hour or so to get to grips with this..

Grafana setup

I’ve expanded the top one here – I have 3 lines in that one graph – see the blank ALIAS BY  - if you want to change the source name (in this case battery) put something in there.

And that  is how I got to this point…

Graphing

So top left you see battery, moisture and temperature from the MI Flora unit. I chose to separate off light level as the range of values is WAY higher… and on the right I’ve started with another unrelated sensor incoming from MQTT and as you see using the ALIAS I’ve renamed the stream (topic) to “Kitchen OS Temperature” – yes, it really IS that cold inland Spain right now.

But this is merely the beginning – now I will put all my other data from the UK and Spain into here – lots of it and some day I might even manage to extract individual graphs back into Node-Red Dashboard, but right now, I’m happy with these pages. Looking good.

There’s a lot to take in – and I am as you can see NOT an expert at this yet so before firing questions back – you might want to spend some time looking at the links on this blog entry and using Google. If you think another database is better by all means write in to say why – and if you think there’s a better easy-to-fit-in-Pi graphing system, preferable one that works with SQLITE – again do let us know. I started all of this with zero information late afternoon yesterday and had it all working by the end of last night – so it’s not THAT big a deal even if my clumsy writing makes it look that way.

Not entirely sure what’s going on the this battery level on my first day – but it did occur to me that the unit is intended for putting into plant-pots – not sitting outside in the sopping rain so I’ve covered it in cling-film!! As we’re off back to the UK for a few weeks,  I don’t want to come back to a rusted heap.  Note the Mi Flora battery at the top – seems to be self-recovering!!

I have to say, having taken the plunge I DO like Grafana. It isn’t perfect… I tried the GAUGES plug-in which was great using one gauge but as soon as I tried more than one, all SORTS of things went wrong – eventually I stopped using it.. but the graphs are fine. I’m already seeing benefits of monitoring – the top right graph shows the effect of watering a blocked-in area of the patio in Spain – the watering has been good for nearly 3 days up to now and by the look of it will be good for a week. As we pay per litre, such info is useful. Of course, things will be different in the summer but the automatic system currently offline (ice) waters twice a day – this may WELL prove to be totally over the top.

Update: the little Bluetooth sensor has now been running for several weeks and the battery remains at 80%

Grafana output


If you like this post – please share a link to it by social media, by email with friends or on your website.
More readers means more feedback means more answers for all of us. Thank you!

Facebooktwittergoogle_pluspinterestlinkedin

27 thoughts on “Garden Data Collection

  1. Grafana is awesome Pete, though I went a different route and used Graphite for the backend. It has it's own lightweight database called whisper for holding the time-series data (you don't need any other type of database like SQLITE when looking at metrics over a time period like temperature/humidity/etc).

    https://graphiteapp.org/

    It's a bit more involved to install but is definitely worth it as you get a load of math functions which can be applied to the graphs.
    http://graphite.readthedocs.io/en/latest/functions.html

    Also there are countless frontends like Grafana which can be used with Graphite, look at all the different integrations here http://graphite.readthedocs.io/en/latest/tools.html

    Regards

    Adam

    1. graphite looks great - a bunch of guys that I used to work with were fans of graphite for their application that they developed on AWS. It reminds me of ELK stack which is also fabulous for free! https://www.elastic.co/
      kibana does really nice graphs and charts which update in real-time bound to the backend elastic DB

    2. Hi Adam - yet I agree Grafana is awesome - especially after this morning where I just sat down with coffee and easily filtered out that bad value by simply selecting sensible signals in Grafana itself... it's just so annoying that it is not supported by a nice easy to use database like SQLITE - wherein I could have deleted that bad value in seconds. I find it even more amazing because they use SQLITE under the hood.

      It also comes up well on mobile phones which is a plus. I'm busy monitoring watering in the garden in Spain (remotely as I'm in the UK). My neighbour watered the plants 3 days ago and there is still far more moisture in the ground than I would have expected...

      1. Here's another shameless plug for elasticsearch. No I don't work for them. Yes you can delete documents from the elasticsearch db. Yes there is node-red support. Yes nice real-time graphs/charts in kibana. Also available as a complete stack from a number of sources..

          1. I've never installed this on a pi so I can't offer a "defacto fabulous package". Besides I guess what you're going to need will depend on what distro you're running.

            I suggest you start at the official source here:

            https://www.elastic.co/downloads

            There are pre-made package manager packages available for a number of distros there too. See https://www.elastic.co/start

            You'll want both elasticsearch (the document persistence store) and kibana (the UI stack).

            Logstash might be useful too but is not mandatory.. think of logstash as linux pipes on steroids for manipulating/transforming event data from any form -> a stored elasticsearch document

            Note this stuff can be installed theoretically anywhere on the LAN, it does not necessarily need to be installed on the pi itself and the node-red -> elasticsearch stuff should work fine as long as there's connectivity between the systems.

              1. Elasticsearch does require Java, it's a Java built platform.

                As I mentioned above I've never installed it on a Pi so I can't offer any personal 1st hand experience advice on good/bad there. However a google search tells me there are number of people who have done this so it's certainly doable and some even report it's surprisingly good despite the constrained resources.

                At the risk of stating the obvious.. how well it's going to actually work in reality is going to depend on a number of factors: what's it sat alongside? what other processes are running? how much load is ES under etc?

                I'd like to try it myself but I don't have a Pi so I can't so I'm going to have to say: Elasticsearch running on a Pi - YMMV

  2. An interesting device.

    You might want to consider using the python json module:
    import json
    and use the json.dumps function combined with a dictionary as the payload, for example:

    json.dumps(dict(item1=1, item2=2, item3=str(3)))

    results in the following mqtt payload:

    {"item1": 1, "item2": 2, "item3": "3"}

    If I've understood your code correctly that means your publish statement would end up being:

    mqttc.publish("miflora/garden", json.dumps(dict(battery=str(poller.parameter_value(MI_BATTERY)), version=str(poller.firmware_version()), sunlight=str(poller.parameter_value(MI_LIGHT)), temperature=str(poller.parameter_value("temperature")), moisture=str(poller.parameter_value(MI_MOISTURE)), fertility=str(poller.parameter_value(MI_CONDUCTIVITY)))))

    Saves a bit of escaping ...

    Great blog - keep finding things of interest to read and try

    David

    1. I could have sworn I replied in here....

      OK - RANGE - 9ft of air then a breezeblock wall then 6ft of air - it retries once or twice but always gets there in the end. I'm leaving this now, we're back off to the UK and I'll be able to monitor it from there. I'm ordering another Mi Flora so I'll have one in each country. Good toy. How long it lasts and how long the battery lasts - something for another blog.

      Graphite - I'll take a second look but at first glance it did NOT look like anyone had gotten around to making a friendly installation for the Pi. I was also thinking - the lack of smoothing in INFLUX can presumably be sorted by using the SMOOTH function in Node-Red.

  3. To add another recommendation of graphite - I've been using this for storage for some time now (since 2008!). It has automatic rollup of data to downsample as it gets older, which ensures your database doesn't grow out of control in size.

    Influxdb it seems possible to setup downsampling, but a lot trickier. Worth thinking about in terms of having the data useful a few years down the line.

  4. Pete,
    Tried to follow your example but no go, did you download the armhf file from somewhere first?

    I looked on the Grafana site but can't find it, the download they show is for an AMD64.

    1. Hi there Phil....

      That's because, in the instructions, I missed off the all important link to the Grafana build for Raspberry Pi - note that I have updated the blog - make sure you get the right one for your Pi.

      I'm loving it - back in the UK now and I can access all my readings over in Spain.

      1. I hope you're feeling better, I got it all installed I'm now struggling to get data from an existing MQTT stream into InfluxDB.

        I guess you formatted your MQTT stream as JSON ? mine is just a topic "Temperatures/Exterior/Ambient" with a data value, I've tried to format it correctly but I can't seem to get the hang of it, I assume I've got to convert it to a name and a value formatted as a JSON string.

        I've got this so far but I don't think it looks right
        msg.measurement="Outside temperature"+":";msg.payload=":"+"'"+tmp+"'"; node.send(msg);

        can you help with the right format?

        1. Hi Phil - I'm off to bed - but - I didn't format anything - the example PYTHON code did it. Then in the orange FUNCTION I simply used JSON.parse to turn that into an object I could use.

          In the general case - given names fred and brian, with values 50 and 60, a suitable JSON string would be...

          {"fred":50,"brian":60}

          If the answers were strings rather than numbers - they would be in quotes too.

  5. I have to say that I am ABSOLUTELY having second thoughts about INFLUX - I think it has been designed for masochists for masochists.

    I have a duff value - temperature sensor reading around reboot of a unit produced a value of -4000 - well you can imagine how well that scales on the graph. In any sensible database it would be something like...

    delete from logger where uk_temp<40

    But no - you can't DO that apparently. I cannot figure out how to simply delete a point.

  6. I've often wondered about editing data in the time based databases but that's where the maths functions of Graphite would come into their own, you wouldn't be able to remove the data point from the database but I'm sure you could remove it from the graph see http://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.removeAboveValue

    I know you said before about not finding any good guides for installing Graphite, these are the 2 I referred to for installing on my BBB.

    https://www.digitalocean.com/community/tutorials/how-to-install-and-use-graphite-on-an-ubuntu-14-04-server

    https://www.linode.com/docs/uptime/monitoring/deploy-graphite-with-grafana-on-ubuntu-14-04

    Also you could probably just add some code into a function to do a sanity check on the incoming data and throw it out if it's not between a min and max threshold before sending it to influx.

    Adam

  7. I've been using InfluxDB and Graphana for ages now. I replaced MongoDB with InfluxDB as MongoDB is horrid on 32bit machines.

    InfluxDB is great as long as you don't push the number of data points too far. Use the auto-trim and summarise features. I trim my minute data to a week but create 1hr average data for long-term charts.

  8. Apologies if I have got the wrong end of the stick but shouldn't it be

    sudo apt-get install python3-pip (not python-pip) for the correct version of pip.

    Also shouldn't it be

    sudo pip3 install paho-mqtt (not paho-client) - couldn't get paho-client to wok at all.

    1. Hmm, I'll have to pass this one to my friend Jay, Steve as I didn't originate that bit. Jay, you in here?

      However - that did work for me - weeks later that sensor is sitting happily collecting garden data.

  9. Pete - just a heads up for anyone following your examples here using a RPi and the stock Raspbian version...

    I copied your Node-RED Flow using MQTT input -> Function -> InfluxDB output (with minor tweaks to the function). I'm testing a BME280/ESP-12F setup and everything was working fine but no data was being written to InfluxDB. I noticed an error indicating "Object() has no method 'assign'". A quick Google and I found the pre-installed Node.js is an old one and needed to be updated.

Leave a Reply

Your email address will not be published. Required fields are marked *