This entry is all about InfluxDB, Grafana and a cute little Bluetooth garden sensor!
Some time ago a pal of mine and I did some swaps of spare kit and I ended up with a Mi Flora sensor. A rather pleasing looking device, the MI Flora has a 3v lithium button cell and can transmit by Bluetooth LE, the light level, temperature, battery level and moisture.
Battery is supposed to last for months. I’ve provided an AliExpress link above which puts the units at under £8 Inc. postage. No doubt that price will vary dramatically depending on where you chose to spend your money. Amazon charge more like £13.
So having pulled off the protective tab which meant the battery would work, I put the unit to one side as I had no idea how to read it.
A week or so later in Spain, with poor weather my pal Jay came on to ask me if I’d used it. I had not – and so we decided to give it a go. As it turned out I had some cheap Bluetooth 4 dongles I’d bought on a whim last year. I plugged one into my Raspberry Pi 2. The program “hcitool” lets you scan for Bluetooth devices and as it happens it was already installed on the Pi. From the command prompt:
sudo hcitool lescan
Instant success as the Mi Flora device with it’s MAC address was revealed. I made a note of the MAC address.
I would need Python 3 to experiment – but this was already installed on the Pi and some code to get the information from the unit. This code turned out to be ok with a little modification. I ended up creating a folder called /home/pi/myflora and dumping the lot in there.
Jay had modified the program to send MQTT. That of course needed the PAHO client. In order to do that I needed something called PIP.
sudo apt-get install python3-pip
sudo pip3 install paho-mqtt
Note NOT pip (which works for Python 2.7) but pip3.
I then added to the MI flora directory a test program which Jay sent me called bathroom.py – presumably because he’d put the sensor in his bathroom – I adjusted the output slightly as I was putting my unit in the garden
import sys
import paho.mqtt.client as mqttfrom miflora.miflora_poller import MiFloraPoller, \
MI_CONDUCTIVITY, MI_MOISTURE, MI_LIGHT, MI_TEMPERATURE, MI_BATTERYpoller = MiFloraPoller(“C4:7C:8D:32:08:63”)
print(“Getting data from Mi Flora”)
print(“FW: {}”.format(poller.firmware_version()))
print(“Name: {}”.format(poller.name()))
print(“Temperature: {}”.format(poller.parameter_value(MI_TEMPERATURE)))
print(“Moisture: {}”.format(poller.parameter_value(MI_MOISTURE)))
print(“Light: {}”.format(poller.parameter_value(MI_LIGHT)))
print(“Conductivity: {}”.format(poller.parameter_value(MI_CONDUCTIVITY)))
print(“Battery: {}”.format(poller.parameter_value(MI_BATTERY)))# Publishing the results to MQTT
mqttc = mqtt.Client(“miflorabathroom”)
mqttc.username_pw_set(“admin”,”xxxx”)
mqttc.connect(“192.168.1.19”, 1883)
mqttc.publish(“miflora/garden”, “{ \”battery\” : ” + str(poller.parameter_value(MI_BATTERY)) + “, \”version\” : \”” + str(poller.firmware_version()) + “\”, \”sunlight\” : ” + str(poller.parameter_value(MI_LIGHT)) + “, \”temperature\” : ” + str(poller.parameter_value(MI_TEMPERATURE)) + “, \”moisture\” : ” + str(poller.parameter_value(MI_MOISTURE)) + “, \”fertility\” : ” + str(poller.parameter_value(MI_CONDUCTIVITY)) + ” }”)
mqttc.loop(2)
# End of MQTT section
So the idea was that running “python3 garden.py” would send some MQTT to my Mosquitto MQTT broker – and indeed it did (change MAC address for the MI Flora and MQTT details in the above). Subscribing to “miflora/garden” did the trick – I could fire off the program and with in seconds I’d see the resulting JSON package coming out into my MQTT-Spy client.
Next job was to get Node-Red talking to the MI Flora – I used an INJECT node, set to retrigger every 15 minutes – firing off to an EXEC node which contained nothing more than “python3 /home/pi/miflora/bathroom.py”
That worked – so now I had a regular supply of data in the form of an easy to use JSON package.
I split the package up – and fed it into a Node-Red Dashboard graph – and that was my first disappointment. If you fire more than 4 items into those graphs – the popup fails to show all of them – and the graphs are not very good anyway. I think a lot more work needs to go into that.
I do use GROVESTREAMS on-line (which is REALLY flexible) to store data remotely but there’s a limit as to how far back you can read for free – that was not the case when I first started using them and wrote the Node-Red-Contrib-Grove node to make it all easy.
So now I was on a search for a decent graphics package that would run on the Raspberry Pi (or similar). NOT as easy as it sound, unless you’re very easily pleased when it comes to graphics.
After a lot of experimenting I decided to give Grafana a shot. I think if I started writing this blog again, I would would have Grafana in the title because this really, as it turns out, is the star of the show.
Having seen a few screenshots, I had visions of populating a SQLITE 3 database and firing that data into Grafana.
Erm, no. Despite (wait for it) using SQLITE internally to store users and dashboards, the package does not actually support SQLITE for data !!! I’ve explained that now, to save you some confusion later. When I did the install, it kept referring to SQLITE and could I HELL figure out how to set up a database for my data – that’s why.
As a Pi user (that means SD use so I’m not keen on MySQL as SD has limited WRITE capability and I worry about databases shortening the life of the SD – see the later article on SD life – very informative) the options for data sources did not look appealing to me at all, the least offensive being INFLUX and that is the one I settled on – partly as there is a node for it available for Node-Red. Despite the pretty website this is still very preliminary at version 1.02. Influx doesn’t use terms like tables and fields which kept me going for a couple of hours – but it is good at storing time-stamped data – indeed you don’t even do the time-stamping. You create an empty database and start firing data at the DB in pairs – stream name – and data value – it really is very simple once you get started. I guess you could think of it as a single table with pairs of data-name and data-value. I went down a few dead ends and discovered if you got your stream names wrong – good luck renaming them as INFLUX does not support renaming!
Anyway, it ‘s all very easy once you get started. I did things backwards and installed Grafana first. Here’s what I did – no guarantee it will work for you.
You might want to consider reading this before using InfluxDB on the Pi for long-term use – there are very few build options for small SBCs and this article gives the impression it likes more RAM than we would normally have on the likes of the Pi. I do have to say though that it is working just fine on mine after several days of operation. Comments welcome.
I grabbed the Grafana file for my Pi from here. https://github.com/fg2it/grafana-on-raspberry
sudo dpkg -i grafana_4.1.2-1487023783_armhf.deb
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server
That was Grafana up and running on port 3000 – user name admin, password admin to actually DO anything – and quickly change that password!!! I did that, went into the ini file and disabled user registration as the admin can do that and opened up port 3000 redirect in my router so I could access this from the outside world. That all worked well.
wget http://ftp.us.debian.org/debian/pool/main/i/influxdb/influxdb_1.0.2+dfsg1-1_armhf.deb
sudo dpkg -i influxdb_1.0.2+dfsg1-1_armhf.deb
That set up influx and that ran without further ado at port 8083. I did NOT open that up to the outside world. In there I created a new empty database called “logger”
In Grafana – which supports influx without any changes, I followed Engineer John’s instructions to set up the data-source and a new Dashboard.
So the next thing – would be to get some data. I played with this for some time – should I keep incoming data source names simple – as you can’t rename them? Then Jay pointed out that in Grafana you can ALIAS names. That did it. So in influxDB node for Node-Red, you fire in a payload – and depending where you want to put your data – you use msg.measurement.. Yes, exactly, what’s wrong with msg.topic you might ask. So here is my little MI Flora MQTT node firing values into InfluxDB
![MiFlora[6]](https://tech.scargill.net/wp-content/uploads/2017/02/MiFlora6.jpg "MiFlora[6]")
And that yellow function block…
node.status({fill:”blue”,shape:”dot”,text: msg.payload});
tmp=JSON.parse(msg.payload);
msg.measurement=”battery”; msg.payload=tmp.battery; node.send(msg);
msg.measurement=”sunlight”; msg.payload=tmp.sunlight; node.send(msg);
msg.measurement=”temperature”; msg.payload=tmp.temperature; node.send(msg);
msg.measurement=”moisture”; msg.payload=tmp.moisture; node.send(msg);
msg.measurement=”fertility”; msg.payload=tmp.fertility; node.send(msg);
Yes, I know – far better ways to do this – but it was my first stab at it and it worked. For general use however I added an incoming function that merely copies msg.topic into msg.measurement and that means any single data-source coming in via MQTT can be blasted straight into InFluxDB.
Yes, that’s write – you don’t create tables or fields – you just send in field names and data and that’s it!
I also fed in some data from my general sensors – with horrible topics like pergola/lighting but of course thanks to the Grafana alias ability – names like that would not show up in Dashboards.
Having stored up some data it was time to get it into graphs – I took the VERY simplest approach of showing lines as I have at this point no idea what some of the Grafana functions do – I DO know that smoothing the data might be fun as Grafana relies on the data-source to do that and Influx doesn’t do that. What IS nice is that you don’t have to worry about things like “I want the last 5 minute’s worth” or “show me the last month’s worth” as Grafana does all of that for you – from here it gets easy.
New dashboard and under metrics ADD ROW – takes an hour or so to get to grips with this..
I’ve expanded the top one here – I have 3 lines in that one graph – see the blank ALIAS BY – if you want to change the source name (in this case battery) put something in there.
And that is how I got to this point…
So top left you see battery status from my deep discharge battery and the two MI Flora units, then moisture levels from the two units – light levels – and on the right centre – a whole host of temperature readings, overall humidity and finally just to play with – the UK temperature split off and displayed differently.
There’s a lot to take in – and I am as you can see NOT an expert at this yet so before firing questions back – you might want to spend some time looking at the links on this blog entry and using Google. If you think another database is better by all means write in to say why – and if you think there’s a better easy-to-fit-in-Pi graphing system, preferable one that works with SQLITE – again do let us know. I started all of this with zero information late afternoon yesterday and had it all working by the end of last night – so it’s not THAT big a deal even if my clumsy writing makes it look that way.
Not entirely sure what’s going on the this battery level on my first day – but it did occur to me that the unit is intended for putting into plant-pots – not sitting outside in the sopping rain so I’ve covered it in cling-film!! As we’re off back to the UK for a few weeks, I don’t want to come back to a rusted heap. Note the Mi Flora battery at the top – seems to be self-recovering!!
Update: the little Bluetooth sensors have now been running for months and the battery levels remain fine – the Bluetooth range however is crap – best to have a Bluetooth dongle nearby to talk to them – don’t even think about two thicknesses of wall!
Update 27/03/2017: I’ve been hitting issues with out of bounds temperature data – i.e. a couple of duff readings crept in – with values like 2000 and -64 which of course flattened the data completely. I realised quickly that InfluxDB has no DELETE function and as far as I could tell – things like comparing data with a value made the graph line disappear.
I put something up on the web and someone came back and asked if I was using InfluxDB 1.2 which apparently has a DELETE function. Well, I wasn’t – I was way back at 1.01 – worse – the same folder where that came from only went up to 1.1
However, thanks to THIS helpful chap – I easily updated to Influx 1.2 without the issue he described. I STILL can’t add something like “where value < 100” but maybe I can delete ??? Anyone with more info on this please by all means chip in…
Update 18/04/2017: I’ve just fitted the second of these units and also fixed a minor error in the code above. All that is needed for a second unit is a copy of the program with different MAC address and different MQTT publish address. Sure you could use the same one twice with command line parameters but I took the easy option.
Bluetooth RANGE is becoming an issue. The Pi is in my office – breezeblock wall then the garden – the second unit I tried maybe 15ft or so from the office wall – not a chance. Clearly BT4 is good for power, not so good for range.
So – there’s a nice job for an ESP32… BT in – WIFI out. A relay!
Update 30/04/2017: At the time of writing, InfluxDB is up to version 1.2.2 and if you follow the link I gave above but instead of 1.2.0 use 1.2.1 in the two relevant links – you will get the latest version of InfluxDB. Similarly I found Grafana 4.20 – same place as before but called https://bintray.com/fg2it/deb/download_file?file_path=main%2Fg%2Fgrafana_4.2.0_armhf.deb
I just repeated the Grafana install routine to update it using the new simpler name grafana_4.2.0_armhf.deb – all worked perfectly.
I’m pretty happy with Grafana and I now know how to delete data points (by time) but if you really want to use SQLITE and do your own thing – take a look at the excellent work that Csongor Varga has done – https://www.youtube.com/watch?v=nkKf26oKzhQ
If you like this post – please share a link to it by social media, by email with friends or on your website.
More readers means more feedback means more answers for all of us. Thank you!
Nice post – good job. For similar task I used https://corlysis.com/ It configures Grafana and InfluxDB for me.
Well, I thought I’d give that a go – even though it is very easy now to install your own database and Grafana.
Things didn’t go so well for me. I created an account and created a database. I grabbed an API number.
And from there it went downhill. In their docs they refer to a CURL line – I substituted my database name and api key – and having not the foggiest idea how to use CURL, I tried that from a browser window. Well, that didn’t work so I tried it from the command line on a Raspberry Pi – that didn’t work – bad request.
Then it suggested sending the data via the Influx command line – well if you have that, you have Influx already so why on earth would you then want to send off to a third party.
Finally there was a Python example. Personally I’d have found one in node.js more useful.
After trying the “getting started” I went to “docs” – which simply took me back to “getting started”.
So – I gave Python a go and.. “NameError: name ‘requests’ is not defined”
Comparing that lot to a simple install and a node-red node that lets you drop in data, for now, I think I may stay with the local installation.
But thanks for the info.
I am also battling to publish data to Corlysis. The example Curl works but when I try to publish from NodeRed using the inFkuxDB flow node, I get:
msg : error
“Error: getaddrinfo ENOTFOUND https https:80”.
I don’t think I am setting up my Corlysis server properly. Can you perhaps post your Nodered flow?
easy install instructions for grafana and influxdb : https://www.circuits.dk/install-grafana-influxdb-raspberry/
I wonder if it would work on other machines – I’d love to incorporate these into the script – after all why not – it’s an excellent graphing system.. busy tomorrow dropping off relatives at airport but could be worth investigation next week?
just saying, i already noted some errors in there… have to test the instructions… 1st line:
curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -source /etc/os-release
should be actually be 2 lines…
curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add –
source /etc/os-release
That works
Note that in February Xaomi separated the Mi Flora devices into international and chinese versions, and via the automatic updated loaded firmware which bricked the older international versions. For a fuller story, see the comments on the Google App download. I _hope_ that the devices are still useable from eg. an RPi, but I wouldn’t bet on it.
Annoying, because I bought a cheap Android phone specifically to run the things.
Will
Looking to give this a go when my miflora arrives, just wondering about the PAHO-MQTT bit – will this conflict with Mosquitto running on my pi?
Essentially my pi3 runs nodered and MQTT for some other shed related IoT stuff (ill blog about all this one day) and im looking to use this same pi to connect to the miflora.
The Miflora connects to the Pi (or whatever) by Bluetooth – the code on the Pi talks to the Miflora by Bluetooth – and to your MQTT broker by the PAH-MQTT client… on conflict. I’m using Mosquitto as well.
That’s makes sense was just wondering if they would clash as I’ve one pi doing everything (running node red, mosquito mqqt broker and now planning on added PAHO-MQTT for the miflora)
Well, you’re just adding the client- not the broker – as for lots running on the Pi – I have endless traffic going via MQTT including all the controls, testing, keeping a Nextion display up to date… none of which takes much of the power of the Pi or similar.
You can have multiple clients on one machine – but just one broker – unless of course you change ports for the other.
Just in case anybody wants to use the InfluxDB data with Imperihome I have attached a short node-red routine that converts the graphing request from Imperihome to a format suitable for InfluxDB and then converts the results back again, This uses a very simple database holding time and values for each measurement generated by a single miflora sensor. Because I didn’t use the same device names in the Influx and Imperihome there is a lookup routine that will need to be adapted, Also my Influx data has automatic rollups after 3 months to min/max values so I have dropped the need for grouping to reduce data transfers. Hopefully the node-red import will survive wordpress formatting.
——————————————–
[{“id”:”2edbabe2.4897d4″,”type”:”http in”,”z”:”9347d869.41bd58″,”name”:”History request”,”url”:”/home/devices/:deviceID/:paramKey/histo/:startdate/:enddate”,”method”:”get”,”swaggerDoc”:””,”x”:96,”y”:2304,”wires”:[[“97b8805f.34077”]]},{“id”:”5204c7ac.dc4618″,”type”:”influxdb in”,”z”:”9347d869.41bd58″,”influxdb”:”cfaa88d0.d9dce8″,”name”:”Make DB request “,”query”:””,”x”:475,”y”:2306,”wires”:[[“268d4288.76635e”]]},{“id”:”afbda41f.bcacb8″,”type”:”http response”,”z”:”9347d869.41bd58″,”name”:”Return history data”,”x”:773,”y”:2310,”wires”:[]},{“id”:”268d4288.76635e”,”type”:”function”,”z”:”9347d869.41bd58″,”name”:”Convert date format for Imperihome”,”func”:”var results=msg.payload;\nvar newresults=[];\nfor (i=0; i \” + start + \” AND time < \" + end;\n\nreturn msg;","outputs":1,"noerr":0,"x":277,"y":2374,"wires":[["5204c7ac.dc4618"]]},{"id":"cfaa88d0.d9dce8","type":"influxdb","z":"","hostname":"127.0.0.1","port":"8086","protocol":"http","database":"garden1","name":"Garden DB ","usetls":false,"tls":""}]
Great article Peter.
I’m a keen gardener as well as a hacker so I must get one of these.
Its gone up. Its £15.71 now from Ali
Yeah I did notice that – a shame – still – you get lots of sensors for the money. Just be aware Bluetooth does not have stunningly good range – in my case I had to put a Bluetooth adaptor from the Pi – on a USB extension lead – 2 metres – so it was right up against the wall – and it then reaches into my small garden area. If you’ve a big garden it might be that you want the Bluetooth dongle in the window.. I’m currently running two of these MiFlora units without issue – one of them has been on for months. I wrapped them both in clingfilm as they’re not waterproof – you’ll need to think about that if they are going outside.
Just to name it – anyone tried grafana and/or influxDB as docker image?
Link the config and DB files to local storage make it that easy to update / upgrade.
I have a testinstallation on a x86 host, currently not on a rpi.
regards,
andreas
Some hints on docker for rpi and infulxDB for docker:
Docker
From https://www.raspberrypi.org/blog/docker-comes-to-raspberry-pi/
curl -sSL https://get.docker.com | sh
InfluxDB
From https://hub.docker.com/r/library/influxdb/
$ docker run -p 8086:8086 -v $PWD:/var/lib/influxdb influxdb
Modify $PWD to the directory where you want to store data associated with the InfluxDB container.
Influxdata just released Chronograf – it looks a little bit more lightweight compared with grafana, but i currently dont know if it is really a competitor to grafana… for now.
As with my comments on Elastic Search earlier, I wouldn’t recommend using Docker for this unless you are running on an otherwise empty Pi.
I tried Docker on my NAS and while it worked, there was nowhere near enough memory to make it really useful. The Pi will have similar issues.
Honestly, after the initial installation, updates of both InfluxDB and Grafana are totally painless anyway so there really isn’t any point in the overheads of Docker for this.
Thank you Pete for ruining my weekend 🙂
I have grafana and influxdb up and running and it was well worth it!
BTW:
https://repos.influxdata.com/debian/pool/stable/i/influxdb/influxdb_1.2.2-1_armhf.deb
can be installed with dpkg on the pi!
Garry
neo2 is arm64 and can’t find a recent repository with packages for this arch…
only one found is standard debian repo, but a year old, 2.6 version, we are at 4.2…
http://ftp.us.debian.org/debian/pool/main/g/grafana/
even the suggested repo by grafana site itself (with wrong link, too!) contains only x86+64 arch packages, no arm…
https://packagecloud.io/grafana/testing
anybody, suggestions?
Excellent news Garry. Sadly I did not have the same luck with the NEO2 – (arm64) – can’t seem to get a working influx/grafana setup on arm64 – I thought it might be nice to separate this off from the main board and put it on a tiny Neo2 instead… well, 1/3rd the cost of a Pi! Still – someone might come up with a solution.
Garry, did you use it with node-red-contrib-influxdb?
I got an error: TypeError: Object function Object() { [native code] } has no method ‘assign’
Thanks.
George
Erm I used the influx out node – worked perfectly from day one… Initially using an older Influx then updated Influx to 1.2
I just used the influx out Node like Pete, no problems. Using Influx 1.2.2.
Please note everyone – I’ve updated this blog as both InfluxDB and Grafana have later versions – which I installed without issue, noted near the end of the blog. InfluxDB 1.21 and Grafana 4.2.0 at the time of writing – have fun.
Hi
I don’t know if this was behind the rogue data you had but I noticed recently that if the temperature dropped below zero the data was being reported in the mid 6,500’s. The problem lies in the _parse_data function in the miflora_poller.py program. Negative data is coming from the device in 2’s complement but is not being adjusted for. I found the easiest way to correct was to add to that function a test for the temperature being greater than a sensible limit (say 100 degrees) and if so taking the 2’s complement repeating the conversion and negating.
Steve – do you wish to provide us with your fix to the PY program?
A little less elegant that my initial suggestion but just as effective.
In the last function in miflora_poller.py which is called _parse_data().
After the line
res[MI_TEMPERATURE] = float(data[1] * 256 + data[0]) / 10
Add the following condition
if res[MI_TEMPERATURE]>1000:
res[MI_TEMPERATURE]-=6535.6
which will correct the conversion assuming temperatures of +1000 degrees are not normal in Spain.
(Be aware that wordpress appears to remove the indentation from the line following the IF statement)
I will implement that now – and yes, +1000 is not normal in Spain – mind you +45 will be in a couple of months hopefully 🙂
Can you clarify – that this was meant to go on 2 lines in line with each other – I ask that because I’m not yet used to the spacing sensitivity of Python… or was this supposed to be all one line??
I’ll assume the second line was meant to be indented – and that makes sense…
PEOPLE – anyone using that code should add Steve’s mod.
Peter, maybe this can be useful for you:
Working with time series data in Node Red with SQLite
Working with time-series data – Part 2
Found an affordable (about 100€) and hackable (using ESPEasy) weather station, with wind and rain sensors!
https://www.letscontrolit.com/wiki/index.php/VentusW266
Pete – just a heads up for anyone following your examples here using a RPi and the stock Raspbian version…
I copied your Node-RED Flow using MQTT input -> Function -> InfluxDB output (with minor tweaks to the function). I’m testing a BME280/ESP-12F setup and everything was working fine but no data was being written to InfluxDB. I noticed an error indicating “Object() has no method ‘assign'”. A quick Google and I found the pre-installed Node.js is an old one and needed to be updated.
we never use stock nodejs… if you use THE script, you get node 4 (til a couple of weeks ago) or 6 (after that)… try from shell to run: node -v
See Antonio’s comment.
Apologies if I have got the wrong end of the stick but shouldn’t it be
sudo apt-get install python3-pip (not python-pip) for the correct version of pip.
Also shouldn’t it be
sudo pip3 install paho-mqtt (not paho-client) – couldn’t get paho-client to wok at all.
Hmm, I’ll have to pass this one to my friend Jay, Steve as I didn’t originate that bit. Jay, you in here?
However – that did work for me – weeks later that sensor is sitting happily collecting garden data.
I’ve been using InfluxDB and Graphana for ages now. I replaced MongoDB with InfluxDB as MongoDB is horrid on 32bit machines.
InfluxDB is great as long as you don’t push the number of data points too far. Use the auto-trim and summarise features. I trim my minute data to a week but create 1hr average data for long-term charts.
I’ve often wondered about editing data in the time based databases but that’s where the maths functions of Graphite would come into their own, you wouldn’t be able to remove the data point from the database but I’m sure you could remove it from the graph see http://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.removeAboveValue
I know you said before about not finding any good guides for installing Graphite, these are the 2 I referred to for installing on my BBB.
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-graphite-on-an-ubuntu-14-04-server
https://www.linode.com/docs/uptime/monitoring/deploy-graphite-with-grafana-on-ubuntu-14-04
Also you could probably just add some code into a function to do a sanity check on the incoming data and throw it out if it’s not between a min and max threshold before sending it to influx.
Adam
I have to say that I am ABSOLUTELY having second thoughts about INFLUX – I think it has been designed for masochists for masochists.
I have a duff value – temperature sensor reading around reboot of a unit produced a value of -4000 – well you can imagine how well that scales on the graph. In any sensible database it would be something like…
delete from logger where uk_temp<40
But no - you can't DO that apparently. I cannot figure out how to simply delete a point.
Thanks, I’ll try again later but what’s a Gragana file ? 😉
Trust me the way my stomach feels this morning – managing a reply is about all I can manage – expecting perfection is way OTT. Fixed.
I hope you’re feeling better, I got it all installed I’m now struggling to get data from an existing MQTT stream into InfluxDB.
I guess you formatted your MQTT stream as JSON ? mine is just a topic “Temperatures/Exterior/Ambient” with a data value, I’ve tried to format it correctly but I can’t seem to get the hang of it, I assume I’ve got to convert it to a name and a value formatted as a JSON string.
I’ve got this so far but I don’t think it looks right
msg.measurement=”Outside temperature”+”:”;msg.payload=”:”+”‘”+tmp+”‘”; node.send(msg);
can you help with the right format?
Hi Phil – I’m off to bed – but – I didn’t format anything – the example PYTHON code did it. Then in the orange FUNCTION I simply used JSON.parse to turn that into an object I could use.
In the general case – given names fred and brian, with values 50 and 60, a suitable JSON string would be…
{“fred”:50,”brian”:60}
If the answers were strings rather than numbers – they would be in quotes too.
OK, goodnight.
The thing is my data is used elsewhere and not formatted as JSON like you Python data.
I’ll keep trying though.
Pete,
Tried to follow your example but no go, did you download the armhf file from somewhere first?
I looked on the Grafana site but can’t find it, the download they show is for an AMD64.
Hi there Phil….
That’s because, in the instructions, I missed off the all important link to the Grafana build for Raspberry Pi – note that I have updated the blog – make sure you get the right one for your Pi.
I’m loving it – back in the UK now and I can access all my readings over in Spain.
To add another recommendation of graphite – I’ve been using this for storage for some time now (since 2008!). It has automatic rollup of data to downsample as it gets older, which ensures your database doesn’t grow out of control in size.
Influxdb it seems possible to setup downsampling, but a lot trickier. Worth thinking about in terms of having the data useful a few years down the line.
I keep looking but every setup blog I’ve seen so far is horrendous… surely by now there’s a single page setup somewhere? Bearing in mind I already have Grafite.
Pete.
What sort of range are you getting out of the Mi Flora ?
I could have sworn I replied in here….
OK – RANGE – 9ft of air then a breezeblock wall then 6ft of air – it retries once or twice but always gets there in the end. I’m leaving this now, we’re back off to the UK and I’ll be able to monitor it from there. I’m ordering another Mi Flora so I’ll have one in each country. Good toy. How long it lasts and how long the battery lasts – something for another blog.
Graphite – I’ll take a second look but at first glance it did NOT look like anyone had gotten around to making a friendly installation for the Pi. I was also thinking – the lack of smoothing in INFLUX can presumably be sorted by using the SMOOTH function in Node-Red.
An interesting device.
You might want to consider using the python json module:
import json
and use the json.dumps function combined with a dictionary as the payload, for example:
json.dumps(dict(item1=1, item2=2, item3=str(3)))
results in the following mqtt payload:
{“item1”: 1, “item2”: 2, “item3”: “3”}
If I’ve understood your code correctly that means your publish statement would end up being:
mqttc.publish(“miflora/garden”, json.dumps(dict(battery=str(poller.parameter_value(MI_BATTERY)), version=str(poller.firmware_version()), sunlight=str(poller.parameter_value(MI_LIGHT)), temperature=str(poller.parameter_value(“temperature”)), moisture=str(poller.parameter_value(MI_MOISTURE)), fertility=str(poller.parameter_value(MI_CONDUCTIVITY)))))
Saves a bit of escaping …
Great blog – keep finding things of interest to read and try
David
Grafana is awesome Pete, though I went a different route and used Graphite for the backend. It has it’s own lightweight database called whisper for holding the time-series data (you don’t need any other type of database like SQLITE when looking at metrics over a time period like temperature/humidity/etc).
https://graphiteapp.org/
It’s a bit more involved to install but is definitely worth it as you get a load of math functions which can be applied to the graphs.
http://graphite.readthedocs.io/en/latest/functions.html
Also there are countless frontends like Grafana which can be used with Graphite, look at all the different integrations here http://graphite.readthedocs.io/en/latest/tools.html
Regards
Adam
graphite looks great – a bunch of guys that I used to work with were fans of graphite for their application that they developed on AWS. It reminds me of ELK stack which is also fabulous for free! https://www.elastic.co/
kibana does really nice graphs and charts which update in real-time bound to the backend elastic DB
Hi Adam – yet I agree Grafana is awesome – especially after this morning where I just sat down with coffee and easily filtered out that bad value by simply selecting sensible signals in Grafana itself… it’s just so annoying that it is not supported by a nice easy to use database like SQLITE – wherein I could have deleted that bad value in seconds. I find it even more amazing because they use SQLITE under the hood.
It also comes up well on mobile phones which is a plus. I’m busy monitoring watering in the garden in Spain (remotely as I’m in the UK). My neighbour watered the plants 3 days ago and there is still far more moisture in the ground than I would have expected…
Here’s another shameless plug for elasticsearch. No I don’t work for them. Yes you can delete documents from the elasticsearch db. Yes there is node-red support. Yes nice real-time graphs/charts in kibana. Also available as a complete stack from a number of sources..
I’ll bite. Got a recommended RPI install?
I’ve never installed this on a pi so I can’t offer a “defacto fabulous package”. Besides I guess what you’re going to need will depend on what distro you’re running.
I suggest you start at the official source here:
https://www.elastic.co/downloads
There are pre-made package manager packages available for a number of distros there too. See https://www.elastic.co/start
You’ll want both elasticsearch (the document persistence store) and kibana (the UI stack).
Logstash might be useful too but is not mandatory.. think of logstash as linux pipes on steroids for manipulating/transforming event data from any form -> a stored elasticsearch document
Note this stuff can be installed theoretically anywhere on the LAN, it does not necessarily need to be installed on the pi itself and the node-red -> elasticsearch stuff should work fine as long as there’s connectivity between the systems.
Doesn’t Elastic require Java? If so, that’s rather a killer for a Pi or similar SBC since it needs so much memory to do anything useful.
Elasticsearch does require Java, it’s a Java built platform.
As I mentioned above I’ve never installed it on a Pi so I can’t offer any personal 1st hand experience advice on good/bad there. However a google search tells me there are number of people who have done this so it’s certainly doable and some even report it’s surprisingly good despite the constrained resources.
At the risk of stating the obvious.. how well it’s going to actually work in reality is going to depend on a number of factors: what’s it sat alongside? what other processes are running? how much load is ES under etc?
I’d like to try it myself but I don’t have a Pi so I can’t so I’m going to have to say: Elasticsearch running on a Pi – YMMV
Well my Pi is already pretty heavily loaded and my experience with Java is that it is pretty useless without loads of memory so I will avoid. Probably OK though if you have a spare Pi to run it on.
Hi Pete,
Are you sending MQTT from a remote location or just logging into a node-red instance in your Spain location? I’m wondering because I’m trying to come up with a way to send sensor data from a remotely located ESP8266.
Thanks!
The MQTT brokers in Spain and the UK talk to each other – in the way ESPs talk to the broker – and exchange info.