Grafana and InfluxDB

Peter ScargillGrafana, InFluxDB, Raspberry Pi 2, Raspberry Pi 3, Raspbian, stretch32 Comments on Grafana and InfluxDB

GrafanaSome time ago I wrote a blog entry about garden sensors inside of which was buried some information about using Grafana and InfluxDB. At the time the install was not that easy – and when along came STRETCH for the Raspberry Pi (2 and 3) it got worse.

However, reader Antonio and I have been working on this and we have an install for the latest Grafana with Influx.

But firstly let’s discuss what this is all about.  Back in the dim and distant past, people were happy with simply line graphs and horrible fonts to display data until along came a number of services to produce pretty information which was easy to generate. There are many such services where you can send your Raspberry Pi or simply data, which may be from a variety of sources to a service – sometimes free, sometimes not – to log and display your data.

I’ve never been that happy with this though I did write a node to work with one of the providers – but I’ve always preferred the idea of keeping the data locally. I worked originally with MYSQL and various graphing packages that ran on Debian but there was never a simple to use system until Grafana came by.

Grafana can work with a variety of databases – but works well alongside InfluxDB. I’m not saying InfluxDB is great – but it DOES make life easy when working with Grafana because it uses Time Series data – in which all you send is a data series name – and a current value – that’s it… the rest is done in the background. You don’t need to worry about time and creating records or tables etc.

For Node-Red there is a node in which you simply send two items – msg.payload – which is the value you want to graph – and msg.measurement which is the name of the series. I usually use a function to copy the more familiar msg.topic into msg.measurement – that makes life even easier as you can send incoming MQTT data straight to the database for use by Grafana.

So having introduced the subject – what does Grafana look like in action? It runs from any modern browser – I use Chrome – and looks like this.

Grafana

Well, that’s what my setup looks like – yours may end up looking completely different. A key point here is the ease of adding data. As new points are added for previously unused sets – they just become available in Grafana for you to use, via drop-down menus.

Now, while Antonio and I have managed to get this running on Stretch for the Pi2 and Pi3 I (after updating “the script” to work with Stretch) – would love to hear details of working up to date (i.e. version 4.4 or better Grafana) running on the likes of Orange Pi Zero, NanoPi Neo, M3 etc…  that would be nice. A quick look at Xenial suggests they’re still on version 2.6 !!

Once the setup is up and running – you need to create a database. InfluxDB used to have a web interface for this but it seems they’ve binned it – there is a command line interface however and all you actually need it for is creating the database – along the lines of (from the command line – I used user PI)

infliux
create database “logger”
exit

I called my database logger for no real reason.

You can also create a “user” in this interface – but it won’t be needed unless you enable password authorisation (see later on)

CREATE USER admin WITH PASSWORD ‘<password>’ WITH ALL PRIVILEGES

Once that’s done – in Grafana itself – you add the database logger and create a new blank dashboard – and from there it is a simple matter to add graphs etc. using the data.  In Node-Red there’s a node called InfluxDB and I’ve described above how to use that. With that Node-Red installation are other useful nodes including a batch node. I’ll leave it to the reader to investigate further.

Node-Red

Here is our first attempt at making the Pi2/3-with-stretch installation simple – this does not include creating the database (though it could) or adding any security. By default anyone can create an account in the Grafana database. There is a config option to stop that once you’ve created an account (port 3000).

So what we have is a simple script – and as with “the script” – just make a file (say, grafanasetup.sh) – copy this info to it, make sure the CRLF format is LF only for Linux – save the file and give it execute permission. Then run it – and somewhat later on – you should have a working InfluxDB/Grafana setup.

[pcsh lang=”bash” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]

# Create InfluxDB and Grafana 4.4.3 (latest)
source /etc/os-release
OPSYS=${ID^^}
curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
test $VERSION_ID = "8" && echo "deb https://repos.influxdata.com/debian jessie stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
test $VERSION_ID = "9" && echo "deb https://repos.influxdata.com/debian jessie stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
curl https://bintray.com/user/downloadSubjectPublicKey?username=bintray | sudo apt-key add -
[[ $OPSYS == *"BIAN"* ]] && [[ $(uname -m) == *"armv6"* ]] && echo "deb https://dl.bintray.com/fg2it/deb-rpi-1b jessie main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
[[ $OPSYS == *"BIAN"* ]] && [[ $(uname -m) == *"armv7l"* ]] && echo "deb https://dl.bintray.com/fg2it/deb jessie main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get -y $AQUIET remove --purge grafana grafana-data
sudo apt-get -y $AQUIET autoremove
sudo apt-get -y update && sudo apt-get install -y apt-transport-https curl influxdb grafana
sudo systemctl daemon-reload
sudo systemctl enable influxdb
sudo systemctl start influxdb
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

[/pcsh]

September 10, 2017

I’ve just tried this on a Roseapple Pi recently upgraded to “Stretch” Debian. To be honest I made such a mess of the upgrade I’m surprised it worked at all – but it did. I added the database to Influx as described above – then went to Grafana. At this point Grafana had nothing but the default username and password…  admin and admin – a good idea to change that – easy. The job then was to add that new database – easy – add datasource – type influxdb – any username and pass as you’ve not set up security yet. That’s it – you’re then ready to add a dashboard.

Chronograf

Think of this as the “extras” section as we have a working setup above as long as security isn’t an issue. While reading about InfluxDB, I thought I’d look to see why they gave up on the graphical setup interface. Well, it turns out there is something called Chronograf and up to now I’ve not seen much about it – however in this French post I found this:

sudo wget https://dl.influxdata.com/chronograf/nightlies/chronograf_nightly_armhf.deb
sudo dpkg -i chronograf_nightly_armhf.deb
sudo systemctl start chronograf

With nothing to lose as I’d backed up my installation, I gave it a whirl. With no apparent errors I tried bringing up Chrome browser and pointing to the Pi at port 8888

Sure enough it worked and though creating a database didn’t seem to do any good – once I had some data thanks to the above setup…. I could query data and even create a user with password – but when connecting to Grafana I found I could still access the data without reference to that user or password.

Passwords

It seems you still have to go into the config file to set this up (I could be wrong). I found the config file at /etc/influxdb and enabling security under the http section.  I set that to true and rebooted after which – the setup in Grafana required a username and password in the data source section. Not https: at this point but better than nothing. Here (thanks to Antonio) is a simple command line to change that config file.

sudo sed -i -e ‘s/.*auth-enabled = false/  auth-enabled = true/g’ /etc/influxdb/influxdb.conf

64-bit Success

With all of this success, the only thing that was missing was to put Grafana onto something a little more meaty – a 64 bit machine.  As it happens I’m messing with NEO PLUS2 boards right now and when I tried to install on them – the database worked (InfluxDB) but Grafana would only install an early version. Well, thanks to Antonio asking all the right questions and spending time on this – here’s a list of instructions – which worked on the Plus2 at least.

[pcsh lang=”bash” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]

sudo apt-get -y update
sudo apt-get install -y apt-transport-https curl
sudo dpkg --add-architecture armhf
echo "deb [arch=armhf] https://dl.bintray.com/fg2it/deb jessie main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
curl https://bintray.com/user/downloadSubjectPublicKey?username=bintray | sudo apt-key add -
sudo apt-get -y update
sudo apt-get -y remove --purge grafana grafana-data
sudo rm -rf /var/log/grafana /etc/grafana
sudo apt-get -y update && sudo apt-get install -y grafana:armhf
sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

[/pcsh]

and if you need to start from scratch…

[pcsh lang=”bash” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]

sudo apt-get -y remove --purge grafana grafana-data
sudo rm -rf /var/log/grafana /etc/grafana
sudo apt-get -y update && sudo apt-get install -y grafana:armhf

[/pcsh]

October 28 2017

I just installed on a pair of NEO2 boards which are sitting in Spain running on solar while I’m suffering the rain in Britain….  the installation went flawlessly putting the boards at version 4.6 – be wary of any installation that sets you back to version 2.x

32 thoughts on “Grafana and InfluxDB”

  1. take a look at Frenck latest video, on how he sets up grafana+influxdb+glances on home assistant, but more of that look at how powerful and easy is to push in influx and pull out with grafana all your crazy statistics and graphs 🙂

    p.s.: Frenck is one of the major developer of Hass.IO addons, you can see him setup up some of them in the video…

    1. yuppie! Frenck’s streaming live while coding the long awaited Nginx Proxy Manager addon! https://www.youtube.com/watch?v=X2u9cK9E1sg

      a reverse proxy allows you to have an https exposed dyndns for example, and have the SAME (with a SINGLE exposed public port) redirect for example:
      https://nodered.something.net to your local nodered on port 1880, while https://ha.something.net can redirect to home assistant on the same machine on port 8123 OR on a different machine at all, and so on… and terminating (offloading) the SSL session on the proxy allows you to have all you exposed to internet traffic encrypted, but all your local traffic in faster and easier to manage http format… this addon is a piece of art for all this stuff: https://nginxproxymanager.jc21.com/

  3. I did a new install. It didn’t complete by the time I had to go to bed so I don’t know if there were any hiccups. I assume it will be fine.

    Thank you for your help.

    1. mods are already in script from starting of may, even if not detailed here… they install grafana and influx in different ways then before, thanks to proper release of arm64 packages… but other then that, not so much changes for other platforms, it only changed the way they are installed, not how they work…

      oh, notice: none of them is populated with the usual credentials put in at script run beginning, so they’re at their defaults (admin admin for grafana, none for influx, AFAIK)

  5. Peter, we should modify this article given recent mods i made to the script a couple of weeks ago, and the release of proper arm64 deb packages, the install info at the end are now obsolete…

  6. Just updated Grafana to version 4.6 without issue using the script that Antonio put together – on a FriendlyArm NEO2+ board.

  7. Re the loss of the admin gui from InfluxDB.

    Just to say that there is a quite nice plugin for the later versions of Graphana that does a nice job of letting you easily add standard admin queries.

    In fact, there are some amazing plugins for Graphana now. Checkout the one that builds a box/flow diagram from a db query!

    1. Ah, now you see I didn’t know that. Mind you the command line version of the admin is quite reasonable… if you come across links that make it easy for people – please do put them in here. Next time I’m working on Grafana (I put that and influxDB on all my boards by default now) I’ll take a look – meanwhile my original in Spain is happily collecting weather data – day in, day out. Best thing I’ve found so far for these boards.

  9. sent info to Peter on how to force install of armhf packages on arm64 boards, so now even neo2 is a good choice… he’s doing the tests and will post his thoughts when done

  10. I can see that one script does some backing up and the other updates Node Red, but what about the other new additions to the script? What would happen if I ran the whole new script on my existing set up?
    thanks

    1. mmm… additions are new nodes, not much more, we are working mainly to let it run on as many SBC as possible, and correct some minor bug… for new nodes, you can run just this:

      cd ~/.node-red
      npm install moment node-red-contrib-config node-red-contrib-grove node-red-contrib-diode node-red-contrib-bigtimer node-red-contrib-esplogin node-red-contrib-timeout node-red-node-openweathermap node-red-node-google node-red-node-sqlite node-red-node-emoncms node-red-node-geofence node-red-contrib-ivona node-red-contrib-moment node-red-contrib-particle node-red-contrib-web-worldmap node-red-contrib-sqldbs node-red-contrib-ramp-thermostat node-red-contrib-graphs i2c-bus node-red-contrib-isonline node-red-node-ping node-red-node-random node-red-node-smooth node-red-contrib-npm node-red-node-arduino node-red-contrib-file-function node-red-contrib-boolean-logic node-red-contrib-blynk-websockets node-red-contrib-chatbot node-red-dashboard node-red-node-darksky node-red-node-serialport node-red-contrib-owntracks node-red-contrib-opi-gpio

  12. I’m sure it’s been asked before but.. I have a working Pi3 all set up with Mqtt, Node red etc from an earlier script. What happens if I run the most recent script over the top of it? Will I lose my customization?

  13. Pete,

    Just ran this script on an OrangePiPlus2 running Armbian. Just trying to work out whether I’ve struck an Armbian only error, or not.

    When enabling influxdb, an error is returned ……

    Executing /lib/systemd/systemd-sysv-install enable influxdb
    Failed to execute operation: Too many levels of symbolic links

    Just out of interest (NOT looking for technical support from your good self !) – Did you see this (or a similar) error when you were developing this script ? If not, then I’ll assume that the error is either Armbian specific, or maybe OrangePi hardware specific. My gut feeling at the moment is that it is Armbian specific, and probably relates to the specific versions of the packages available under Armbian.

    Anyway, I’ll let you know when I’ve got it working – the info might be useful to other OrangePi users.

    Regards,

    RoyG

    1. Update —- I managed to successfully complete the configuration if the influxdb service after a reboot.

      Then the grafana service refused to enable, plus further investigation showed that the grafana install has some errors that I hadn’t noticed.

      I eventually did apt-get remove –purge grafana grafana-data.

      Then successfully did a reinstall of grafana from this link …..

      http://www.d0wn.com/install-grafana-4-on-a-pi/

      Both applications seem to have installed OK. Now to do some testing.

      Many thanks for the lead. Grafana will fit very nicely into two of the projects that I’ve got underway at the moment. Somehow, your posts often seem to come up with relevant info just at the time I need that info 🙂

      Thanks,

      Roy Gilby

    2. Not having used an OrangePiPlus2 I can’t tell you but if it is 64 bit, Grafana is not going to run and we can’t find a 64 bit repository. I was trying to get it to run on a NEO2 (which would have been nice). Actually we did manage with some messing around to get InfluxDB running on the NEO2 but not Grafana – I was at it late last night and ended up with version 2.6 which is ancient and not very good. I note from the source that version 5 is there but not released – it would be really nice if they’d do official releases for ARM devices, 32 bit and 64 bit. Do let us know if you have any success.

      1. Pete,

        I’ve got Grafana up and running using the link in my previous reply.

        Just working on InfluxDB now. InfluxDB serviceis running, but the influx command line tool isn’t installed …. so that’s the next step.

        RoyG

        1. on an ORIGINAL pi, you can use the script in this page, we tested it on both raspbian jessie and stretch… influx will work on ubuntu based distro, too, but grafana not for now, can’t find an updated package nor their src.deb ones, and don’t want to go down the road of standard compile… nor we can find arm64 packages, so add your request for them to the link in my other comment, thanks 🙂

        2. oh, usually if the command line influx is not there, you’re using an old package… check if it’s 2.6.0:
          dpkg -l | grep influx
          you need newer version for the cli utility, use script in this page

        3. Pete,

          I ended up installing and purging InfluxDB, and reinstalling using information from this site: http://www.d0wn.com/influx-db-1-2-0-on-armbian-or-raspberry-pi/

          The influc command line tool then appeared, and I was able to create the database that I needed for Grafana. And, I was then able to configure that DB in Grafana, and that all works.

          So, time permitting, I should be able to start pumping some data into the system in the day or so from my personal weather station, and then we’ll see what Grafana can do – or probably more accurately, what my old age brain can make of Graphana.

          I suspect that the problems I’ve encountered with installing Grafana and Influxdb on Armbian on the O Pi have been a result of out of date or broken packages.

          Chronograph installs and runs OK on the O Pi. There is plenty of English language doco for Chronograph at https://docs.influxdata.com/chronograf/v1.3/introduction/.

          I just have to get Chronograph to listen on a port other than 8888, as that is the same port that the OrangePi Monitor program listens on. Just been trying to work out how to change the port number or uninstall OrangePi Monitor, or alternatively change the port number that Chronograph listens on. As yet unsolved, but the doco for Chronograph looks good, so I’m sure the answer is just waiting to be read.

          Anyway, that’s a job for Wednesday.

          Regards,

          RoyG

        1. How to Re-Configure The Default Port for the armbianmonitor Daemon.

          The armbianmonitor daemon is called rpimonitord.

          Edit the following rpimonitord script:

          sudo nano /usr/bin/rpimonitord

          Search for the following line, and change the “8888” to a network port number that does not conflict with any otherports already in use on your computer.

          $this->{‘daemon’}->{‘port’} ||= 8888;

          The network port that rimonitord is listening on is mentioned two other times in the script, but these mentions are in comments. For completeness, search for these other two mentions of “8888” and make the same port number changes.

          Rebooting the computer seems to be the easiest way to ensure that the rpimonitord Daemon has closed any previous instances, and is now listening only on the newly specified port.

          Armbianmonitor then starts up (in my case) on http://localhost:8778, whilst Chronograph then starts up successfully on http://localhost:8888.

          Now, it’s onto trying to make use of these applications.

          Many thanks,

          RoyG

Comments are closed.