Monthly Archives: April 2017

Garden Data Collection

Grafana, InFluxDB, Node, Node Red, node-red-dashboard,

MiFloraThis entry is all about InfluxDB, Grafana and a cute little Bluetooth garden sensor!

Some time ago a pal of mine and I did some swaps of spare kit and I ended up with a Mi Flora sensor.  A rather pleasing looking device,  the MI Flora has a 3v lithium button cell and can transmit by Bluetooth LE, the light level, temperature, battery level and moisture.

Battery is supposed to last for months. I’ve provided an AliExpress link above which puts the units at under £8 Inc. postage. No doubt that price will vary dramatically depending on where you chose to spend your money. Amazon charge more like £13.

So having pulled off the protective tab which meant the battery would work, I put the unit to one side as I had no idea how to read it.

A week or so later in Spain, with poor weather my pal Jay came on to ask me if I’d used it. I had not – and so we decided to give it a go. As it turned out I had some cheap Bluetooth 4 dongles I’d bought on a whim last year. I plugged one into my Raspberry Pi 2. The program “hcitool” lets you scan for Bluetooth devices and as it happens it was already installed on the Pi. From the command prompt:

sudo hcitool lescan

Instant success as the Mi Flora device with it’s MAC address was revealed. I made a note of the MAC address.

I would need Python 3 to experiment – but this was already installed on the Pi and some code to get the information from the unit. This code turned out to be ok with a little modification. I ended up creating a folder called /home/pi/myflora and dumping the lot in there.

Jay had modified the program to send MQTT. That of course needed the PAHO client. In order to do that I needed something called PIP.

sudo apt-get install python3-pip

sudo pip3 install paho-mqtt

Note NOT pip (which works for Python 2.7) but pip3.

I then added to the MI flora directory a test program which Jay sent me called bathroom.py – presumably because he’d put the sensor in his bathroom – I adjusted the output slightly as I was putting my unit in the garden

import sys
import paho.mqtt.client as mqtt

from miflora.miflora_poller import MiFloraPoller, \
MI_CONDUCTIVITY, MI_MOISTURE, MI_LIGHT, MI_TEMPERATURE, MI_BATTERY

poller = MiFloraPoller("C4:7C:8D:32:08:63")
print("Getting data from Mi Flora")
print("FW: {}".format(poller.firmware_version()))
print("Name: {}".format(poller.name()))
print("Temperature: {}".format(poller.parameter_value(MI_TEMPERATURE)))
print("Moisture: {}".format(poller.parameter_value(MI_MOISTURE)))
print("Light: {}".format(poller.parameter_value(MI_LIGHT)))
print("Conductivity: {}".format(poller.parameter_value(MI_CONDUCTIVITY)))
print("Battery: {}".format(poller.parameter_value(MI_BATTERY)))

# Publishing the results to MQTT
mqttc = mqtt.Client("miflorabathroom")
mqttc.username_pw_set("admin","xxxx")
mqttc.connect("192.168.1.19", 1883)
mqttc.publish("miflora/garden", "{ \"battery\" : " + str(poller.parameter_value(MI_BATTERY)) + ", \"version\" : \"" + str(poller.firmware_version()) + "\", \"sunlight\" : " + str(poller.parameter_value(MI_LIGHT)) + ", \"temperature\" : " + str(poller.parameter_value(MI_TEMPERATURE)) + ", \"moisture\" : " + str(poller.parameter_value(MI_MOISTURE)) + ", \"fertility\" : " + str(poller.parameter_value(MI_CONDUCTIVITY)) + " }")
mqttc.loop(2)
# End of MQTT section

So the idea was that running “python3 garden.py” would send some MQTT to my Mosquitto MQTT broker – and indeed it did (change MAC address for the MI Flora and MQTT details in the above).  Subscribing to “miflora/garden” did the trick – I could fire off the program and with in seconds I’d see the resulting JSON package coming out into my MQTT-Spy client.

Next job was to get Node-Red talking to the MI Flora – I used an INJECT node, set to retrigger every 15 minutes – firing off to an EXEC node which contained nothing more than “python3 /home/pi/miflora/bathroom.py”

That worked – so now I had a regular supply of data in the form of an easy to use JSON package.

I split the package up – and fed it into a Node-Red Dashboard graph – and that was my first disappointment. If you fire more than 4 items into those graphs – the popup fails to show all of them – and the graphs are not very good anyway. I think a lot more work needs to go into that.

I do use GROVESTREAMS on-line (which is REALLY flexible) to store data remotely but there’s a limit as to how far back you can read for free – that was not the case when I first started using them and wrote the Node-Red-Contrib-Grove node to make it all easy.

So now I was on a search for a decent graphics package that would run on the Raspberry Pi (or similar). NOT as easy as it sound,  unless you’re very easily pleased when it comes to graphics.

After a lot of experimenting I decided to give Grafana a shot. I think if I started writing this blog again, I would would have Grafana in the title because this really, as it turns out, is the star of the show.

Having seen a few screenshots, I had visions of populating  a SQLITE 3 database and firing that data into Grafana.

Erm, no. Despite (wait for it) using SQLITE internally to store users and dashboards, the package does not actually support SQLITE for data !!!  I've explained that now, to save you some confusion later. When I did the install, it kept referring to SQLITE and could I HELL figure out how to set up a database for my data – that’s why.

As a Pi user (that means SD use so I’m not keen on MySQL as SD has limited WRITE capability and I worry about databases shortening the life of the SD - see the later article on SD life - very informative) the options for data sources did not look appealing to me at all, the least offensive being INFLUX and that is the one I settled on – partly as there is a node for it available for Node-Red.  Despite the pretty website this is still very preliminary at version 1.02. Influx doesn’t use terms like tables and fields which kept me going for a couple of hours – but it is good at storing time-stamped data – indeed you don’t even do the time-stamping. You create an empty database and start firing data at the DB in pairs – stream name – and data value – it really is very simple once you get started. I guess you could think of it as a single table with pairs of data-name and data-value.  I went down a few dead ends and discovered if you got your stream names wrong – good luck renaming them as INFLUX does not support renaming!

Anyway, it ‘s all very easy once you get started. I did things backwards and installed Grafana first. Here’s what I did – no guarantee it will work for you.

You might want to consider reading this before using InfluxDB on the Pi for long-term use - there are very few build options for small SBCs and this article gives the impression it likes more RAM than we would normally have on the likes of the Pi. I do have to say though that it is working just fine on mine after several days of operation. Comments welcome.

I grabbed the Grafana file for my Pi from here. https://github.com/fg2it/grafana-on-raspberry

sudo dpkg -i grafana_4.1.2-1487023783_armhf.deb
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server

That was Grafana up and running on port 3000 – user name admin, password admin to actually DO anything – and quickly change that password!!! I did that, went into the ini file and disabled user registration as the admin can do that and opened up port 3000 redirect in my router so I could access this from the outside world. That all worked well.

wget http://ftp.us.debian.org/debian/pool/main/i/influxdb/influxdb_1.0.2+dfsg1-1_armhf.deb

sudo dpkg -i influxdb_1.0.2+dfsg1-1_armhf.deb

That set up influx and that ran without further ado at port 8083. I did NOT open that up to the outside world. In there I created a new empty database called “logger”

logger

In Grafana – which supports influx without any changes, I followed Engineer John’s instructions to set up the data-source and a new Dashboard.

So the next thing – would be to get some data.  I played with this for some time – should I keep incoming data source names simple – as you can’t rename them? Then Jay pointed out that in Grafana you can ALIAS names. That did it. So in influxDB node for Node-Red, you fire in a payload – and depending where you want to put your data – you use msg.measurement.. Yes, exactly, what’s wrong with msg.topic you might ask. So here is my little MI Flora MQTT node firing values into InfluxDB

MiFlora[6]

And that yellow function block…

node.status({fill:"blue",shape:"dot",text: msg.payload});

tmp=JSON.parse(msg.payload);

msg.measurement="battery"; msg.payload=tmp.battery; node.send(msg);
msg.measurement="sunlight"; msg.payload=tmp.sunlight; node.send(msg);
msg.measurement="temperature"; msg.payload=tmp.temperature; node.send(msg);
msg.measurement="moisture"; msg.payload=tmp.moisture; node.send(msg);
msg.measurement="fertility"; msg.payload=tmp.fertility; node.send(msg);

Yes, I know – far better ways to do this – but it was my first stab at it and it worked. For general use however I added an incoming function that merely copies msg.topic into msg.measurement and that means any single data-source coming in via MQTT can be blasted straight into InFluxDB.

Yes, that’s write – you don’t create tables or fields – you just send in field names and data and that’s it!

I also fed in some data from my general sensors – with horrible topics like pergola/lighting but of course thanks to the Grafana alias ability – names like that would not show up in Dashboards.

Having stored up some data it was time to get it into graphs – I took the VERY simplest approach of showing lines as I have at this point no idea what some of the Grafana functions do – I DO know that smoothing the data might be fun as Grafana relies on the data-source to do that and Influx doesn’t do that.  What IS nice is that you don’t have to worry about things like “I want the last 5 minute’s worth” or “show me the last month’s worth” as Grafana does all of that for you – from here  it gets easy.

New dashboard and under metrics ADD ROW – takes an hour or so to get to grips with this..

Grafana setup

I’ve expanded the top one here – I have 3 lines in that one graph – see the blank ALIAS BY  - if you want to change the source name (in this case battery) put something in there.

And that  is how I got to this point…

Grafana imagery - my stats

So top left you see battery status from my deep discharge battery and  the two MI Flora units, then moisture levels from the two units – light levels – and on the right centre – a whole host of temperature readings, overall humidity and finally just to play with – the UK temperature split off and displayed differently.

There’s a lot to take in – and I am as you can see NOT an expert at this yet so before firing questions back – you might want to spend some time looking at the links on this blog entry and using Google. If you think another database is better by all means write in to say why – and if you think there’s a better easy-to-fit-in-Pi graphing system, preferable one that works with SQLITE – again do let us know. I started all of this with zero information late afternoon yesterday and had it all working by the end of last night – so it’s not THAT big a deal even if my clumsy writing makes it look that way.

Not entirely sure what’s going on the this battery level on my first day – but it did occur to me that the unit is intended for putting into plant-pots – not sitting outside in the sopping rain so I’ve covered it in cling-film!! As we’re off back to the UK for a few weeks,  I don’t want to come back to a rusted heap.  Note the Mi Flora battery at the top – seems to be self-recovering!!

Update: the little Bluetooth sensors have now been running for months and the battery levels remain fine – the Bluetooth range however is crap – best to have a Bluetooth dongle nearby to talk to them – don’t even think about two thicknesses of wall!

Grafana output

Update 27/03/2017: I've been hitting issues with out of bounds temperature data – i.e. a couple of duff readings crept in - with values like 2000 and -64  which of course flattened the data completely. I realised quickly that InfluxDB has no DELETE function and as far as I could tell - things like comparing data with a value made the graph line disappear.

I put something up on the web and someone came back and asked if I was using InfluxDB 1.2 which apparently has a DELETE function. Well, I wasn't - I was way back at 1.01 - worse - the same folder where that came from only went up to 1.1

However, thanks to THIS helpful chap - I easily updated to Influx 1.2 without the issue he described. I STILL can't add something like "where value < 100" but maybe I can delete ??? Anyone with more info on this please by all means chip in...

Update 18/04/2017: I've just fitted the second of these units and also fixed a minor error in the code above.  All that is needed for a second unit is a copy of the program with different MAC address and different MQTT publish address. Sure you could use the same one twice with command line parameters but I took the easy option.

Bluetooth RANGE is becoming an issue. The Pi is in my office - breezeblock wall then the garden - the second unit I tried maybe 15ft or so from the office wall - not a chance.  Clearly BT4 is good for power, not so good for range.

So - there's a nice job for an ESP32... BT in - WIFI out.  A relay!

Update 30/04/2017: At the time of writing, InfluxDB is up to version 1.2.2 and if you follow the link I gave above but instead of 1.2.0  use 1.2.1 in the two relevant links - you will get the latest version of InfluxDB. Similarly I found Grafana 4.20 - same place as before but called https://bintray.com/fg2it/deb/download_file?file_path=main%2Fg%2Fgrafana_4.2.0_armhf.deb

I just repeated the Grafana install routine to update it using the new simpler name grafana_4.2.0_armhf.deb - all worked perfectly.

I'm pretty happy with Grafana and I now know how to delete data points (by time) but if you really want to use SQLITE and do your own thing - take a look at the excellent work that Csongor Varga has done - https://www.youtube.com/watch?v=nkKf26oKzhQ

If you like this post – please share a link to it by social media, by email with friends or on your website.
More readers means more feedback means more answers for all of us. Thank you!

Facebooktwittergoogle_pluspinterestlinkedin

NanoPi Neo 2

FriendlyArm, Nanopi Neo 2

NEO 2You may well have read my review of the NanoPi NEO back in August 2016 – a nice little H3 unit available in two versions, 256Meg and 512Meg RAM. Well, the new unit has an H5 64-bit processor and comes with 512MB RAM and although there is only one USB connector – there are three USBs available. Size is 40mm square! The NEO 2, however is more expensive than the original at $15 + post etc.

There are lots of add-ons available for it including WIFI, a TTL-RS232 module (just an FTDI but cheap enough), power dock, prototype board etc etc – you can read about it here so I won’t go into too much detail. Personally I wonder about the logic of bringing out add-ons for such a small board – would you not be better off buying more complete board in the first place? Where this board scores is size – it really is small.

So – nice looking - I’m not sure why they chose hardwired Ethernet over WIFI – I’d have gone the other way and kept the height down – but there you are. The existing case for the NEO does not fit the new model incidentally.

The board DOES have fast Ethernet which is a plus – but then if you were going to use it as a NAS the 512Meg RAM might be a limitation?

NEO 2

So you can log into the board via the FTDI or by normal micro-USB serial. At this point I headed off to the WIKI to get the operating system image..

One thing that worried me – and I’ve complained about this before – the requirements are:  A NEO2 (obviously), a micro TFT card, a power supply and “a computer running Ubuntu 14.04 64 bit” -  FRIENDLYARM  - YOU MUST BE JOKING! Most of us in the west have Microsoft Windows PCs. I do have one PC running Linux – it is running MINT and I’ve no intention of changing that just for one new board!!!

I was a little disappointed to find that the website has only Ubuntu Core available – unlike the M3 and M1 both of which have excellent DEBIAN images which work straight out of the box. Again – this fascination with Ubuntu – a bit disappointing to those of us who like Debian and have used it on previous boards – however – the current version is 16.04 which is up to date.

At the time of writing, the ARMBIAN site had only nightly releases available and they refer to WIFI – so not exactly customised to this board which by default does not have WIFI !!

So, despite desire for Debian, I downloaded the Ubuntu Core software from the FriendlyArm WIKI site. I blew the image file with Win32 Disk Imager as usual and plugged it into the NEO 2 board. From what I could read, this would expand on first use and might take some time.

Power on and both the green and blue lights flashed for a moment then the green went full on – and the blue continued to flash. A quick check with Advanced IP Scanner showed the board sitting at 192.168.0.193.

I opened winSCP as usual and I tried logging in as root with password fa.

Having ran the script and that having created a PI user – I went off to log in as user PI – loaded and ran the script again. While all of this was running the processor was running at 53c, so depending on use, the heatsink may not be needed after all.

I cannot tell you how many times I have impressed on various companies – you cannot just put out hardware without software support – and Ubuntu 16.04 (if you have to use Ubuntu) is pretty standard stuff.

And yes the board is fast. Total Time using the nightly built from Armbian to install the script was 46 minutes – as against several hours on the Raspberry Pi Zero WIFI !! Using the default Ubuntu from FriendlyArm, my script took about an hour.

I have however noted some issues with the utility program as I could not get the unit to set up for 3.5mm jack audio. I got an error message.  There were a couple of other minor niggles which I've reported and no doubt will be fixed.  I've also noted that for the first time, they've created a PI user but when I used that for the script, it kept asking me for the password which was marginally annoying so I ran:

sudo visudo

and changed the sudo line to read

%sudo   ALL=(ALL) NOPASSWD: ALL

That sorted that out. Annoying but not that critical.

A brief check of my script and everything worked - so at least there are no negative changes and far less messing about than there was when I started this blog entry.  I tried a typical Chinese WIFI dongle known to work with the M3 and similar boards – although IFCONFIG was aware that something was plugged in, no WLAN messages came up.  GPIO apparently is on the way.

I think a little more time is needed before we can say this is a winner.  I never understand why companies release boards without full GPIO support when the openly available competition such as the Raspberry Pi has support by the shedload.

Where I WAS disappointed was when testing the SERIAL ports – there are 4 of them – with serial port 0 being used for debug. Node-Red recognised all 4 of them and even said it was connected – but ONLY serial port 0 seems to be working. That is a real shame as there simply isn’t enough info out there to figure out how to get the others to work. Whether this is an issue with the operating system or the Node-Red serial port I don’t know but that’s a disappointment.

Update 9 May 2017

I've now heard back from FriendlyArm on the subject of GPIO and the issues I had - they have fixed the most trivial - the temperature display - here is the link - https://www.sendspace.com/file/eqh907

But they still have not addressed the almost total lack of information on GPIO, SPI, I2C and serial ports. I decided to go see, because this board uses the H5, if there was a WIRINGPI or WIRINGOP library for it. Well, what a disaster – there are several for the H3 but the only links I could find suggesting an H5 build FAILED until I came across this link - http://www.jianshu.com/p/0ee31099983e

I tried the instructions:

git clone https://github.com/kazukioishi/WiringOP.git -b h5
cd WiringOP
chmod +x ./build
sudo ./build

and sure enough it all compiled. I then went off to my favourite easy-to-use facility GPIO

By trial and error I discovered that setting pins 7 and 10 to outputs worked. That corresponded to FriendlyArm 198 and 67 respectively (goodness knows where they get these numbers from).

Of course – as the numbers were wrong – I could not see how things like serial and I2c could work – it would be up to FriendlyArm to amend the code for their board and I have pointed this out to them - it seems sensible to me that if the market leader i.e. Raspberry Pi -as open source WiringPi with the GPIO utility, then it would make sense for those bringing out products in a similar market space to modify this code for their own use - indeed I believe that Orange Pi do have a version called WIRINGOP to do this and that too is open source. Let’s hope FriendlyArm take notice as their products are good but when you can't do something as easy as GPIO without having to resort to fumbling around with sys/class etc that's not very good – but at least that’s an easy way to get to normal IO. The likes of PWM would not work either and I found the i2cdetect command missing. FriendlyArm pointed out that this link takes you to the I2c pins but that still doesn't help as the likes of GPIO is expecting other pins.

Friendlyarm have recently written to say that they WILL bring out a version of WIRINGPI to go with this board – the sooner the better – if you want this – be sure to write to them to let them know others are interested.  The GPIO utility is really easy to use.

Facebooktwittergoogle_pluspinterestlinkedin

Amazon Polly Speech

General

Regular readers will know that I am an Ivona fan. In Node-Red I use the free Ivona service to provide high quality speech for my Raspberry Pi in Node-Red at the heart of my home control setup. Well, Ivona is now defunct. Amazon Polly is a replacement.

I’ll clarify that, Ivona is SOON to become defunct and you can’t create new accounts. The Amazon Polly system, is for most purposes a replacement for Ivona.

So – if you go to the Ivona site – you will see the reference to Amazon on the front page. The short, sharp answer is: Polly works, it is effectively free and it is as good as or better than Ivona. Read on.

So the Amazon system “Polly” works via an account. I have an Amazon Developer account and when I tried to add Polly – it said I didn’t have the right permissions – so – I added user Pete to my account and made him part of the Polly group – and that didn’t work either – then I noted something about payment and realised I’d not put any payment details in – I did that – and all of a sudden the thing came to life and I got the only things I needed – my user ID and secret password.

DON’T PANIC about payment – there is a free tier of up to (wait for it) 5 million characters per month for the first 12 months then $4 per million characters – by which point you probably won’t need any – read on) – for my purposes there is not a hope in hell I’ll ever reach the free limit. In addition  - the way I use it is the way they seem to want you to  – download a phrase as a file (MP3) and save it with a meaningful file name.  Next time you want that phrase – check to see if the file already exists – if so, play it, if not, get a new file from Amazon. In a typical use case that I might have, once the messages are used once there is very little chance of me needing to download anything and hence NO chance of incurring charges at least in the first year.

There are no doubt more elegant ways to do this than calling a command line from Node-Red and sometime someone will write a node to do it – might even be me – but right now this works perfectly and as far as I know it is the only published solution for Node-Red and Polly. If I’m wrong please do tell.

I’m assuming you have your credentials – don’t worry about location – they don’t have a location for England but all it means is to tell the code which server to use. Ireland works for me and it is working from here in Spain.

You need to grab the command line code – I used this on a Pi2.

sudo pip install awscli

Once that was in I used:

aws configure

to set up the user ID, secret key and location which I’d already set up on the Amazon site.

That done I tried this:

aws polly synthesize-speech --output-format mp3 --voice-id Amy --text "Hello my name is peter." peter.mp3

The resulting file was sitting in the /home/pi directory – this used the voice Amy (British female) to store a phrase into peter.mp3.  Good for testing but as you’ll see the final solution is much better.

The rest is about queuing messages, storing them with meaningful names, playing them back and making sure you don’t re-record a phrase you have already recorded. If you don’t like Amy – use another voice. If you want different voices for different phrases then you could incorporate the name into the filename (I’ll leave that to the reader).  If you want to add sound effects – just put .MP3 files in the relevant folder with your sound effects and call them by name.

Polly Speech

Looking at the above diagram, a test inject passes  what you want Polly to say in the payload.

The first function looks to see if the payload has something in it and if so it pushes that onto a stack. The code then looks to see if speech is busy – if not and if there is something on the stack, it checks – if it is an mp3 file it sends the file to the MP3 player. If it is not an mp3, it looks to see if you’ve already created an mp3 for that speech, if so it plays that file, otherwise it passes the message onto Amazon to create the file – which is then played back.

It would have been nice to process new speech while playing something else back but that would get more complicated, involving more flags. As it stands this is easy to understand. You can fire in more speech or .MP3 files while one is playing and they will simply be queued.

You clearly need your Amazon account setup and Node-Red for this – you also need MPG123 player. Both Node-Red and MPG123 are in my standard script.

Here is the code I used in each of those functions…. the MPG123 exec node simply has mpg123 for the command and the append payload ticked. The AWS exec node has  aws for the command and the append payload ticked.

Here is the code for the three yellow function nodes:

if (typeof context.arr == "undefined" || !(context.arr instanceof Array)) context.arr = [];
if (typeof global.get("speech_busy") == "undefined") global.set("speech_busy", 0);

if (msg.payload !== "") context.arr.push(msg.payload);
if ((global.get("speech_busy") === 0) && (context.arr.length)) {
    msg.payload = context.arr.shift();
    global.set("speech_busy", 1);
    if (msg.payload.indexOf(".mp3") == -1) {
        var fs = global.get("fs");
        var mess = msg.payload;
        var messfile = mess.toLowerCase();
        messfile = messfile.replace(/[.,\/#!$%\^&\*;:{}=\-_`~()]/g, "");
        messfile = messfile.replace(/ /g, "_");
        
        if (fs.existsSync("/usr/audio/" + messfile + ".mp3")) {
            msg.payload = "/usr/audio/" + messfile + ".mp3";
            return [null, msg];
        }
        else {
            var voice = "Amy";
            msg.payload = 'polly synthesize-speech --output-format mp3 --voice-id ' + voice + ' --text "' + mess + '" /usr/audio/' + messfile + '.mp3';
            global.set("speech", messfile);
            return [msg, null];
        }
                    
    }
    return [null, msg]; // mp3 or synth        
}


// now play function node
msg.payload="/usr/audio/" + global.get("speech") + ".mp3";
return msg;

// clr busy function node
context.global.speech_busy=0;
msg.payload=""; return msg;

That was version 1. But ultimately I wanted new speech to be processed by Amazon WHILE a previously recorded item was playing (assuming a previously recorded item was playing and the next item had to be created).

Several hours later I came up with this – it appears to work!

Polly 2

and here is the latest code – probably not QUITE as straightforward to read – but when you run it – indicators on the EXEC functions (brown) show clearly that the software is able to play a recorded message while fetching a new one. Could do with some extreme testing.

 

if (typeof context.arr == "undefined" || !(context.arr instanceof Array)) context.arr = [];
if (typeof global.get("speech_busy") == "undefined") global.set("speech_busy", 0);
if (typeof global.get("create_speech_busy") == "undefined") global.set("create_speech_busy", 0);

if (msg.payload !== "") context.arr.push(msg.payload);
if (context.arr.length) {
    msg.payload = context.arr.shift();
    if (msg.payload.indexOf(".mp3") == -1) {
        var fs = global.get("fs");
        var mess = msg.payload;
        var messfile = mess.toLowerCase();
        messfile = messfile.replace(/[.,\/#!$%\^&\*;:{}=\-_`~()]/g, "");
        messfile = messfile.replace(/ /g, "_");
        messfile = "/usr/audio/" + messfile + ".mp3";

        if (fs.existsSync(messfile)) {
            if (global.get("speech_busy")==1) { context.arr.unshift(msg.payload);  return [null, null]; }
            else { global.set("speech_busy", 1); msg.payload = messfile; return [null, msg]; }
        }
        else {
            if (global.get("create_speech_busy")) { context.arr.unshift(msg.payload);  return [null, null]; } else
            {
            context.arr.unshift(msg.payload);
            global.set("create_speech_busy",1);
            var voice = "Amy";
            msg.payload = 'polly synthesize-speech --output-format mp3 --voice-id ' + voice + ' --text "' + mess + '" ' + messfile;
            return [msg, null];
            }
        }
    }
    if (global.get("speech_busy")==1) context.arr.unshift(msg.payload); 
    else {  global.set("speech_busy", 1); return [null, msg]; } // mp3 or synth        
}

and here, the two small function nodes

// clear creating
global.set("create_speech_busy",0);
msg.payload="";
return msg;

// clear playing
global.set("speech_busy",0);
msg.payload=""; return msg;
Facebooktwittergoogle_pluspinterestlinkedin

RTK.GPIO Board

RTK.GPIO

RTK GPIO BoardI was recently given the opportunity thanks to a good friend, to check out the RTK.GPIO board, a small PCB with microUSB socket which promises to emulate the IO of a Raspberry Pi.

The board, sure enough has a Raspberry Pi connector looking remarkably similar to the Pi with 3v3 and 5v outputs.  The Kickstarter project advertises “an easy way to be able to add a Raspberry Pi GPIO header to your desktop or laptop.

The project managed to gain pledges of over £10k  which is pretty amazing for a simple board with little more than a processor and interface chip.

Claiming correctly that the Raspberry Pi add-on market is huge, the project states that there is an add-on board or HAT for “almost every electronics project for the Raspberry Pi”.  That kind of sets expectations for the board, sadly, a little too high.

Headlines like that set the pulse racing – could we test almost everything Pi I/O related on a PC? Well, no. Read on and you see that the IO is “basic” -  and you can do basic I/O – i.e. flashing a few lights. Well, you can do that on the BBC Micro-Bit – a horrible little “educational” board which can also do very little (why they didn’t go for the ESP8266 and Arduino-compatible software is beyond me given the massive user bases and hence support).

The Kickstarter page compares this simple IO board to Arduino, Raspberry Pi Zero and BBC Micro-Bit  with large user-bases. An Arduino Chinese copy is supposed to be £8-10 (wrong, Nano boards are available at under £2 inc postage and immediately programmable – possibly a tad more with the connectors pre-soldered). I won’t go into the others – you get the point.

Near the bottom of the Kickstarter page is a link to the software.  At the bottom of THAT page is a link to the LATEST software – except that it isn’t – it’s a 404 page actually. The actual software page, despite being quite old now (2014) has some recent updates but is not entirely error-free.

There is a reference to stretch goals including I2c but it is left a little vague as to what happened there (given the age of the board you’d think this would be updated).

The software is in two halves, code for the actual board and Python for the PC. I was hoping for some I2c examples but as far as I could tell most of the examples were variations on the same thing – flashing a few lights on and off. One example goes a little further and programs up an LCD display (again simple I/O) which might be marginally entertaining but again is merely using simple I/O

Had this board featured i2c and other serial protocols, opening up possibilities to control a wide range of devices, then I could see it being worth having on the bench for beginners… but to flash a few lights, for me, a cheap Chinese Arduino beats this hands down with a decent IDE that is easy for beginners and opens up a whole world of possibilities long after your first flashing light– whether you are using Linux (including Raspberry Pi) or a PC. For £1.15 or slightly more for the soldered version and the free Arduino IDE you get WAY more for beginners and advanced programmers alike.

Some time ago I was after extending an ESP8266 and looked at various ways to do this – one of which was simply to turn an Arduino into an I2c slave to control  ports – input and output along with analog input and PWM out. I wasn’t at the time thinking of an educational tool but the code therein was simple and would take very little time to convert the code to running off the serial port of virtually any PC, making it even easier to use.

If anyone wants to run with that and turn it into an “educational tool” – by all means.

If it is Raspberry Pi you’re after, well the Pi Zero WIFI is around the same price as the RTK.GPIO, it easy to use with the vast number of example programs out there and with WIFI built in is way more powerful.

Facebooktwittergoogle_pluspinterestlinkedin

Summer Migration

Travel

As Maureen and I start our summer migration to southern Spain, don’t expect anything new in here until around 17th – but of course – people are commenting constantly on various blog entries so I do hope you’re registered so you won’t miss anything.

I’m quiet excited, when we get to Spain, as well as the usual house-fixing chores, I have a new solar panel to fit (it is currently in the back of the car with a very expensive piece of plywood protecting it from several months-worth of “stuff” we’re talking over there. The plywood is likely to be re-purposed as a charging bench when I get there).

I’m looking forward to further delving into Espruino and the possibilities therein, I’ve more stuff to review and no doubt Antonio and I will continue to bring new developments to “the script” – which for the benefit of new readers is a comprehensive set of tools sitting on top of Debian/Raspbian/Ubuntu for working with IOT – most of which springs up from my original home control project.

I plan to do some more work with 433Mhz peripherals and RFLink and of course, sunbathing. Anyway, all of that to come – thanks for your recent contributions, I and no doubt others have learned a lot in the past few weeks, especially about making our SBC projects a bit more reliable.

For now, I’m about to go off on my travels and as you are no doubt aware, updating WordPress in the car is just – well, a non-starter really so if you’re expecting answers from me – a little patience please. Our trip will take 4 days in total, armed with cats and most of the gadgets I will need in the coming months.

Facebooktwittergoogle_pluspinterestlinkedin

ESP12S

esp12S, ESP8266

ESP12SI’m looking forward to trying out the new ESP12-S – this one apparently has a better antenna – and by the look of the schematic (see page 15 – thanks Alexander) the resistors on reset, GPI))0,2 and 15 are already there.

That should simplify layout somewhat!

ai-thinker.com - www.ai-thinker.com/uploadfile/2016/0909/20160909073755898.pdf

Hmm, board layout with 0.1 pins, 3v3 reg with decent heatsink area underneath, maybe 3 mosfets…

So this should work with existing layouts. I’ve kept back an IoTBEAR adaptor and ordered a couple of these from Ebay sub- £2 each.

Facebooktwittergoogle_pluspinterestlinkedin

Espruino For You Sir

General

As a once-dedicated C programmer (I learned with the K&R book) I originally dismissed Javascript as an interpreted toy (great for websites of course), that is until I started playing with Node-Red.

At that point, after much struggle, I started to get reasonably good with it and once I got used to the string handling, C by comparison started to feel annoyingly awkward when dealing with strings and objects, despite it’s vastly superior speed.

Some time ago I looked at Espruino in a bored moment – saw that it was very much under-developed – didn’t even handle WIFI at the time – a bit pointless on an ESP8266 -  and dismissed it.

Fast forward to this week.  I had set in place some copying of files from here to Spain which, according to FTP was going to take an hour. I could not tell you why but suddenly I got the urge to go see what was happening with Espruino.

So – I believe the model here is that they developed JavaScript for a range of devices – including the ESP8266 and their funding comes from selling the boards and from contributions. I can’t help with boards right now as I have some of my own but I could not resist grabbing the code and sticking it into one of my ESP12 boards to see how they’re coming along. 

Regular readers will know that I like to revisit projects, most of the time great ideas unless they really take off, end up in the bin but just sometimes they go on to be something better.

So off I went to the site. As I’m using their software for free – here’s a push for them – take a look here  – maybe it is worth looking at one of their boards?  Anyway, I grabbed their general ESP code – and using the horrible ESPTOOL but the much prettier ESP8266FLASHER.EXE  I blew the modules into my 4MB ESP-12-E module.  I grabbed the code and followed instructions on this page.

Programming Espruino

Node the bulk of the code at 0x1000 – don’t make the mistake I made of making that 0x10000 – it won’t work and will cause hours of fun. All 4 files are in their download along with instructions.

I grabbed their web IDE (Espruino IDE) to test the code at 115K baud…. my own serial terminal does as well but theirs is prettier. For the life of my I could not get it to start up properly in their IDE – but in my terminal, on hitting reset on the ESP8266 I got this.

_____                 _

|   __|___ ___ ___ _ _|_|___ ___

|   __|_ -| . |  _| | | |   | . |

|_____|___|  _|_| |___|_|_|_|___|

          |_| http://espruino.com

1v91 Copyright 2016 G.Williams

Espruino is Open Source. Our work is supported

only by sales of official boards and donations:

http://espruino.com/Donate

Flash map 4MB:512/512, manuf 0xe0 chip 0x4016

>

Now I have to say two things – for a complete beginner I found the documentation totally confusing – making reference to LED1 which doesn’t exist and other things – but it turns out you have to read again more carefully. Secondly when I went onto their forums totally confused – two guys came in immediately and helped me.

Once I had the board responding which ultimately did not take a lot of effort – I went back to their IDE and the board was then responsive – I still don’t know why this is – but just so you know if it happens to you – you can talk to the thing with a serial terminal.

Here’s the first issue I encountered…

 

Espruino Web IDE

Every time I hit enter – the interpreter would come back with undefined. I could not fathom this out until someone pointed out that the interpreter always returns either a value – or if no value is defined, undefined…. kind of makes sense though not what you’d expect from, say, a BASIC interpreter.  Once I’d gotten to grips with this I got more ambitious.

1+2

The answer to that was “=3” – seems fair enough.

If I reset the board however I could get nothing out of their IDE – I went back to my simple serial terminal – no problem.

My next problem was following examples – one example referred to flashing a light called LED1 – turns out this doesn’t exist in their ESP8266 implementation but the ports can be expressed as numbers where 0 is GPIO0.

var  on = false;
setInterval(function() {
  on = !on;
  digitalWrite(0,on);
}, 200);

I entered the above – perfectly valid JavaScript – and blow me – it WORKED!

Of course – on resetting this would be lost unless you typed “save()” or “reset()” to clear everything out. Running that timer command with different values WITHOUT hitting reset would result in multiple timers.

So – reset – then run that code then save… and the flashing light survives power off!

Espruino does not sadly handle hardware PWM – a shame as Espressif HAVE implemented this at quite reasonable resolution (14 bits) but then their implementation is not perfect – turn it off and that timer keeps on ticking anyway – making a mess of other high speed events like SPI. So maybe that’s why they implemented software PWM – which, well, works!

Some things you should know – they’ve not implemented use of GPIO16 and space is limited at least at first glance – 12K for program storage (including OTA they only seem to use 1 Meg so that does leave nearly 3MB available on an ESP12)… you’ll find the limitations here – but to be fair many of those are ESP8266 limitations – no DAC for example – well of course there isn’t – the ESP8266 doesn’t HAVE a DAC. So the list of limitations is not as bad as it seems by any means. I’ve no idea why they did not implement GPIO16 as a simple output – I use it all the time.

What they DO have are libraries for a good number of peripherals so you might well want to take a look.

Speed? Well, they use the ESP8266 at 160Mhz by default (and that’s fine – I don’t know why we normally stick with 80Mhz given that the extra speed doesn’t take much more power) and claim that for general I/O the unit should be fast enough for general purpose use.

I used example code to connect to my WIFI – no problem – but when I tried to MQTT – I came unstuck  -their source library for MQTT simply would not load successfully.

I read that you had to put the REQUIRE into the right hand side of their IDE to load the module -  I did that and it worked but I really would like to understand how to load the module WITHOUT their IDE – as mentioned above I’m not too happy about it’s behaviour from power up.

So – armed with the MQTT library it was as simple as…

mqtt=require("MQTT").connect({
  host: "192.168.0.20",
  username: "admin",
  password: "mypass"
});

mqtt.subscribe("test/espruino");

mqtt.on('publish', function (pub) {
    console.log("topic: "+pub.topic);
    console.log("message: "+pub.message);
  });

I fired many messages at it from an MQTT publisher and sure enough  - they came in.  If one can get both WIFI and MQTT running on this interpreter and if it stays up reliably then maybe here’s yet another way to write code for the ESP8266.

To test, I disconnected the WIFI access point – and reconnected – a message came up to say MQTT was disconnected.. but on re-applying power to the WIFI access point… nothing.   When I ran the MQTT code again it connected (indicating that the WIFI connection automatically reconnected itself). What is needed now is a check for broken connection and indefinite MQTT reconnect and resubscribe attempts until it is back in running order.  If anyone has done this already – please do let us know.

http://forum.espruino.com/conversations/303129/

Lots of modules…

http://www.espruino.com/modules/

I can see this 12k storage for code being a REAL bottleneck – I do hope that gets sorted…

Facebooktwittergoogle_pluspinterestlinkedin

File Share Quickies

hfs, ngrok

Following on from an initial comment by Keith Howell in the blog about NGROK, Antonio and I were chatting today about TKaiser’s little testing script and I wanted to share something with him. Skype of course is useless at this unless it’s a very small file - but we’ve used a tool called HFS in the past which opens up a simple file-sharing web-based system on your PC, meant for short term use of course.

Well, you can then of course give someone else access but that’s a pain as you have to go doing port redirects on your router – a little OTT for a simple one-off file sharing.

HFS

Combining two little utilities makes short work of this.

HFS lets you collaborate on files… while NGROK opens up the server port to the outside world without going anywhere near your router. Above you see the local version of HFS – and you can drag and drop files into that file area on the left… below you see the web version in this case on port 83 but any port will do as long as it isn’t in use.

HFS

If you look at the first image – top left where the house and the backslash is – you can add username and password for access by a simple right-click.

So – now you can go mess with your router and do a port redirect so someone can access this to grab files… OR open a command box in Windows and…

ngrok http 83

At which point the command line will show you the URL for someone to type in order to get the web page above! That’s it.  If you want to do stuff like secure access (HTTPS) and more you have to pay for NGROK but for this simple version – it’s all free.  Whenever you use NGROK you are given a unique access address so this is really meant for just short term use. Note again – use of port 83 here is arbitrary.

Both of these packages are installation-free – just stick them somewhere convenient like the desktop.

https://ngrok.com/download  (mac, Windows, Linux)

http://www.rejetto.com/hfs/?f=dl

Facebooktwittergoogle_pluspinterestlinkedin

Raspberry Pi 3 Serial

Raspberry Pi, Raspberry Pi 3

Just bought a new Raspberry Pi 3 with WIFI and Bluetooth?  Using it in a project with serial? Happy?

Well, you might not be so happy when you find out what I just found out.  I had my Node-Red project that has been controlling the house for around 18 months on a Pi2 and I was starting to have issues -  I thought it time to upgrade. So I made a new installation, faithfully copied everything over, plugged in the board… everything was fine.

Except it wasn’t – nothing was coming out of the serial port. Well, it turns out they stole the serial port for the Bluetooth. There is a second port – of a kind – which can be set to work with the GPIO pins 14 and 15…  but Node-Red serial node was NOT having it – I put a scope on the output – nothing.  I followed instructions to use an alternative “serial port” – which appeared and – the Node-Red Serial port input would not recognise it.

Anyway, it isn’t all bad – thanks to Dave at IBM who put me onto this link…

http://raspberrypi.stackexchange.com/questions/45570/how-do-i-make-serial-work-on-the-raspberry-pi3

Ignore most of the content and head to the end – Answer number FIVE.  This got me back up and running with a high speed serial port – but – no Bluetooth of course. I’m assuming I can just stick in a Bluetooth dongle – time will tell.

Essentially assuming an up to date Pi3, the key line is the one where you add 

dtoverlay=pi3-miniuart-bt”"

to the /boot/config.txt file.

I’m liking FriendlyArm more and more (I’d like them even MORE if they’d put more RAM in their boards) but at least, for today, I’m up and running.

Facebooktwittergoogle_pluspinterestlinkedin