June 21, 2021 Update
Regular readers may be aware of The Script, created and developed over many months to simplify the often torturous job of putting together a bunch of utilities onto a Raspberry Pi including Node-Red, Mosquitto MQTT broker and graphing tools such as Grafana along with text editors, a basic web page, lots of utilities, editors etc. all as described herein.
I’ve not updated “The Script” for several months but I just realised I’d not even mentioned in here that it installs “Cockpit” as I’d forgotten all about that useful utility. Now it has come to my attention, I’ve just ran Cockpit in it’s web-page – default port being :9090 so in my case, from my PC I can access Cockpit in my RPi4 from my PC web browser as 192.168.x.x:9090
Cockpit is a general utility for checking all aspects of operation off, again in my case, my RPi4. And guess what, among the reporting it tells me what processes are running – one of them is DOCKER – I’m not using Docker any more and had it not been for spotting it under “processes” in Cockpit, I’d never have known I’m wasting resources. That’s useful!
May 14 2020 Update
The Script is being constantly updated and improved – last updates this morning after discovering a flaw in yesterday’s updates – and with more updates on the way,
Here’s the direct link to self-documented The Script itself. As user pi in your home folder, do this:
The in the same default pi folder, run this…
You will need to enter some passwords and select some options then you can go have a cup of tea. Reboot when done and before attempting to use.
For the benefit of newbies, here is an exact example of usage:
Starting off with a fresh Raspberry RPi 3 or 4, grab the latest Raspbian LITE (or the full version if you really need it – but NOT the NOOBS version, please) and go to the Raspberry Pi site to grab the operating system.
On, for example a Windows PC, run something like BalenaEtcher to flash Raspbian Buster onto a microSD – I suggest 16GB or 32GB unless you need more room for large amounts of data or media – I’ve never gone past maybe 20GB – I generally use Sandisk or Samsung for known reliability.
Once done and verified, while the uSD is still in the PC, copy an empty file called “ssh” without the quotes onto the same microSD (which I’ll refer to from here merely as SD). Then plug the SD into the RPi, connect the RPi to your RJ45 network, plug in a 5v power supply able to handle 3A and the Pi should power up.
Using something like (free) Advanced IP Scanner on your PC, find the RPi and make a note of it’s IP address. Using something like (free) MobaxTerm on the PC, open an SSH session to that address and log into the Pi using user pi and the default password.
Before you do anything else do this on the RPi:
sudo apt update sudo apt upgrade
At this point I personally install rpi-clone (the instructions are on the github rpi-clone site and also elsewhere in my blog) and then, in the pi folder (default) after installing another blank, good quality SD into the top left USB socket on the RPi via an adaptor, I would then use:
sudo rpi-clone -f -U sda
Maybe 15 minutes later the job will be done and you have a backup of everything – subsequent backups should not need the -f option (which partitions and formats the SD) and will run more quickly, backing up only that which has changed since the last backup.
THEN grab “The Script” as below and run it – again as below.
Note that “The Script” may appear to stall from time to time as some operations occur which have pointless warnings deliberately turned off. It is not possible to turn off ALL warnings due to third party scripts running within “The Script” itself. The running time of “The Script” is dependent on a number of factors, several minutes on a Raspberry Pi 4 and much longer on older hardware.
Finally after rebooting with a working script in place, I personally would take ANOTHER good SD and clone – could save a lot of work.
End of example
Aidan Ruff and I originally developed a script to install Node-Red and several other packages onto the RPi for our own home control purposes. This has since mutated into the very popular “The Script” with help from several people and in particular my friend Antonio “Mr Shark”. “The Script” of course makes use of other scripts put together by countless other developers to install various programs on the Pi and other SBCs.
What is this about?
- Node-Red at port 1880 and lots of useful NODES
- Mosquitto (MQTT) with Websockets
- Webmin at port 10000 (off by default)
- NGINX at port 80
- HA-Bridge at port 82 (suitable for Amazon Alexa)
- SQLite and PHPLiteAdmin
- MC File manager and editor (mcedit)
- Grafana with InfluxDB (OFF unless you select in the menu)
- Cockpit (visible in a web browser at port 9090)
- Lots more – see the menu screen
Typically this setup would be used as the basis of a nerve-centre to control and monitor home control gadgets such as the ESP8266 units we discuss in this blog, Sonoff controllers and much more. This software can make for a good start in controlling the home – and more importantly can be done YOUR WAY.
Today the ONLY “noisy” script within is the SQLITE section which throws up all sorts of warnings – but which works fine, non-the-less. The rest work quietly without issue. The Node-Red installation now relies on the FIRST option in the Node-Red list and relies on the install-upgrade script by the IBM guys. Enjoy.
As well as it’s original purpose of effortlessly setting up a Raspberry Pi, “The Script” also runs well with several other boards with RPi 3 and 4 being the most recently tested. See the right hand side of the image below to see what the script does, given a basic operating system install. We currently suggest NOT using this with DIET PI, original Pi or the Raspberry Pi Zero as we are no longer testing either and the latter pair are just TOO SLOW. Note that I refer to Stretch and Buster – in June 2021 I am using the latest RPi operating system.
For simplicity, no local Blynk server is installed in the script – see this blog entry, note that I use all lowercase for the Blynk directory name though personally I’ve stopped using Blynk in favour of Node-Red Dashboard.
On none-Raspberry Pi devices, Apache is the default webserver. On the Raspberry Pi, NGINX as now the default webserver as it is compatible and far more modern – you can of course change these options when running the script. Also, whichever web server you use – if you plan to use node-red-contrib-amazon-echo – you might want to shift the working port for Apache (or NGINX) from 80 to 86. Do this in /etc/apache2/sites-enabled/000-default.conf (one reference to change) or if you are using NGINX, change /etc/nginx/sites-enabled/default – two references to 80 change to 86 (my choice). The Amazon echo hub in NR I then set to 8980 and rebooted. The next para is taken from “A Good Day for Alexa” blog entry:
For some time I used port 8980 to avoid being the hated Linux ROOT user and, on my Raspberry Pi 3 and 4 boards, I moved the web server to port 86 and using “iptables” have redirected port 8980 traffic to port 80 to keep Amazon happy while continuing to use port 8980 to keep Linux happy as PI user. All of this of course only matters if you want to use that particular node in Node-Red – with generation 3 or later DOT and similar Alexa devices.
sudo apt-get install iptables-persistent sudo iptables -I INPUT 1 -p tcp --dport 80 -j ACCEPT sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8980 sudo netfilter-persistent save sudo netfilter-persistent reload
March 2 2019
Added a useful file/directory operations node to the installed Node-Red nodes along with the strangely missing node-red-contrib-influxdb node. Note that in order to use this you must have opted to install Grafana and Influxdb. Databases in Influxdb are added at the command line (I do this as user pi) simply by invoking the command “influx” and once in, using the commands “CREATE DATABASE mydatabase” (as an example) followed by “exit”. All without quotes and using your choice of database name.
Uploaded a new version with mods for Raspbian and in particular Node-Red and nodejs v10.x as currently used by the Node-Red guys on the Raspberry Pi.
Raspberry Pi users should log in a user PI and start with a clean Raspbian. Enable any hardware you want working before installing the script – like I2c.
Raspberry Pi users get the script in your /home/pi directory as per the link at the top of this blog entry – if you are familiar with the tools available (see the script which is of course a readable text file) that should be that – read no further as the rest of this article is old and you will see a file list of what gets installed, in the script itself. Also when the script has run, you should find a simple home-page available in your Raspberry Pi browser which links to the main tools available.
Non-Raspberry Pi users start in your ROOT directory as user ROOT, initially.
Originally designed to run under Debian – reader Antonio and I have also made the script compatible with Xenial in some cases. It now handles Raspbian and the Armbian versions of Debian and Xenial (Ubuntu). Today (July 2019) it is mainly being tested with Raspbian Buster (the current official Raspberry Pi operating system).
One issue I have occasionally with new boards is lack of remote root access to install the script. For example on a Banana Pi 3 I had this.. there was PI access for the likes of WinSCP but not root access. This was easily solved as user Pi – after using “sudo passwd root” to ensure I had a known root password.
sudo sed -i 's/prohibit-password/yes/' /etc/ssh/sshd_config sudo systemctl restart sshd
That won’t work in all cases but I’m parking that here for reference. The first line is included in The Script and you can probably ignore the above.
The easiest way to get the pre-2019 script – as ROOT user, in your ROOT directory is:
wget –no-check-certificate https://bitbucket.org/api/2.0/snippets/scargill/ekon6o/master/files/script.sh
This can now be run easily: bash script.sh – run as user root, it will start up, copy itself into PI user area, creating that if it does not exist then will give instructions.
Before I begin… no, we’re not a support shop – you may or may not end up having to tweak things for your system but please don’t write in to say “I tried this on a Mac and it doesn’t work”.
The level of interaction after the beginning is now almost zero thanks to a new menu-driven approach and many improvements. We continue to add/modify features as appropriate.
The Virtual Machine (historical information)
Let’s take a look at what we did to get this running on a VM – you can of course duplicate all of this as you go along. We used the excellent VMware of VirtualBox on Windows PCs. This explanation covers VMware.
So – first things first – as I’ve done a video of the VirtualBox installation complete with sample use of Node-Red, here we’ll discuss the VMWARE version. Both of these were done as test mechanisms before using the script on real devices such as the Orange Pi Zero, Raspberry Pi, Odroid and other devices discussed at length in the blog.
We loaded up a copy of VMware (I used VirtualBox but the procedure is similar – VMware player is free of you can use the full commercial workstation) and made sure it works. That much is up to you. There is a wizard for creating new virtual machines – we chose a CUSTOM machine. Steps from that point on were:
NEXT – NEXT – “I will install the operating system later” – NEXT
Linux / Ubuntu 64-bit – NEXT – d:\test – NEXT
Number of processors – 1 – number of cores – 1 – NEXT
Memory for this machine – 1024MB – NEXT
Use Bridged Connection – NEXT
I/O Controller type LSI Logic – NEXT
Disk Type SCSI – NEXT
Create a new virtual Disk – NEXT
12GB (I used 16GB – 8GB would have done) – Store virtual disk as a single file – NEXT
At this point we gave the file a name – in our case d:\test\Ubuntu 640bit.vmdk – NEXT – FINISH
Under settings – hardware – CD/DVD we ticked “Use ISO image file” and selected the file we loaded from the Ubuntu website – “Ubuntu-16.04.1-server-amd64.iso” – OK.
At this point you’ll note reference to AMD64 – that had me as well but it works a treat on INTEL hardware.
Start off the VM – you get asked which language to use (ENGLISH) and various options will come up – the first being “Install Ubuntu Server” – that’s the one.
Once again you’ll get a language question – English. Location – in my case “United Kingdom” – if you don’t fit into the rules use “Other”.
Time Zone in my case – EUROPE and then country. Locale – in my case United Kingdom.
Hostname – we went with the default “Ubuntu” – but call it whatever you want.
At this point you get asked to create a user – it is important you make a user “pi”. Enter password also (though the new script will create user PI if it does not exist).
Partitioning – “use entire disk” (remember this is a virtual disk – not your real disk). You are then asked to confirm erasing disk – go with that. You then get asked to confirm AGAIN. YES.
HTTP Proxy just leave blank – CONTINUE.
Install security updates automatically.
Add OpenSSH server and CONTINUE (important for remote access)
Install GRUB bootloader – YES.
Installation is complete – CONTINUE. Let it run.
And voilà – one working copy of Ubuntu.
When run, the script will generate a Pi user if it does not exist – will make that user part of the correct groups and will ask you to restart the script as user Pi.
The BASH Script
This part is about running the script and the easiest way is to start in a terminal as user ROOT in the root directory (in non-Raspberry Pi installations).
Do the WGET shown above and you will end up with a script file called “script.sh” in the root directory.
At this point BACK EVERYTHING UP to avoid disappointment – no, REALLY.
The script will if necessary create the Pi user but either way will now ask you to log in as user pi and repeat the procedure in the /home/pi directory – i.e. bash script.sh.
The point of this is to avoid another way which would be to create a file, load in the script contents ensuring you’re using Linux LF only format – then run the script after setting up permissions – this way you don’t have to do any of that.
In the end you will see a menu which will vary slightly depending on which system you are using – you won’t be asked about GPIO on a Raspberry Pi for example.
For the VM we left everything as you see here and tabbed to OK.
At this point you will be asked for a USER and ADMIN names (which can be user and admin if you like) and corresponding passwords. From there on – everything should happen with no user interaction.
After maybe 15 minutes to an hour – or a LOT longer on a Raspberry Pi Zero WIFI – you’ll see an invite to REBOOT – so REBOOT. When the board/VM boots up you will find hopefully a working system.
Assuming you’ve installed everything:
- Node-Red runs at port 1880
- Webmin at port 10000 (Webmin needs Pi or ROOT username and pass until we figure out how to add in the ADMIN user).
- Apache runs at port 80
- HA-BRIDGE will run at port 82 – It seems that Amazon Echo is quite happy with this but the Google product wants port 80!
Node-Red should include MQTT which will need admin username and password as will Node-Red itself. USER name and password is used by Node-Red Dashboard.
There is also the excellent MC – a file manager and editor.
I hope this brief introduction sets you off in the right direction and of course no doubt there will be discussing in the notes below.
Update 05 Jan 2017
At the time of writing I’m having issues with PIXEL in VirtualBox and DEBIAN on VirtualBox in terms of Apache not loading.. I’m sure we’ll sort that – meantime some useful links. Debian may not come with SUDO depending on which installation – here’s a fix link that worked for me in my VirtualBox image.. If SSH setup won’t allow remote access using passwords, here’s a link that helped me. Pixel of course will run in VirtualBox but by default it runs of a virtual DVD which is about as much use as an ashtray on a motorbike as nothing gets saved. Once running however it is a breeze to move over to the Virtual hard disk – you’ll find that part way down the comments in this link. So right now I have VirtualBox running Xenial, Pixel and Debian and the only issues I have are with Pixel and Debian – Apache – watch this space. I’ve also improved the input for user and admin.
Important: Whatever you are using should have the SUDO command built in – it may not be obvious but Debian often does NOT have this installed by default – sort this FIRST. You should also make sure you have Pi user with SUDO group privilege – PI already has this of course and you may have added a user Pi – if not..
su -c “bash makepi”
Or, Here’s a useful link – takes seconds and is easy… basically – as root you can do – apt-get install –y sudo….. and you should make sure that your PI user is part of the sudo group.
The script works with both NanoPi NEO2 and the NanoPi M1+ and features temperature monitoring and other improvements.
If doing this as a non-Pi user – i.e. ROOT on a machine with no Pi user – follow instructions above OR a slightly easier way…
Having uploaded the script and given permissions… as ROOT
as before – but when it has completed and wants you to log in as PI – stay where you are and:
su – pi -c /home/pi/script.sh
That might save you a little hassle.