The Script

Regular reader will know about the script that Aidan Ruff and I originally developed to put Node-Red and several other packages onto the Raspberry Pi for our own home control purposes.

Home Control 2017

WELL - there is a new version of the script, modified to include logging and work to handle Raspbian Stretch (tested on Raspberry Pi  2 and 3). The script also runs well with several other boards running Debian, Ubuntu and more.  See right hand side of the above image for what it does. Read on…

Originally designed to run under Debian – reader Antonio and I have also made the script compatible with Xenial. So now Raspbian, DietPi (Debian) and the Armbian versions of Debian and Xenial (Ubuntu) as well as the DietPi VM.

One issue I have occasionally with new boards is lack of remote root access to install the script. For example on a Banana Pi 3 I had this.. there was pi access for the likes of WinSCP but not root access. This was easily solved as user Pi - after using "sudo passwd root" to ensure I had a know root password.

sudo sed -i 's/prohibit-password/yes/' /etc/ssh/sshd_config
sudo systemctl restart sshd

That won’t work in all cases but I’m parking that here for reference.

The easiest way to get the script – as ROOT user, in your ROOT directory is:

wget --no-check-certificate http://bit.do/the-script

Before I begin... no, we’re not a support shop – you may or may not end up having to tweak things for your system but please don’t write in to say “I tried this on a Mac and it doesn’t work”).

The level of interaction after the beginning is now zero thanks to a new menu-driven approach and many improvements including using a VM (virtual machine) to test the whole thing at accelerated speeds. We continue to add/modify features as appropriate.

What is this about?

What we have here is an explanation of how to get Xenial running in a VM – the purpose of this is merely as a base to test the new script mentioned below which will install somewhat more than the original script (designed originally for Raspberry Pi but now working on several Debian-based devices and now Xenial) – namely:

  • Node-Red at port 1880 and lots of useful NODES
  • Mosquitto (MQTT) with Websockets
  • NODEJS/NPM
  • Webmin at port 10000
  • Apache at port 80
  • HA-Bridge at port 82 (suitable for Amazon Alexa)
  • SQLite and PHPLiteAdmin
  • MC File manager and editor
  • Grafana with InfluxDB (only tested on a few setups – so OFF by default)
  • Lots more – see the menu screen

Typically this setup would be used as the basis of a nerve-centre to control and monitor home control gadgets such as the ESP8266 units we discuss in this blog, Sonoff controllers and much more. Armed with this software, controlling the home is easy – and more importantly can be done YOUR WAY.

The Virtual Machine

Let’s take a look at what we did to get this running on a VM – you can of course duplicate all of this as you go along. We used the excellent VMware of VirtualBox on Windows PCs. This explanation covers VMware.

So – first things first – as I’ve done a video of the Virtualbox installation complete with sample use of Node-Red, here we’ll discuss the VMWARE version.  Both of these were done as test mechanisms before using the script on real devices such as the Orange Pi Zero, Raspberry Pi, Odroid and other devices discussed at length in the blog.

We loaded up a copy of VMware (I used VirtualBox but the procedure is similar – VMware player is free of you can use the full commercial workstation) and made sure it works. That much is up to you. There is a wizard for creating new virtual machines -  we chose a CUSTOM machine. Steps from that point on were:

NEXT - NEXT – “I will install the operating system later” – NEXT
Linux / Ubuntu 64-bit – NEXT – d:\test  - NEXT
Number of processors – 1  - number of cores – 1 – NEXT
Memory for this machine – 1024MB – NEXT
Use Bridged Connection - NEXT
I/O Controller type LSI Logic - NEXT
Disk Type SCSI - NEXT
Create a new virtual Disk - NEXT
12GB (I used 16GB – 8GB would have done) – Store virtual disk as a single file - NEXT
At this point we gave the file a name – in our case d:\test\Ubuntu 640bit.vmdk -  NEXT – FINISH

Under settings – hardware – CD/DVD we ticked “Use ISO image file” and selected the file we loaded from the Ubuntu website – “Ubuntu-16.04.1-server-amd64.iso” – OK.

At this point you’ll note reference to AMD64 – that had me as well but it works a treat on INTEL hardware.

Start off the VM – you get asked which language to use (ENGLISH) and various options will come up – the first being “Install Ubuntu Server” – that’s the one.

Once again you’ll get a language question – English. Location – in my case “United Kingdom” – if you don’t fit into the rules use “Other”.

Time Zone in my case – EUROPE  and then country. Locale – in my case United Kingdom.

Hostname – we went with the default “Ubuntu” – but call it whatever you want.

At this point you get asked to create a user – it is important you make a user “pi”. Enter password also (though the new script will create user PI if it does not exist).

tmp6049Home encryption – we said no. When asked if the time zone was correct we said yes.

Partitioning – “use entire disk” (remember this is a virtual disk – not your real disk). You are then asked to confirm erasing disk – go with that. You then get asked to confirm AGAIN. YES.

HTTP Proxy just leave blank – CONTINUE.

Install security updates automatically.

Add OpenSSH server and CONTINUE (important for remote access)

Install GRUB bootloader – YES.

Installation is complete – CONTINUE. Let it run.

And voilà – one working copy of Ubuntu.

When run, the script will generate a Pi user if it does not exist - will make that user part of the correct groups and will ask you to restart the script as user Pi.

The BASH Script

This part is about running the script and the easiest way is to start in a terminal as user ROOT in the root directory.

do the WGET shown above and you will end up with a script file called “the-script”.

At this point BACK EVERYTHING UP to avoid disappointment – no, REALLY.

Execute as:

bash the-script

The script will if necessary create the Pi user but either way will now ask you to log in as user pi and repeat the procedure.

The point of this is to avoid another way which would be to create a file, load in the script contents ensuring you’re using Linux LF only format – then run the script after setting up permissions – this way you don’t have to do any of that.

In the end you will see a menu which will vary slightly depending on which system you are using – you won’t be asked about GPIO on a Raspberry Pi for example.

tmpE673

For the VM we left everything as you see here and tabbed to OK.

At this point you will be asked for a USER and ADMIN names (which can be user and admin if you like) and corresponding passwords. From there on – everything should happen with no user interaction.

After maybe 15 minutes – or an hour - ore a lot longer on a Raspberry Pi Zero WIFI   – you’ll see an invite to REBOOT – so REBOOT. When the board/VM boots up you will find hopefully a working system.

Assuming you've installed everything:

  • Node-Red runs at port 1880
  • Webmin at port 10000 (Webmin needs Pi or ROOT username and pass until we figure out how to add in the ADMIN user).
  • Apache runs at port 80
  • HA-BRIDGE will run at port 82 – It seems that Amazon Echo is quite happy with this but the Google product wants port 80!

Node-Red should include MQTT which will need admin username and password as will Node-Red itself. USER name and password is used by Node-Red Dashboard.

There is also the excellent MC – a file manager and editor.

MC and MC EDIT

I hope this brief introduction sets you off in the right direction and of course no doubt there will be discussing in the notes below.

Update 05 Jan 2017: At the time of writing I'm having issues with PIXEL in VirtualBox and DEBIAN on VirtualBox in terms of Apache not loading.. I'm sure we'll sort that - meantime some useful links.   Debian  may not some with SUDO depending on which installation - here's a fix link that worked for me in my VirtualBox image.. If SSH setup won't allow remote access using passwords, here's a link that helped me.  Pixel of course will run in VirtualBox but by default it runs of a virtual DVD which is about as much use as an ashtray on a motorbike as nothing gets saved. Once running however it is a breeze to move over to the Virtual hard disk - you'll find that part way down the comments in this link.  So right now I have VirtualBox running Xenial, Pixel and Debian and the only issues I have are with Pixel and Debian - Apache - watch this space.  I’ve also improved the input for user and admin.

Important: Whatever you are using should have the SUDO command built in – it may not be obvious but Debian often does NOT have this installed by default – sort this FIRST. You should also make sure you have Pi user with SUDO group privilege – PI already has this of course and you may have added a user Pi – if not..

wget https://bitbucket.org/api/2.0/snippets/scargill/E5rG7/master/files/makepi
su -c "bash makepi"

Or, Here’s a useful link – takes seconds and is easy…  basically – as root you can do - apt-get install –y sudo….. and you should make sure that your PI user is part of the sudo group.

Update March 31, 2017:  the new version of the script "the script" works with the brand new NanoPi NEO2 and the NanoPi M1+ and features temperature monitoring and other improvements.

If doing this as a non-Pi user – i.e. ROOT on a machine with no Pi user – follow instructions above OR a slightly easier way…

Having uploaded the script and given permissions… as ROOT

./script.sh

as before – but when it has completed and wants you to log in as PI – stay where you are and:

su - pi -c /home/pi/script.sh

That might save you a little hassle.

Update 19 Aug 2017: Script updated to handle Raspbian Stretch on Raspberry Pi – tested on RPi 3. Also added code to help with non-root access to GPIO using ONOFF library. See blog entry on GPIO.

Facebooktwittergoogle_pluspinterestlinkedin

304 thoughts on “The Script

  1. I've been doing lots with running Ubuntu(Kubuntu,Lubuntu,Xubuntu,etc) from liveCD ISO files on my HD and with a partition or USB thumb drive labeled casper-rw there's an overlay(persistent) filesystem capability too.

    So my thought is, instead of installing Ubuntu in a VM, why not download the Ubuntu ISO file and setup a VM with the CDROM pointing to the file. Then create a small 8GB hard disk image and boot the vm into the live mode. partition and change the label of the 8GB partition to casper-rw and reboot. Now you have a quickly booting Ubuntu system to experiment loading up your home automation stuff(docker, node red, etc).

    For the noobs it gets them through the installation process fast and lets them play in Ubuntu land for a bit. If they like it, they can then generate another VM but this time go for the installation of Ubuntu so it's more able to handle updates.

  2. Hi Pete / Antonio,
    I've used Pete's script several time in the older format and had few problems. I ran the new one this morning with the menu system and found this problem on my Raspberry pi 3 loaded with Raspbian:-

    The script does not detect my Raspbian OS correctly. It thinks that it's not a genuine Raspbian OS. Here's my OS:-

    pi@raspberrypi:~ $ cat /etc/os-release
    PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
    NAME="Raspbian GNU/Linux"
    VERSION_ID="8"
    VERSION="8 (jessie)"
    ID=raspbian
    ID_LIKE=debian

    I changed this line in your script:-
    if [[ $OPSYS == *"RASPBIAN"* ]];then
    to
    if [[ $OPSYS == *"raspbian"* ]];then
    and it detects my OS correctly.

    Many thanks to both of you for your great work.
    Keith

    1. Correct, Keith - I've now put a case converter in to make all results upper-case - so that should not happen again - please report back any other issues as this is all new. I've also just put a handler in for the CANCEL option which now works.

  3. Hi Pete,
    Here's another one:-

    Line 255
    MYMENU=$(whiptail --title "Main Raspberry Pi Selection Checklist" --radiolist \
    This only allows you to check ONE option.

    I think it should be :-
    MYMENU=$(whiptail --title "Main Raspberry Pi Selection Checklist" --checklist \

    Thanks for the 'CANCEL' handler.

    Keith

  4. Pete,

    There's a typo that prevents NPM from updating:-

    Line 268
    "upgradednpm" "Upgrade NPM to latest version " ON \
    I think that this should read:-
    "upgradenpm" "Upgrade NPM to latest version " ON \

    to match line 715
    if [[ $MYMENU == *"upgradenpm"* ]]; then

    1. Also, lines 721 and 722 should be swapped?

      720 # Add CU to enable serial VT100 mode for terminals
      721 printf "${ICyan}============================ Installing CU...etc
      722 if [[ $MYMENU == *"installcu"* ]]; then

  5. New update - should now check to ensure it is user PI that has logged in! If root and there is no user PI it should create one and ask you to log out and try again. !! Thanks to Mr Shark for this one.

  6. Hi Pete,

    Nice video. Always good to see things in action. Can I suggest maybe a video on messing with Node Red Dashboard, be nice to see its capabilities.

      1. Hi John - did you use the latest script as of today... I just installed on an Ubuntu real machine - and Debian and Ubuntu VMs without issue... but not on a Pi3. The issue with NPM was resolved ages ago - perhaps on a Pi the existing version is causing a problem?

        As for X terminal if the terminal is too small when you start the script it is no good - that is a "feature" of the menu script... just make the terminal a little larger.

          1. Hi Pete,

            I'll help with the testing on the Pi 3. I'm flashing a fresh Jessie image to a sd card now. Does it matter if I run apt-get upgrade before I run the script or should I just start with the default image?

          2. Pete, I'm happy to help test with Raspberry Pi if you want. I have a lot of experience with shell scripting too.

            "The issue with NPM was resolved ages ago"
            On a pi with the latest Jessie, NPM is not installed (or if it is, it's not on $PATH)

            1. I THINK that the Pi issue is now resolved and 2 changes made today to SSH and some helpful text for menus - please report only issues with software from this evening's date ONWARDS - and thanks to EVERYONE for their help - shell scripting experience always comes in handy - as you can see!

  7. I've been trying this script on a Raspberry Pi 3 with Jessie downloaded today.
    A couple of points:
    1 I see the whole menu if in command line mode, but on an X terminal a few characters are missing on the left hand side - so you can't select the options.
    2 I get a lot of errors eg Failed to fetch http://packages.dotdeb.org/dists/jessie/InRelease Unable to find expected entry 'all/binary-armhf/Packages' in Release file (Wrong sources.list entry or malformed file)
    3 As with the old script, it attempts to use NPM to install things and fails because NPM is not installed. "sudo: npm: command not found" Presumably NPM is already installed with other operating systems? Could it be added to the prerequisites?
    4 A logfile is useful to trawl through for error messages. You can get one with the command ./script.sh 2>&1 | tee script.log

    Regards,
    jb

    1. both right... just sent patches to Peter, i did tests on dietpi and ubuntu, all right, missing raspbian, if someone want to test, in latest script change this (from line 367):

      if [[ $OPSYS != *"UBUNTU"* ]]; then
      #enable php7 on debian jessie - no need for xenial
      echo "deb http://packages.dotdeb.org jessie all" |sudo tee /etc/apt/sources.list.d/dotdeb-php7.list
      echo "deb-src http://packages.dotdeb.org jessie all" |sudo tee -a /etc/apt/sources.list.d/dotdeb-php7.list
      wget https://www.dotdeb.org/dotdeb.gpg -O - | sudo apt-key add -
      sudo apt-get update
      fi

      with this:

      if [[ $OPSYS == *"DEBIAN"* ]]; then
      #enable php7 on debian jessie DIETPI
      echo "deb http://packages.dotdeb.org jessie all" |sudo tee /etc/apt/sources.list.d/dotdeb-php7.list
      echo "#deb-src http://packages.dotdeb.org jessie all" |sudo tee -a /etc/apt/sources.list.d/dotdeb-php7.list
      wget https://www.dotdeb.org/dotdeb.gpg -O - | sudo apt-key add -
      sudo apt-get update
      fi
      if [[ $OPSYS == *"RASPBIAN"* ]]; then
      #enable php7 on debian jessie RASPBIAN
      echo "deb http://repozytorium.mati75.eu/raspbian jessie-backports main contrib non-free" |sudo tee /etc/apt/sources.list.d/mati75-php7.list
      echo "#deb-src http://repozytorium.mati75.eu/raspbian jessie-backports main contrib non-free" |sudo tee -a /etc/apt/sources.list.d/mati75-php7.list
      sudo apt-key adv --keyserver pgpkeys.mit.edu --recv CCD91D6111A06851
      sudo apt-get update
      fi

      unfortunately, dotdeb repository has not arm packages, had to find them elsewhere...

      1. Right everyone - ... menu fixed - narrowed in for smaller screens and simplified.

        I've implemented the changes above suggested by Mr Shark. I have to go out - I will not be back until tomorrow mid-day - here's hoping these changes sort the Pi out.

        1. i think in the raspbian section, if wolfram and openoffice are checked to be removed, this should happen BEFORE initial general update/upgrade in prerequisites... no need to update something and then remove it after 😉

    2. 1 Peter reduced menu size

      2 my patch in comments here, and already included in latest online script, solved php7 on raspbian

      3 no npm problems, just installed latest raspbian (november 2016), launched script, everything is up and running...

      i made install in 2 batch, just to check everything working...
      1st run: modpass prerequisites mosquitto apache screen mpg123 phpsysinfo addindex installcu installmc wolfram office
      2nd run: modpass (to reinsert password for node-red and webmin) nodejs nodered webmin java upgradenpm
      not a single problem...

    3. pi@raspberrypi:~/.node-red $ npm -v
      4.0.5
      pi@raspberrypi:~/.node-red $ node -v
      v4.7.0
      pi@raspberrypi:~/.node-red $ npm view node-red version
      0.15.2
      pi@raspberrypi:~/.node-red $ npm list --depth=0
      /home/pi/.node-red
      ├── bcryptjs@2.4.0
      ├── node-red-contrib-bigtimer@1.4.86
      ├── node-red-contrib-blynk-websockets@0.0.3
      ├── node-red-contrib-boolean-logic@0.0.3
      ├── node-red-contrib-config@1.0.1
      ├── node-red-contrib-esplogin@1.1.9
      ├── node-red-contrib-file-function@1.1.2
      ├── node-red-contrib-gpio@0.9.1
      ├── node-red-contrib-graphs@0.3.5
      ├── node-red-contrib-grove@1.0.7
      ├── node-red-contrib-isonline@1.1.15
      ├── node-red-contrib-ivona@0.0.2
      ├── node-red-contrib-moment@2.0.0
      ├── node-red-contrib-npm@1.0.0
      ├── node-red-contrib-particle@0.0.4
      ├── node-red-contrib-timeout@1.0.4
      ├── node-red-contrib-web-worldmap@1.0.22
      ├── node-red-dashboard@2.2.1
      ├── node-red-node-arduino@0.0.12
      ├── node-red-node-darksky@0.1.14
      ├── node-red-node-emoncms@0.0.11
      ├── node-red-node-geofence@0.0.16
      ├── node-red-node-google@0.1.0
      ├── node-red-node-ledborg@0.0.17
      ├── node-red-node-openweathermap@0.1.14
      ├── node-red-node-ping@0.0.13
      ├── node-red-node-pushbullet@0.0.10
      ├── node-red-node-random@0.0.8
      ├── node-red-node-smooth@0.0.9
      ├── node-red-node-sqlite@0.1.2
      └── raspi-io@7.1.0

  8. Very interesting stuff! As I have an issue with the loss of mouse events in Chrome on Jessie I will try another installation on an 8Gb SD with the proposed Ubuntu image.
    I am a bit confused about all usernames and passwords needed. Should I use PI or pi as a user name? An overview of all needed un&pw's where to use what and when to create would be appreciated.
    Another item I have not touched is access from outside of home through internet, what to do on router(s) to reach my home server from outside?
    Keep on going the good work!

      1. Will give it a try with VMWare and that takes some time. The description is very extensive so I might be able to come all the way through. So no big hurry anyhow, but thx for the swift attention to this.

        1. Does not show the same options as described although the same Ubuntu image.
          There comes a question BIOS or EFI and default is BIOS, but I have no idea whether this is right. I can try EFI for a better result, but that is a shott in the dark. No question about adding SSH or GRUB.
          Display settings are not usable. No display is detected. It uses host settings and does not completely fit in window of VMware.
          How to change to ROOT user? I am a Windows user. This will take some googling.

          1. full album of the VM creation in Vmware: http://imgur.com/a/X8xLi
            you could use Vmware Player, too, as it's free and despite the name, allows to create VMs, too...
            in vm, no real difference between bios and efi, choose what you want...
            this setup uses UBUNTU 16.04.1 LTS SERVER, and so, no desktop environment... if you use a desktop iso, be sure to install vmware tools at the end, to enable graphic support (it's in the vmware player/workstation menus)...
            root is useless in ubuntu, it has no password, when you need to do something ubuntu asks for YOUR password, as first user created has SUDO rights, so can do administrative tasks without ever know root password or become root... but, many windows users used to be "administrator", tend to enable it... bad security habit...

            in a system with root enabled, an attacker only need its password to be harmful... in a system with a regular user, attacker has to know user name, user password, and THEN root password, if used has no sudo rights... but, your choice...

            1. Thx.
              I installed on W10-64 the desktop version. You are showing install on linux. Diff is menu with SSH and you did not cross Samba server, probably needed for W10 PC's on the network.
              Pete is advising some work in ROOT. Should I remove Root user?
              I will start from scratch again in EFI mode with one of the premade VM's you mention hereafter.

    1. the username and password that are asked by Peter's script are used for:
      user: only for node-red dashboard
      admin: mosquitto, phpliteadmin, node-red main interface
      webmin uses it's own access, which is root or pi, and their respective password

          1. ALWAYS use latest script, NEVER ask for help using old ones, as this could bring problems, to you and to us not understanding what's causing what else... in case, just say I'M USING VERSION FROM XYZ (DATE)... but better using latest, as said...

    2. for the internet access... without discussing security issues which will take way too much time... you need to create "virtual servers" or "nat" or "port forwarding" or whatever they are called in your internet router... if you have access to its admin interface, or you need to ask your ISP to do it... and open, for example:
      external port 1880 --> yourdeviceIP port 1880 (for node-red)
      external port 1883 --> yourdeviceIP port 1883 (for mosquitto)
      external port 80 --> yourdeviceIP port 80 (for apache)
      leave webmin and ssh away, for your good, if you don't know what you're doing... and use strong passwords...

  9. I've done another install on my Pi3 with a clean image of Jessie. I used the defaults on the script and I got through to the end without any errors. I did get a warning of an unused variable 'status' when the sqlite was installing though.

    Comments:-
    I though my Pi had crashed to start with until I saw the -qq switches on the update and upgrade. I'll remove the -qq next time I run it.

    I did like the elapsed time feature of the older scripts (now gone).

    Some packages recommend 'apt-get autoremove'. Maybe put this at the end of the script?

    Otherwise - awesome!! Well done Peter & Antonio.

    1. Hi Keith

      Can you clarify - Pi3 - you say Jessie - was this a generic Jessie or are you talking about Raspbian - very important to clarify.

      The qq options are to try to quieten down the "noise" coming off the installs - after all a BOATLOAD of messages telling us what has been successful are actually of no use - it would be nice indeed to figure out a reliable way of ONLY showing error messages - as they often get lost in the "noise". The old elapsed feature - yes I agree with you but that was based on a notion of interaction - incidentally the old script is still there for those who want to use it.

      Apt-Get autoremove - I see no reason why we should NOT do a cleanup at the end - Antonio - your thoughts?

      1. yes, as said on skype, when you install "something", it brings together an other lot of packages... when you remove "something", those addons packages are left behind... to remove these packages not needed anymore (as the packages using them was already removed), you do the "autoremove"... to remove the downloaded (let's say the setup ones, in windows style) and already installed packages, you do the "clean", too

      2. Hi Peter,

        Sorry, yes it was Raspbian straight from the raspberrypi.org website.

        I liked the elapsed time as it gave some idea of performance. You often mentioned how many minutes it took to install on your blog.

        I'm a ardent console watcher, so like to see the 'noise' 🙂 It gives me a warm feeling that things are still moving along as opposed to being stuck.

        1. in me, too 😀

          we added a (no)quiet menu option yesterday, you can use it now... but time depends on your menu selection, so compare is valid only on default selections... and can change from microsd to microsd on the same platform...

  10. Hi Peter,
    Great video & script. I've installed Ubuntu Xenial Desktop as a VM in Virtualbox and followed your instructions. However, the script has stalled at the point below. Tried twice but getting the same result.
    ================= Installing Node-Red ====================
    make: Entering directory '/usr/lib/node_modules/node-red/node_modules/bcrypt/build'
    CXX(target) Release/obj.target/bcrypt_lib/src/blowfish.o
    CXX(target) Release/obj.target/bcrypt_lib/src/bcrypt.o
    CXX(target) Release/obj.target/bcrypt_lib/src/bcrypt_node.o
    SOLINK_MODULE(target) Release/obj.target/bcrypt_lib.node
    COPY Release/bcrypt_lib.node
    make: Leaving directory '/usr/lib/node_modules/node-red/node_modules/bcrypt/build'
    make: Entering directory '/usr/lib/node_modules/node-red/node_modules/ws/node_modules/utf-8-validate/build'
    CXX(target) Release/obj.target/validation/src/validation.o
    SOLINK_MODULE(target) Release/obj.target/validation.node
    COPY Release/validation.node
    make: Leaving directory '/usr/lib/node_modules/node-red/node_modules/ws/node_modules/utf-8-validate/build'
    make: Entering directory '/usr/lib/node_modules/node-red/node_modules/ws/node_modules/bufferutil/build'
    CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
    SOLINK_MODULE(target) Release/obj.target/bufferutil.node
    COPY Release/bufferutil.node
    make: Leaving directory '/usr/lib/node_modules/node-red/node_modules/ws/node_modules/bufferutil/build'

  11. Just tried it on a fresh install of Armbian (Ubuntu 16.04.1 LTS) on my Olimex A20-Olinuxino-Micro, and it seemed to work flawlessly (meaning I didn't notice any major errors, and all the components seem to be functioning), in case anyone has the same platform and is interested in trying it.

  12. I just registered to share a comment about your script. A few days ago I booted the Raspbian "Pixel for PC" from a USB flash drive onto an old Toshiba Satellite laptop (M115-S3094). This afternoon I tried running your script on that machine. It worked! I rebooted and now have node-red running at 127.0.0.1:1880 with your large array of nodes to try out. Webmin is also working. I haven't tried anything else yet. Just wanted to let you know that the script works even with "Pixel for PC". I have a very limited knowledge of Linux so to me this seems quite a trick! Thank you and Happy New Year!

      1. as far as i could see in the few minutes i used it, it has no ssh server... so
        sudo apt-get install ssh # (to install ssh server+client)
        sudo passwd # to give root a password
        then it should work, as always

      2. I struggled with scp on the Pixel OS - always timed out when trying to send a text file to my Windows 10 machine. I was able to setup and use a PuTTY connection. I had to turn on the SSH server (OpenSSH_6.7, already there, shown in Webmin) and allow authentication by password. Finally, I installed WinSCP and was able to drag and drop a text file from the Pixel machine into a directory on the Windows 10 computer.

          1. Thank you for your help. I located the ssh block in the script, then I viewed the sshd_config file and saw the appended lines about UseDNS and ClientAlive... I have learned that Pixel for PC version 1.1 has the ssh server off by default. I don't understand Linux command syntax well enough to know if that would have any effect on the script. I do see the last line in the block restarts the ssh server, but the Webmin interface said the ssh server was off. Thanks again, no response necessary. The problem is most likely operator error. Right now I have a working system and plenty to learn!

            1. use raspi-config: https://www.raspberrypi.org/blog/a-security-update-for-raspbian-pixel/
              or, as per this link https://www.raspberrypi.org/documentation/remote-access/ssh/ :
              "For headless setup, SSH can be enabled by placing a file named 'ssh', without any extension, onto the boot partition of the SD card."

              these are the lines i added/modified to sshd_config:

              PermitRootLogin yes # to allow root login directly... i would not do this, but some wants...
              UseDNS no # to reduce the time between inserting user and the following password prompt
              TCPKeepAlive no # these 3 lines are for disable ssh timeouts
              ClientAliveInterval 30
              ClientAliveCountMax 100

  13. Just completed a rather long script install on a Raspberry Pi 2 having added a timestamp to the beginning of each section - I seem to recall some folk liking to keep track - also added (thanks Antonio) a progress bar. The screen is still WAY too noisy for me but we're working on that.

    The script despite taking ages, worked perfectly - bog standard Raspberry Pi2 with the November Raspbian.

    The FriendlyArm S2 also works but I'm just kitting that out with a new eMMC-based Debian to try again, leaving it to run overnight. I'll do a writeup on this board soon, probably tomorrow.

  14. Thanks Peter,

    Trying to run the script on a Virtualbox running MX-Linux (Jessie) and cannot get pst the npm installation part. Attached image shows the errors. I've tried a couple of times with no luck.

    Any ideas?

    1. Clearly, NPM was not installed. So you need to look back in that installation and see the earlier section to find out why not.

      Interestingly the only issue I'm currently having on VirtualBox is with Apache - in both the PIXEL and DEBIAN 8.6 images, Apache won't load and so of course PHPLITE INFO won't load (I'm sure we'll fix that today and that it is to do with getting the right installation image) - but NPN on VirtualPC - Pixel, Debian and Ununtu are all fine

      If anyone wants to jump in on this and help - by all means.. I'm thinking on the Apache thing we're going to need a menu option so the system knows this is a VM on a PC... but that still leaves the question of which apt-get etc.

        1. I've purged the virtualbox OS and started over. Get the same problem. npm seems not to be insstalling. The text below is what comes prior to the image II posted just above here.

          ====
          ==== Installing Installing Node-Red - Total Time: 00h:10m:02s
          ====
          sudo: npm: command not found
          --2017-01-05 12:04:48-- https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/nodered.service
          Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.20.133
          Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.20.133|:443... connected.
          HTTP request sent, awaiting response... 200 OK
          Length: 888 [text/plain]
          Saving to: ‘/lib/systemd/system/nodered.service’

          /lib/systemd/system/node 100%[===================================>] 888 --.-KB/s in 0s

          2017-01-05 12:04:48 (14.6 MB/s) - ‘/lib/systemd/system/nodered.service’ saved [888/888]

          --2017-01-05 12:04:48-- https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/node-red-start
          Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.20.133
          Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.20.133|:443... connected.
          HTTP request sent, awaiting response... 200 OK
          Length: 2111 (2.1K) [text/plain]
          Saving to: ‘/usr/bin/node-red-start’

          /usr/bin/node-red-start 100%[===================================>] 2.06K --.-KB/s in 0s

          2017-01-05 12:04:49 (9.68 MB/s) - ‘/usr/bin/node-red-start’ saved [2111/2111]

          --2017-01-05 12:04:49-- https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/node-red-stop
          Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.20.133
          Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.20.133|:443... connected.
          HTTP request sent, awaiting response... 200 OK
          Length: 226 [text/plain]
          Saving to: ‘/usr/bin/node-red-stop’

          /usr/bin/node-red-stop 100%[===================================>] 226 --.-KB/s in 0s

          2017-01-05 12:04:49 (2.63 MB/s) - ‘/usr/bin/node-red-stop’ saved [226/226]

          Failed to get D-Bus connection: Unknown error -1

            1. This Debian Jessie, distro is MX-Linux https://mxlinux.org/ . Tried both 32 and 64 bit versions.

              The computer is a Panasonic Toughpad with the same distro as the Host, using Virtualbox.

              I'm willing to test!

              Let me download the Ubuntu. Never been a fan of Ubuntu, but I will give it a go.

              1. please, while reporting bugs, EXPLICIT what script version you are using (and try to use ALWAYS the latest...), and which distro... if it's not 1 of those I listed, JUST SAY IT, we have no crystal balls to guess, nor we can make script work on EVERY single version of distros out there...

                1. Repaired this part of the problem, but editing the /etc/ls-release file. Changed the Metamorphosis to jessie, that way nodesetup recognizes the distro and proceeds.

                  Still can get past the installation of the nodes tho. Appears to a systemd problem.

                  Fun playing.

  15. apache+php7 install on standard debian+pixel sorted out... they report 32bit arch as i686... patch sent to Pete, which is trying pixel now, i tried debian, everything fine now...

    npm, still don't know, as it's not affected by php part of the script...

  16. I just finished running your script in Virtualbox 5.1.12 and everything appears to have worked flawlessly. I have used Virtualbox many times over several years for various reasons so that part was easy. The script itself took less that 15 minutes but I did not time it to get an exact time (Win 7 i5 2500K host). My "real" server is running Ubuntu 14.04 desktop and I'm looking to upgrade it to Xenial. I have been using MySQL vs SQLlite but your script has everything else I use plus more.

    Thanks for a great way to test things before I try upgrading my real server.

    John

    1. Hi john

      Well, the purpose of using SQLITE is to keep writing down to a minimum for SD memory on little SBC boards such as the pi - for simple logging, MYSQL though it would be my preferred DB is well over the top. Still - no doubt you can install that separately on your VM - and if you look at the script itself you'll see it isn't that hard to add sections - just be aware if you add anything to the menu - you need to change 2 values at the top of the menu to make it larger (because it isn't intelligent enough to autosize).

      1. Just getting back to the virtualbox installation of the script on Xenial and I can't seem to find Bigtimer in node-red. It is in .node-red/node_modules and I was able to update it to 1.4.88 using npm. However it still does not appear as an available node on the node-red web page. esplogin, grove and timeout appear under the scargill heading but no bigtimer. I don't understand enough about node-red to diagnose such issues. Can anyone point me in the right direction?

  17. PLEASE NOTE ALL - a BUG crept into NODE-RED-CONTRIB-BIGTIMER - I have updated this and fixed this bug which would have stopped bigtimer running.

    To fix on existing setups, go to your .node-red directory (probably /home/pi/.node-red
    and as PI user, from there merely...

    npm install node-red-contrib-bigtimer

    Update already but please wait until 9am today (Friday 6/1/2017) or later. The version you should end up with is 1.4.88 or later.

  18. Has anyone successfully run the script on a Pi 1 loaded with DietPi. I'm having no luck.
    "npm command not found"
    And yes I have a PI3 running well, but I have a couple of early PIs which I would like to give to my son.

  19. Hello

    Thanks for your work.

    Two comments :

    For real Raspberry, I would recommend to add to your script the log2ram tool. This is helping to save SD card and avoid corruption (logs are stored in RAM and dump to SD only once an hour per default). -> https://github.com/azlux/log2ram

    For your Debian stuff, you should have a look to the new X86 Raspberry/Pixel official image (if you've not had the info yet). This is made by raspberry org themself to run on PC or MAC and this should be pretty easy to have it on VM with persistence. This will be as close as possible to the regular raspbian ... -> https://www.raspberrypi.org/blog/pixel-pc-mac/

    Happy new year.

    Sebastien

    1. Hi there Sebastien

      Thanks for that and happy new year. Raspberry Pi - do you want to work up the exact instructions to add Logram and put in here why it's a great idea - if it is reliable we can always addt it to the Raspberry Pi menu!!! I'm all for offering to ditch logs actually - ie run in RAM but don't save. I only ever seem to use logs to check why something went wrong - not really interested weeks later and it would lengthen life of SD ??

      Debian - I have PIXEL running in VIRTUALBOX - works a treat - and the script works perfectly on it - it's a good quick way to get Debian and of course includes SUDO which Debian often doesn't . Don't get me wrong however, it is NOT a Raspberry Pi emulator... sound controls are missing - there is no RASPI-CONFIG and no hardware - like ports or anything... Shame - had they thought it through - a little usb board with ports on, made to LOOK like you have ports - would have been nice. Still - it is free and it works. Shame they're not ready to make it work on other boards because some of the alternatives to Pi, while having great hardware have really rubbish software.

      1. pixel raspbian was not meant for virtual machine (so the extra steps needed to have it working in a vm - why not doing as ubuntu, an icon on the desktop of the live cd, INSTALL, done???), and having it on actual physical hardware can bring audio and other stuff working... for ports, you can just passthrough them to the hypervisor of choice, not always easy but doable... clearly no gpio or other hw, only boards you can already have working via usb, like an arduino via firmata, or other boards Pete already tested in his blog entries in the past...

    2. frankly, the only advantage raspbian has is that's official... i don't care about the gui, i use ssh into my system every time, so pixel is useless for me... and, to be honest, it seems windows 95 with its icons... better than the former ones, but quite oldstyle anyway... but everybody says LOOK'MA, DAS ICONS! (there are TONS of better looking icon themes out there, from DECADES...)

      oh, and pixel has no clear instructions to put it in a vm, you have to dig into comments in official blog page to know how to do...

      raspbian has nodered and other stuff in it, but usually is old itself, too...

      ubuntu has newer packages already in its standard repositories, while dietpi has the best starting point for SBC, with low memory, low space needed, logs in ram, and many other optimizations...

      once setup, are all the same, you don't bother about what's underneath, your interest is in the service you use above the OS... which are all the same...

      dietpi is sure my distro of choice, right now...

      if nodered is your sole purpose, http://thethingbox.io is a good choice, too...

    3. Sebastien, this looks interesting.

      Along these lines I've been using CIFS to access disk space on my network shares (CIFS was easier to explain to friends than NFS or AFS). For when I must have local storage I dropped on a cheap 120G SSD drive and cross link it to the drive. I need to sit down and figure out how to mount the SSD as the writable part and the SD as just readable.

  20. Hello Peter

    I have to admit that I am stumped. I have been using Mosquitto for about 6 months but recently I had a crash, lost my installation on my Pi 3, and reinstalled all. Now I cannot get Mosquitto to work. I get this message when I enter this:

    sudo mosquitto_sub -d -t Humidity

    the error I get is:

    Connection Refused: not authorised.

    I used the script to install Mosquitto and Node-Red. Can you point me in the right direction please? I have searched the web but have not made any progress for the past several days. As I said, prior to the crash I had collected 100's of megabytes of data and controlled lighting in my home. Now, I cannot get Mosquitto to work at the simplest level.

    Regards,
    Bill Miller

  21. just to report, the script worked on Orange pi zero with Armbian_5.24_Orangepizero_Ubuntu_xenial_default_3.4.113.7z
    reported execution time 46 minutes.
    Thankyou 🙂

    1. Let me know how your WIFI works - particularly recovery from access point failure (just unplug) and general reliability... that is my one reservation about the Zero and I had no luck at access point failure recovery using Armbian (fine on hardwired ethernet).

  22. Ping times are awful (anywhere from 3ms to 2000ms; mostly 200-300ms).

    Wifi comes back up on restarting my bthhv2b. But, my wireless router is NOT my dhcp server, so it may not be an equivalent test..... Saw your topic on the armbian forums...

    ahhh.. just looked at logs. It did not reconnect to the SAME router (I have 2, and it knows about both...), so not a suitable test.... Interesting that it's happy to migrate though.

  23. looking through available source & firmware, looks like it's going to be wired only for me for the moment; no time to port wifi drivers this week.....

  24. Maybe this is not your field, but I will try this question to you.

    I reinstalled my system on a pi3 a few days ago when an npm update miserably failed and I lost all my nodes. As I am a Linux illiterate this was the only thing I could do.
    Now I found SD Card Copier and made a backup SD card with success, I swapped the written card with the original. As I did not like to do this job by hand every time I searched for another method and found this: https://github.com/aweijnitz/pi_backup.

    As I have an old 2,5 inch laptop hard drive I wanted to use for automatic backupping. Then the problems started as I found out I have a read-only file system and this HD cannot be written to with the above mentioned script. Really nothing goes!

    I suspect this behaviour could have been setup by the Christmas script, but that is strange. I know you are not a linux specialist yourself I cannot understand why this is the normal behaviour of the pi3.

    I have read some articles about corrupt SD cards caused by power outages. I now think I know why you are so keen on having this backup board with a lipo on it. My experience until now is that during all my resetting the pi3 by flipping the power switch myself when I thought things were screwed up I never had a SD card failure. You also have this experience in your house in Spain, where the power is not that reliable too. I found this article https://hallard.me/raspberry-pi-read-only/ that tells it is a good thing to have a read-only file system, but I will not fiddle with my system as described there.

    On the side of all this I noticed a thing about the pi user being a member of the sudo group. In the blog is is written you have to be sure pi is a member of it. What I see when opening the file manager then pi is listed as a member of group pi in the permissions box of the file manager. I have no idea if this is wrong and will have influence on the problems I experience!

    Can any one guide me out of this situation and explain how to be able to use the backup script on an external drive?

    1. So for other people having same problems:
      I installed gnome-schedule, gparted and I could work without learning Linux deeply. I edited the backup script to my needs and now I have a working backup automatically going every night. I formatted the external disk to NTFS so I could detach it from my "dead" RPI just in case and I can write a new image from the latest backup with Win32 DiskImager on my PC to the "dead" SD-card. Occasionally I make a copy with SD Card Copier, as extra safety. All with very little attention needed.
      Why not integrate this in the startup script to make live easier for Linux illiterates.

  25. Just a suggestion,

    How about adding the following to the install list?

    jquery, vnc (remote viewing) and the ftp node.

    Thanks for the good work!!

      1. What I meant by JQUERY was... if you're doing anything with Node-Red Dashboard - like using a template - you have full access to jQuery in there without having to insert a link

  26. Yes, to the Node-Red ftp node (node-red-contrib-ftp). I coundn't find the directory where jquery was located, but I will look again. Will do the same using a wildcard to see which version of the vnc server might already be installed with Raspbian. Might just need to do a configure for automatic startup at boot.

    Thanks.

  27. Ok, just got a couple of new RPi3's and downloaded the latest Rasbian and wanted to install the system "headless". After using the trick to have SSH enabled at boot up (which by the way don't work for Ubuntu even if that is the directions) 🙁 Then after following you direction plus a few more to get a working password for root and pi added to the sudo user group I loaded you script to /home/pi then chmod to 744. When I tried to run it with just ./script.sh I got a (bash: ./script.sh: /bin/bash^M: bad interpreter: No such file or directory) error. When I did it (bash ./script.sh) it just bailed out back to command prompt. No menu, nothing. Can someone tell me what I am missing??? Thanks

    1. Sounds to me like the file is saved in PC format.... needs to be saved in Linux format - that is - with only Linefeeds (I think that's right) and not both CRLF. I've never understood the message when that is wrong.

      1. the first line (called SHEBANG or HASHBANG, and starting with #! ) in a unix script tells the shell which interpreter to use to execute the following lines... the error is just saying that there's no interpreter called:
        /bin/bash^M
        as it's actually called
        /bin/bash

        the ^M is exactly what Peter said, the carriage return char, which should be stripped out...
        you can do it in linux itself:

        sudo apt-get install -y dos2unix
        dos2unix script.sh

        (the same package adds the unix2dos conversion tool, too, just in case...)

        info: https://en.wikipedia.org/wiki/Shebang_(Unix)

      2. Yes, that's what I had done.. I read your note but choose the wrong EOL. I saved it as Mac. It is running now, wo hoo!! Now to see what else I can foul up.

        Thanks.

        1. This new Microsoft Visual Code editor - whereas in Notepad++ it is as easy as selecting EDIT then the format - I'm not finding a similar menu item in the Microsoft editor - which is a bit of a pain - it would be nice if the editors would flash up somewhere what format you're using...

          1. found... honestly, this is done way better and easier in n++... anyway:
            open vs code
            ctrl-p
            ext install beautify
            on the left, install the one by hookyqr (has this icon: https://marketplace.visualstudio.com/items?itemName=HookyQR.beautify )
            reload vs code

            then open a file, and press F1 (or ctrl-p, then > )
            write: eol (and press enter)
            choose the desired eol...

            the extension itself does MUCH more than this, try to write beutify on a js, or css, or html file...

    1. Oh hell - is it THAT easy.... will you look at that - all those other things on the bottom do things as well.... I'm liking this more by the minute.

      Thanks for the feedback on this guys.Oh - and an update, too.

  28. Hi Pete,
    I used your script in my newest video and wanted to thank you for this great job. From the comments, my viewers like it, too!
    One question: Do you have any plans to include InfoxDB and Grafana in your script? It looks like these two are quite famous (and I was not able to get Grafana running)...

    1. Hi Andreas

      Thank you for the kind words... Grafana I'm more than happy with - InfluxDB - not so... I really would like to see someone do a SQLITE interface... I've had a couple of days of disconnects (power cuts in Spain) and some time ago a glitch -and there are values not necessarily out of range but wrong - InFlux simply won't let you do something as simple as pick a record or records and delete them. So if you don't mind I'm going to hang back on that a little in case a better, equally simple solution comes along - and of course - whatever it is has to work on everything from a Pi Zero, through various boards and phones to VMS - as does the rest of the script.

      1. Hi Pete
        I have some trouble with the credentials in node-red dashboard. Admin admin does not work for me...
        Can you guide me to looking up the user and change the password without re-installing everything?

          1. Thanks for your fast answer. Unfortunately, I am a complete Linux beginner. How do I "run" this script? I tried by just pasting it into the terminal and creating a .sh file. Both, unfortunately were not successful...

            1. Don't forget when running a script - it has to be in Linux format - that is line feeds only - you can ensure that in NotePad++ under EDIT - EOL CONVERSION - LINUX - and the script once created (it IS just a text file) needs EXECUTE permissions.

              If you get the format wrong - the resulting error message is not helpful.

  29. I changed to the pi home director and started the two commands (Wget and bash). Unfortunately, I always get the error message "line 5: $'\r': command not found
    ".
    Then, I created a file using sudo nano and copy-paste raw. Same thing...

  30. I was not able to install dos2unix with your command (it did not accept the -y). So, I went the way via notepad++. Now, I get no error message, but the script does not return anymore... (it also does not react to ^C. I have to logout...
    This seems to be really hard... (for me, at least)

    1. Andreas - as the script clearly works - would it not make life easier here to simply reload the operating system image and run again. It doesn't take any interaction once it starts as you know.

      1. You might be right... Did not think, it will be so complicated. But I always learn. So, I will start over. Thanks for your support!

  31. The script was looping endlessly, because of a lacking settings.js in my directory. Now, I am one step further. I get an error message at node-red start. But I have to run...

  32. The second problem was, that, everytime the script runs, it adds a section, which is already there, if node-red was installed by your script...
    Now, it works (I followed the advice of Peter).

    httpStatic: '/home/pi/.node-red/public',
    functionGlobalContext: {
    os:require('os'),
    moment:require('moment'),
    fs:require('fs')
    },

    adminAuth: {
    type: "credentials",
    users: [{
    username: "admin",
    password: "$2a$08$o9n62Pq6s5YVfD2C.inYKucfIOp26eYart4s0XeUB3Cbf7h.GIsrC",
    permissions: "*"
    }]
    },

    httpNodeAuth: {user:"user", pass:"$2a$08$z2m9DthU3ZrfFtDx7ceD..HlFOuJvdX2Plmdkpvg//YNfCYbJ2BpW"}
    }

  33. You are right. I did it then, but ran into my first problem (infinite loop) and forgot about your comment. Sorry about that.
    In the meantime, I found out, why the loop was infinite: If node-red still runs when I start your script it does not create a "virgin" settings.js. So, I added the line:

    sudo service nodered stop;
    at the begin of the script and now, it runs perfectly. Thank you very much for your patience!

  34. WOW! This is great!
    Saved me a lot of work!
    Thank you so much!

    And - don't try this on orange pi official rasbian distro - that is total crap and I wasted 3 days trying to install everything on it. On Ambrian it did everything for less than an hour.

  35. I ran The Script last night on a NanoPi M1+ and it took longer than I could wait so I left it overnight. In the morning it had stopped from a connection timeout at the "change hostname" dialog box.

    Maybe a better option would be to move all user input requirements to the top of the script so it can run to completion unattended after a few user input items at the beginning?

      1. I reran it and sure enough, it's waiting for the hostname and finished. I could have looked at the script but wanted to see what it would do re-running it. 🙂

        Wish List: move hostname prompt to the top and add a menu item to reboot when complete.

          1. oh, the nodejs install from xz package, is less noisy too, but we tested only in vm, so you didn't realize that... needs testing on a arm64 board (you know which 🙂 ), and/or a rpi0 or rpi0w... i added "pv" to have it show a nice progress bar

      2. mmm, we can do that... ask at beginning, and leave the current ip and hostname at the end, where they are... 🙂

        needs testing, as changing hostname while script running already brings some little delays at the end and some warnings, which i suppressed as we do a reboot soon after, and after reboot all is fine...

  36. I ran the script on NanoPi M1, left it over night and hostname dialog or connection timed out. I reran the script and came back in 1 hour and changed the hostname and rebooted. Logged in via ssh as pi user. Browsed to device http://hostname.local and saw the "Pete's Utilities" page. WebMin worked, Node Red didn't, OPI Mon worked, SQLite Admin and Sys Info worked too. Only Node Red didn't work because of the 2nd run of the script and double entries in settings.json.

    Note Red installation had multiple entries for admin and user Auth's so once those were cleaned up in pi/.node-red/settings.js and rebooted it worked.

    In The Script, line #6 of "IMPORTANT" states to not access(run?) the script as SUDO. But my understanding is that on systems without a pi user it has to be run first with sudo and then run as pi user.

    1. The hostname dialog has now been moved to the front of the script.

      If the Node-Red installation has mutliple entied for admin - the script or part of it has been run twice by some means or another.

      If the system has no Pi user, running it the first time as ROOT will create the Pi user and will now move the script - with permissions modified, to the Pi directory - you then log in as Pi (password password unless you change it which is a good idea) and you run again and that completes the operation.

    2. in the version just released of the script, if you try to rerun the node-red setup it will skip the step and go on, preventing double lines... if you want to rerun, at lease rename settings.js, but best rename or move elsewhere the .node-red folder

  37. I am having problems with running the script on the latest release of raspbian jessie lite,
    The problem starts when installing Node-Red,

    --2017-04-21 05:00:23-- https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/nodered.service
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 62.252.172.241
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|62.252.172.241|:443... connected.
    GnuTLS: A TLS fatal alert has been received.
    GnuTLS: received alert [0]: Close notify
    Unable to establish SSL connection.

    I have done a few google searches but cant seem to find any information on this.
    Any help would be appreciated.

    1. Hi Craig

      The script has not been tested on Raspbian Lite, just on the full Raspbian - and amongst the things you can do is remove office and that maths program which collectively take up a lot of room.

      1. no problem on jessie lite, i used that, i don't like wasting space for a gui (plus the tons of gui programs) i will never run... their base is the same, only difference is the presence of the Pixel gui in the full version...

        the problem of craig is the same of that guy we solved in chat a while ago: some of the github round-robin served servers have wrong ssl config... solution: add this line to /etc/hosts and retry:
        151.101.128.133 raw.githubusercontent.com

      2. solved, to put in script... it first tests if the actual github server returns a good certificate, in case does nothing, while in case of a wrong one, it adds a good host/ip association to /etc/hosts... Peter, add these 2 lines before the one at line 754, but after the ELSE on 753

        echo | openssl s_client -showcerts -servername raw.githubusercontent.com -connect raw.githubusercontent.com:443 2>/dev/null | openssl x509 -inform pem -noout -text > /dev/null 2>&1
        [[ $? -ne 0 ]] && echo 151.101.128.133 raw.githubusercontent.com | sudo tee -a /etc/hosts

      1. I've just tried with the recently updated script and still get the same error.
        The first response to this issue i tried and that worked O.K.

        I now have my Pi up and running but if you need me to test any more changes to the script regarding this issue just let me know.

        1. i tried those lines, now in script, and here worked well... did you try before or after that mod i suggested? Which does nothing but the same thing, as 1st line in script now tests if ssl certificate is correct, and 2nd one adds the line you added manually in case 1st line returns an error... so, it should have worked...

          can you remove that line from your /etc/hosts file, and run these 2 lines and report if they say OK or KO? or better, run these 2 lines first WITH and THEN without that line in hosts, and report both results? Thanks

          echo | openssl s_client -showcerts -servername raw.githubusercontent.com -connect raw.githubusercontent.com:443 2>/dev/null | openssl x509 -inform pem -noout -text > /dev/null 2>&1

          [[ $? -eq 0 ]] && echo OK || echo KO

  38. Hi Pete/all,

    just working with my Orangepizero, restoring it after an SD card corruption. Whilst looking for regular writes, I found that WebMin has folders on the SD card which it both logs to and uses for temporary files for serving up info.
    'The script' installs log2ram? This effectively makes '/var/log' a ram disk, but persists it at startup and shutdown, and once per hour?

    As a test, in '/etc/webmin/miniserv.conf' I'm changing
    logfile=/var/webmin/miniserv.log
    errorlog=/var/webmin/miniserv.error
    pidfile=/var/webmin/miniserv.pid
    env_WEBMIN_VAR=/var/webmin

    to
    logfile=/var/log/webmin/miniserv.log
    errorlog=/var/log/webmin/miniserv.error
    pidfile=/var/log/webmin/miniserv.pid
    env_WEBMIN_VAR=/var/log/webmin

    and
    sudo cp -r /var/webmin /var/log/
    and
    sudo nano /etc/webmin/var-path
    (change the path in there too).
    sudo service webmin stop
    then
    sudo service webmin start

    and now webmin is again running, but with no regular SD card writes (except once per hour when log2ram will store everything in var/log).

    webmin probably does a LOT of individual small writes, which could do quite some damage to the SD card over time (e.g. the sessions db is in there....).
    This may be a useful addition to the way the script installs webmin for those of us with fragile media?

    Simon

    p.s. My home-auto node-red flows also persist data to disk (graphs, etc.), so I've moved them to a folder within the /var/logs folder; far better a once-per-hour write than a once per minute - and no need to design SD protection into the flow... but you do risk up to an hours' worth of data if you have an unexpected failure. I decided better lose an hour of data than 4 hours of my time restoring the whole setup with a risk of wife-rage because the hot water did not turn on!
    mosquitto is still writing to SD occasionally.
    I'm using 'sudo iotop -a' to observe disk writes - just leave it on for a while, and offending processes float to the top.

    1. ohhh.. reading above, I don't know of log2ram was a script install, or if it was part of the armbian OS image I started with. If it never got into the script, I highly recommend it :).

    2. after a reboot, webmin did not start; seems log2ram did NOT copy /var/log/webmin/ to /var/log.hdd/ on a sudo reboot.
      Since I believe the folder is only temporary files effectively, I've done a manual copy, then the reboot works. Also from brief reading, log2ram should copy it....

        1. i remember there were references to apache, which should be explicitely added as to start AFTER log2ram start, otherwise it would fail... maybe it's the same for webmin... but on that github there are no more references to apache...

        2. Note: armbian has
          https://github.com/armbian/build/tree/master/scripts/log2ram
          ('updated' - mainly seems to be about config and script location)
          log2ram script appears to be disabled at first look, but in armbian version, actually loads config from /etc/default/log2ram which overrides the enabled=false in the script.
          Mine appears to be set for 50M, of which 28% is in use (including webmin folder + 7M of my own data).
          Rsync was OFF; I just turned it on and it seems to be doing 'reasonable' things.
          also found:
          https://github.com/Fourdee/DietPi/issues/781
          which may be relevant to some rpi users.
          Maybe if we're to use (as I am) /var/log as ram based perm store for NR, we should add the NR service to the 'before' line too....
          (my apache2 was failing to start - but because /var/log/apache2 did not exist).

  39. Nodejs in the /dist/latest-v6.x/ folder is now 10.2 instead of 10.1, which makes the installation fail, then nodered fails too.

    Please note there is also nodejs 7.

      1. Fully agree on not using 7 or 10.2 for that matter. It is just that script now hard fails the nodejs installation. So my suggestion is to either change line 712 (and 711) to use 10.2 instead of 10.1, or use /v6.10.1/ instead of /latest-v6.x/

        Should have been clearer about that.

      2. just checked, and he's right, nodejs 6.10.1 was deleted from node servers and now there's only the newer 6.10.2... every mention to
        node-v6.10.1
        should be changed to
        node-v6.10.2
        in script, now...

        he's NOT right about nodejs7, as it's not STABLE and we use only stable versions...

  40. Is there any reason why the script doesn't install fail2ban to deter hackers?
    And a related question, why do you choose to enable the root login on a Raspberry Pi?

    I do occasionally see strangers attempting to login as pi but nowhere near the number who try as root.

    1. because in general we use this setup locally... i don't enable root on my systems, but others prefer using it, maybe just "habits"... i use vpn to connect to my services from outside... anyway, this is just an easy setup of a bunch of services, suited for most but not everybody, and whomever can modify their setup and make them suited to their needs... feel free to close everything, disable ssh root login, add ssh keypair login instead of user and pass, add fail2ban, etc

    2. The script does not do a lot of things. I've never heard of fail2ban and I don't add anything I'm not familiar with and feel comfortable with. Root? Because here are some things that are a pain in the backside to deal with, without Root. Generally speaking I run everything as user Pi.

      1. Fail2ban:
        If you open SSH to the internet on the default port, you very soon attract login attempts, lots of them.
        grep "Failed password" /var/log/auth.log to see how popular this is with the hackers.
        All fail2ban does is ban an IP address for 10 minutes after 6 attempts. Of course it's configurable, that's the default.
        (On the Pi) sudo apt-get install and it starts working, no setup needed.

        I'd also suggest that logins and passwords for node red, webmin etc should be defined at the top of the script rather than buried in the code.
        Best regards,

          1. Passwords deined at the beginning...
            Lines 357 - 363 in the script:

            username="user"
            userpass="password123"

            adminname="admin"
            adminpass="password123"

            newhostname="myboard"

            Only useful if you...
            Yes that's true. It's just a suggestion for a simple enhancement.

            1. question isn't where password are (they're at qute the beginning of the MAIN SECTION OF THE SCRIPT, the first part are general functions never touched and that a while ago were in a separate file, now included just for easier management), question is if you ever run the script or are questioning just for the sake of it... if you ever run it, you should know that there's nothing you need to edit yourself, and that running it there's an option in menu that uses THOSE defaults and asks for new ones, JUST AT THE BEGINNING OF THE INSTALL process... after those first seconds, you can go to take a coffee or what else, and return after about 15-30 minutes, depending from your board speed... and now, enough...

              1. I was asking because i have on occasion recommended your script on the Raspberry Pi forums and I wondered why you did/did not do a few things that slightly worry me - including enabling root login and not installing fail2ban. But please don't think I'm criticising your work, just asking questions.

                Yes I do run it sometimes. I have never previously done so without modifications to suit my preferences.

                However, as an experiment I just tried running the script unedited on a headless Raspberry Pi 3 with a fresh copy of Raspbian 2017-04-10 to which i added /boot/ssh and /boot/wpa_supplicant.conf.
                It fell over at this point:
                ++ whiptail --title 'Main Raspberry Pi Selection' --checklist '\nSelect items for your Pi as required then hit OK' 30 73 23 quiet 'Quiet(er) install - untick for lots of info ' ON prereq 'Install general pre-requisites ' ON phone 'Install on Android Smartphone - see blog' OFF mosquitto 'Install Mosquitto' ON apache 'Install Apache/PHP/SQLITE + PHPLITEADMIN ' ON nodejs 'Install NodeJS' ON nodered 'Install Node-Red' ON webmin 'Install Webmin' ON screen 'Install Screen' ON java 'Update Java' ON wiringpi 'Wiring Pi for the GPIO utility' OFF mpg123 'Install MPG123' ON modpass 'Mod USER and ADMIN passwords (password123)' ON phpsysinfo 'Install PHPSYSYINFO' ON upgradenpm 'Upgrade NPM to latest version ' ON addindex 'Add an index page and some CSS' ON passwords 'Update ROOT and PI user passwords' OFF installcu 'Install CU for serial VT100 Terminal' ON installmc 'Install MC+MCEDIT file manager + editor ' ON installjed 'Install JED file editor' OFF habridge 'Install HA-bridge on port 82' OFF log2ram 'Install Log2RAM default 40Meg' OFF ' '
                Box options:
                --msgbox
                --yesno
                --infobox
                --inputbox [init]
                --passwordbox [init]
                --textbox
                --menu [tag item] ...
                --checklist [tag item status]...
                --radiolist [tag item status]...
                --gauge
                Options: (depend on box-option)
                --clear clear screen on exit
                --defaultno default no button
                --default-item set default string
                --fb, --fullbuttons use full buttons
                --nocancel no cancel button
                --yes-button set text of yes button
                --no-button set text of no button
                --ok-button set text of ok button
                --cancel-button set text of cancel button
                --noitem don't display items
                --notags don't display tags
                --separate-output output one line at a time
                --output-fd output to fd, not stdout
                --title display title
                --backtitle display backtitle
                --scrolltext force vertical scrollbars
                --topleft put window in top-left corner
                -h, --help print this message
                -v, --version print version information

                ++ wolfram 'Remove Wolfram on a PI to save space' OFF office 'Remove LibreOffice on PI to save space' OFF
                Wolfram Language 11.0.1 Engine for Linux ARM (32-bit)
                Copyright 1988-2017 Wolfram Research, Inc.

                In[1]:=

                (Yes, I did edit the script and insert set -x to obtain the above)

                Not complaining, just letting you know.

                1. Thank you - I am going away for a few days so can't progress this now but I just had someone use the script successfully barring one item which we fixed (nodeJS) today. I have root because I want it - it is not necessary for this script... and I've no interest right now in fail2ban - others can add these things if they want along with anything else. The script will work with a basic Raspbian - I cannot really comment on it's operation outside of that basic scenario or if it is modified.

  41. Ref 'The Script' & mosquitto:
    I think mosquitto needs to have /var/log/mosquitto created and 'chown'ed to mosquitto:mosquitto in order to log...
    In my quest for minimum writes, I set it's persistance path there, and then wondered 'why no logs? why no db?'. only when I created the folder and 'sudo chown mosquitto:mosquitto /var/log/mosquitto' did a restart cause logs + db to be present.
    Can someone check on another system to see if you have mosquitto logs, and if not, we may want to add this to 'The Script'?

    1. must have been something about the state the Armbian OS was in when I installed - maybe *my* (Armbian pre-installed) log2ram did/does not correctly copy to log.hdd on shutdown, and after The Script install, I rebooted before the hourly copy..... mosquitto always worked, it just did not log.... Feels like the same non-copy bug was responsible for Apache not running too. Something to highlight in the main post maybe.
      'In at least one installation where log2ram is pre-installed (e.g. Armbian), it has been observed that folders created in /var/log/ by the installations may not get copied to the /var/log.hdd/ folder at reboot, resulting in services failing or misbehaving. This can be avoided by a manual copy after installation or by waiting an hour and checking /var/log.hdd/ matches /var/log/'.
      Or we could add a 'sudo log2ram write' to the end of the script, which will (quietly?) fail if log2ram is not present, but solve the issue if it is?
      It may just have been some one-off.....

      1. Open an issue on the log2ram github if you have problems (and the last version of course).
        Check you starting log, you will have error from systemd for the log2ram service. And there are the file /var/log.hdd/log2ram.log file to watch too.

  42. Hi Pete!

    Until now, I have used only parts from your script. But now I tried to run it complete, and there was no menu displayed.
    The buggy line was
    "log2ram" "Install Log2RAM default 40Meg" OFF \
    Behind the \ there was a TAB and then LF. I had to remove the tab to get it to work.

    1. It is highly likely your diagnosis is correct as this was a brand new addition to the script. I have loaded up the script -ensured that it is a space before the backslash and saved - so hopefully others will not come across the problem.

      Thanks for bringing this to my attention!!!

  43. Hi Pete,
    As I found out that "G P" is right:

    "The buggy line was "log2ram":

    "Install Log2RAM default 40Meg" OFF \ "
    Behind the \ there was a TAB and then LF.

    I also removed the tab to get it to work.

    Thx.
    Nadav

  44. Hi Pete,

    Love your work.
    The script worked like a charm.
    It took over 10 hours to install on a Pi Zero...

    I started on a similar project a couple of years ago with nodejs,
    never got very far with it because of time issues.
    Recently i wanted to start it up again and then found node-red.
    Love it.. has a huge potential. Now i'll start over with node-red.
    Also like what you're doing with the esp's too.
    Did a small test project with them, and to keep it small i use
    the 128x64 i2c oled display's for status and debuging. I use them on almost anything, they' re so cheap these days.

    I have question about your system:

    Why do you use imperiHome on your phone?
    You could do al that with the dasboard in node-red.

    keep it going,
    may thanks,

    Kiki

    1. Hi there Kiki - thanks for the kind words. Re: Imperihome - I think it depends on what order you read the blog - I am indeed now favouring the Dashboard over Imperihome. However..... there is one little niggle with Dashboard that as far as I know has never been resolved... if you only use Dashboard then the inputs and outputs stay in sync with your phone but imagine you have a timer running in the background - changing the state of things - it is sometimes necessary to refresh the display to make sure it is up to date. In other words I push changes regularly when the phone which is running Dashboard is operating.... and therein lies the issue - the password system - and I've mentioned this in the forums as have others... there is something not quite right about the security/password setup in that it regularly asks for the username and password when you go to the dashboard (only on Android phones, never on my Windows 10 PC) - and that's a pain but not TOO bad.. BUT if you have any kind of refresh going as I do - you get half way through entering the password and the screen refreshes - indeed to use the dashboard reliably I've had to stop refreshing the screen as such.... the thing is - it should not be asking for username and password once you've stored it - but it does from time to time. That is currently unresolved... But generally speaking yes I'm finding it better to work with than Imperihome - which STILL is missing some generic controls - they seem intent more on supporting every device on the planet than providing generic controls that we can use in Node-Red - hence you'll see I spent some time working with templates in Dashboard and various gauges etc..

    2. Pi Zero - yes, very nice but WAY too slow and the script only serves to remind us how slow it can be. I must say - even with the outstanding issues (ease of use of GPIO) I'm favouring the FriendlyArm boards such as NEO etc... slightly more expensive but WAY more powerful and they are a helpful lot.

  45. My pi3 nodered just suddenly died. It worked for months and now I cannot start node-red anymore, all kind of error messages with no clue what is wrong. I did an upgrade of Raspbian and it only gave worse error messages with Syntax error!
    Is it possible to reinstall node-red and then run part of the Script to reinstall all extra nodes or... is a complete reinstall necessary?
    I am now trying to reinstall from backup image from about 5 days back, but that is a separate excercise.

    1. i think you can still recover your flows from the .node-red folder
      about reinstalling, you can just rename .node-red folder and redo just all the npm part, without actually reinstalling node-red itself

      1. I have done a new install ran the Script. I backed up the flows files in .nodered directory. I copied them back to their original location and nodered does not "see" them, stopped and started nodered. Any idea how I can activate my flows? There is a function import, but I only see a map example and cannot freely choose the backed up file!

  46. Now I only have to copy back the database with my gathered data into SQLite, I have a file in the same directory as where the iot.db is. How to import that file into SQLite?

  47. Thanks for your swift answer, but I do not understand the process you suggest. I have a filled otherthaniot.db file in the same directory. I do not have a database in SQLite yet. I like to have a database with the same name as the file, restoring the old situation.

    1. My database can receive new data, but phpliteadmin is not showing the records in the database! When I create the database in phpliteadmin with the name of the file in ~/dbs then I get an unknown error. That can be expected, but how to synchronize phpliteadmin to the file in ~/dbs??? B.t.w new records are created in the database from my restored flow, I see that on my dashboard graphics! So SQLite is functioning.

          1. All kind of wrong permissions needed to change. Now database is accessible again! Other problems coming up like MQTT broker does not start anymore after reboot????????????
            How to start the broker and how to set that it starts automatically at reboot?

            1. honestly... you're having too much problems with an evidently corrupted installation... save what you need and start from scratch... easier... or use a different sdcard in case forgetting something back...

            2. I agree with Antonio - this is sounding like a corrupt installation - maybe SD issues, maybe power - but I would not be continuing - you'll spend more time trying to fix than you would starting from scratch - check power - check that the SD card is a good one.

  48. Thanks Peter, This is good information for beginner like me.
    I just run the script from fresh install dietpi to my old raspberry Pi.

    It seems Nodejs should be change to the latest node-v6.10.3

  49. Thanks for this great script Peter!
    I just installed an Orange Pi PC+ with it and it worked like a charm!

    1. Excellent - enjoy - well it's not just me - I've had lots of help from Antonio - and originally when I was REALLY rubbish at scripting, my friend Aidan Ruff was involved and we've had input and feedback from all sorts of people. I think I probably hold the record for testing hours 🙂

  50. I am again having a PI3-system up and running. As i got the feeling I 'll have to make a better supply,instgnated by your remarks and your newest blog entry, I gathered some components together and made my own UPS. I already had a lipo charger, a 1500mAh 3.7V, a buck converter up to 5V with those little V-meters on the lipo and the output. That whole thing was triggered by the misery I contracted for some reason and a bad supply could have been the reason. I only miss the elegant switching off when the battery goes really low by a power outage that stays out too long. I only have to really test how long it will last without mains, but this stabilizes the voltage much better. I found out the difference between the voltage a the micro USB differs about 0.2 V compared to the pin header value. I can reduce that when I connect my little 2,5 inch HDD on the USB which has extra power from another charger externally.
    I estimate this kind of supply is not a gadget for unstable mains, but also when using a standard charger a necessity.
    When I was running the script I have seen quite a number of warnings. Those might be ignored, but I am wondering if there is a log of all those messages running on the screen or did I miss a check somewhere in the starting menu?
    I did something wrong when asked for the HOST name. There another name is suggested "myboard" and it says something like if you do not want a change just press OK. However the HOST name is changed to myboard from raspberrypi. I once renamed my HOST also and that gave all kind of permission problems so I will not touch that.
    For some reason it took about 1 3/4 hour to finish, but the system is functioning at the same slow pace it did before I forget about that.
    I will start working on a better back-up strategy. At the moment I will start with a copy of the SD card with the SD card copier which is standard in the raspbian jessie firmware. That should bring my up and running again much much faster than doing it from scratch.

    1. There are warnings Leo but that's what they are - warnings - of issues in other people's scripts - over which of course we have no control - usually they are to do with unused variables - why on earth those are not turned off is beyond me.

      On power supply switch off - see my latest blog - I've just hooked up a cheap OLED to my NANO and this can now be added to a supply to make it shut down the Pi or whatever gracefully and recover when the power recovers. All working.

      No there is no log - but by all means make one...

      If you don't change myboard it will be myboard - change it to raspberrypi if you want to call your board raspberrypi.

      The script takes anywhere from 7 minutes on a fast PC with Virtual Linux - to several hours on the lamentably slow Raspberry Pi Zero WIFI.. on the NEO 2here I think about half an hour or so - I'm quite getting to like this little board, I'll like it better when Friendlyarm get around to releasing proper GPIO code and the ability to use more than one of the four UARTS.

      1. I have read your latest blogs about uninterruptable supplies and GPIO, but that is , at least for me, shooting with a canon on a mosquitto(lol!). A whole new field of knowledge. I understand that if not running down the PI3 gracefully it might wrecken the SD card software in a lot of cases. I am surprised why your unit in Spain has never died when you say there is a horrible mains(solar supply?).

  51. The script took approximately 12 hours to run on a Raspberry Pi model B1, but in the end it did finish!

    I found two things that I would like to note:

    1- the rc.local file gets mangled up somewhat, there are two lines on the script with a sed command to change "exit 0" to something else, and one of the two changes "exit 0" to "chmod 777 /dev/ttyAMA0 exit 0" in the comments in the top part of the script, and the line break gets mangled too and thus the rc.local file fails on startup.

    I believe this line is the culprit:
    sudo sed -i -e 's#exit 0#chmod 777 /dev/ttyAMA0\nexit 0#g' /etc/rc.local

    2- I appreciate that IPv6 is still not very much used, but I do not understand why you are shutting it down for good in the script, both in /etc/sysctl.conf and in /etc/avahi/avahi-daemon.conf

    I use IPv6 and had to remove those lines to regain IPv6 access to my Pi. I noticed as I did start the script from an IPv6 SSH connection to my Pi, and as soon as I started the script, logging stopped as I killed my own SSH connection! 😀

    A great script, which helps immensely on starting up with a more IoT-oriented software load on the Pi.

    Many thanks, Peter!

    Luca Bertagnolio

    1. Hi there Luca

      Yes the old Pi boards really are not up to it - I think I threw my last one in the bin. Of interest, comparing your 12 hours - Ununtu running in emulation on a PC takes about 15 minutes - and I think the Pi3 takes way less than an hour. Much depends on broadband speed of course but basically the original Pis are now showing their age.

      That AMAO mod I'll ask Antonio to take a look at - that's a recent change - as for IPV6 -my take on this is that recently it has started to cause issues with people's routers being unable to resolve the addresses so the simplest solution was just to disable it - in the future we'll take another look at that. I am assuming those who use IPV6 specifically remain for now a minority - sorry it cave you problems.

      1. Hi - (and thanks for the work you've put into this and for publishing...)
        I've just run the script on a fresh installation (stretch lite) on a raspberry pi B +
        and the only problem was that rc.local is still being mangled. Simple fix to edit but thought I'd let you know.
        BTW - script took 2hrs 17mins.

  52. This works on the CHIP! I like that I don't have to worry about it too much with a 2500 mAh battery connected. The only challenge I had was that node-red won't accept my login credentials from Safari. I kept thinking I had entered the password wrong and re-ran the script a few times till I figured it out. Chrome worked fine.

    Any cautions with using the CHIP as it has no removable (i.e. easily replaceable) storage?

    1. Excellent - well, the latest version uses log2ram so while it will use up some of your RAM it may improve long term life of the FLASH... As that has a battery you might be able to change what I think we set as an hourly cron job to a daily cron job and keep the writing down even further.

      1. yes, i tested on the chip a while ago, i think about christmas, during the script rewrite we did that time... veeeeery slow, i think as the rpi1 and the rpi0/w... the good, it has an emmc... the bad, it's only 4gb and slow, that, too... but sure for a device of 9$ with embedded storage, is quite good, have to test it's gpio part, it has a good support and documentation on its site... what you did on the nas can sure be replicated on this little monster...

        p.s.: actually the original log2ram script flushes its caches every hour, you changed to every day for yourself 🙂

      2. I read somewhere you said your houses are connected, are your pi servers clustered? I'm not sure if that's the right word though, but basically that they are redundant and that mqtt nodes can failover and publish to any of them?

        Also, thank you for this resource. It's amazing the amount of work you've put into this, and the result is outstanding. I'm working on a design for an open source water treatment controller, something that could handle a fish tank, a hydroponic system, or wastewater, and all of this is a big help with prototyping the control and monitoring side.

        1. They are connected - each has it's own MQTT broker and they share information. It is also a handy way to know if something has gone wrong because either can email me if the other is not responding. They are not clustered however. Right now my ESP code can handle two WIFI access points but only one MQTT server. That could get quite tricky - if one of them went off on a tangent and others didn't.

          AS Robin Williams said .... glad one can be of service 🙂

  53. Housekeeping for The Script

    Peter, Just downloaded the script - in the initial Menu once it is running - there is a typo in the option to Update NPM to latest - it says NPN

    Otherwise looks good.

    In your instructions in relation to VMware - you might also want to give some info on how to install VMware Tools into the VM

    1. Hi - thanks for that - will fix it - be VERY wary about updating NPM to the latest version - I could not get Node-Red to start with it. I saw a comment in the Node-Red Google forums that it DOES - but isn't the recommended option. That why I un-ticked that option by default.

      Ok, script is fixed 🙂

      1. Yep tried that with the NPM upgrade - took a snapshot first just to be safe !!

        I love Virtual Machines !

        Craig

  54. Peter,

    OK weird one here - did a Virtual Machine (Vmware Workstation 12 Pro) deployment of The Script - using the Ubuntu Server ISO as per instructions.

    All worked fine - no issues whatsoever. Assigned a static IP address to the Virtual Machine and can login through SSH with no issues.

    Node-Red comes up fine - however whenever i try and connect nodes to it i am gettting errors. Basically on the canvas all the MQTT nodes have a disconnected underneath them.

    I exported some nodes from another Virtual machine i have been playing with (using Centos and my own manual install of most things) imported them into the canvas and the same thing happens.

    if i change my node (it is an Arduino Mega with ethernet shield) to point from my Test-VM (where it is sending status messages and receiving Time Updates) to the new VM where the Script is i have all the nodes as disconnected.

    I then tried MQQT-SPY and tried to connect to the topics - no good (but works fine for my other test VM)

    I thought maybe a problems with IPTables/Firewall so i shutdown the UFW on Ubuntu but still the same issue - any ideas - something weird with non-standard port numbers, authentication etc ?

  55. Guys ( Mr. Scargill and all those that helped) Thank you all for this effort. And DrFragle and MrShark - thank you for the mention of DietPi - to this point my only choices were Raspbian (nothing wrong with it) or more efforts and NOOB..

    I've just put DietPi on a Raspberry Pi 2 B with minimal additional software, then ran "The Script" as it was four days ago. And tada! working system.
    The note about dos2unix above was a big help, too. Glad I learned to read a bit before jumping in and asking questions.

    I've always gotten to a point (Pi loaded and ready to go) and then wondered where do I go from here? Well, I've got a better idea, now. And a couple Arduinos, a couple of NodeMCUs, and a box of sensors, so I think I might get somewhere, finally.

    One, incompletely informed question - I see WEMO boards mentioned here...would those be any relation to the WEMO switches? Which means I could control the outdoor floods from my Pi?

    Now, where did I put my log in credentials......

    1. Wemo - Wemo D1 - check them out on Ebay or aliExpress - just a cheap, very small form factor ESP-12-based board, I bought a couple - there are also some header boards available like a relay board... so nothing special except they are a nice small size and inexpensive.

      1. maybe it was just a problem with actual naming, as the esp8266 boards are called wemoS, not as plural of wemo but by themselves 🙂
        the full name of the small board is Wemos D1 Mini, as there are also the d1, the d1 r2 (both arduino uno form factor) and the d1 pro and the d1 lite, same as the mini but with more memory the former, and a different chip the latter

        1. Found/recreated those pesky login credentials. The WeMo Switch is actually a Belkin product - and apparently an American thing - according to Amazon, not available for shipment over seas. So I will have to do bit more digging on this side of the "pond". I've had the switch in place for a couple of years for those evenings when I get home after dark. Use the phone to turn on the outside lights - get inside. Use phone to turn off lights.

          There are a few how-tos that mention using an ESP8266 (or similar) and a relay to recreate the same functionality.

          Again, thanks to all y'all that have contributed.

          1. Oh WEMO - as in working with ALEXA - easy - there are a couple of software packages that do that - and if I am not mistaken the new software that ITEAD are putting in their SONOFFS does the same thing now.

            This is probably the best replacement software (tested by me)

            https://github.com/arendst/Sonoff-Tasmota

            then there is this (untested by me)

            https://community.home-assistant.io/t/sonoff-wemo-emulator/11001

            And the ITEAD link where they talk about their product working with Amazon

            https://www.itead.cc/sonoff-work-with-amazon-alexa

  56. Raspbian server... orange pi zero (ssh login)... error:
    [ ok ] Restarting ssh (via systemctl): ssh.service.
    ./script.sh: line 457: 1577 Killed whiptail --title "Installation Aborted" --msgbox "Cancelled as requested." 8 78
    pi@OrangePI:~$ /script.sh: line 457: 1577 Killed

  57. More details:

    + echo 'Dpkg::Progress-Fancy "1";'
    + sudo tee /etc/apt/apt.conf.d/99progressbar
    + echo 'APT::Color "1";'
    + sudo tee -a /etc/apt/apt.conf.d/99progressbar
    + username=user
    + userpass=password123
    + adminname=admin
    + adminpass=password123
    + newhostname=myboard
    + SECONDS=0
    + [[ RASPBIAN == *\R\A\S\P\B\I\A\N* ]]
    ++ whiptail --title 'Main Raspberry Pi Selection' --checklist '\nSelect items fo r your Pi as required then hit OK' 30 73 23 quiet 'Quiet(er) install - untick fo r lots of info ' ON prereq 'Install general pre-requisites ' ON phone 'Install o n Android Smartphone - see blog' OFF mosquitto 'Install Mosquitto' ON apache 'In stall Apache/PHP/SQLITE + PHPLITEADMIN ' ON nodejs 'Install NodeJS' ON nodered ' Install Node-Red' ON webmin 'Install Webmin' ON screen 'Install Screen' ON java 'Update Java' ON wiringpi 'Wiring Pi for the GPIO utility' OFF mpg123 'Install M PG123' ON modpass 'Mod USER and ADMIN passwords (password123)' ON phpsysinfo 'In stall PHPSYSYINFO' ON upgradenpm 'Upgrade NPM to latest version ' OFF addindex ' Add an index page and some CSS' ON passwords 'Update ROOT and PI user passwords' OFF installcu 'Install CU for serial VT100 Terminal' ON installmc 'Install MC+M CEDIT file manager + editor ' ON installjed 'Install JED file editor' OFF habri dge 'Install HA-bridge on port 82' OFF log2ram 'Install Log2RAM default 40Meg' O FF wolfram 'Remove Wolfram on a PI to save space' OFF office 'Remove LibreOffice on PI to save space' OFF
    + MYMENU=
    + [[ '' != *\q\u\i\e\t* ]]
    + AQUIET=
    + NQUIET=
    + [[ '' == '' ]]
    + whiptail --title 'Installation Aborted' --msgbox 'Cancelled as requested.' 8 7 8
    ./script.sh: line 457: 2052 Killed whiptail --title "Installat ion Aborted" --msgbox "Cancelled as requested." 8 78
    + exit

    1. don't have the smallest idea of what you did... lot of what you posted is not even in the script, sure not in latest... nowhere in script are comparison towards strings with all those asterisks... and raspbian is for original raspberry, not orange...

      1. The first post is the normal output, second one is debug output of bash script... The slashes are added by debug in the output.
        Raspbian is Raspbian server version downloaded from orange pi zero pages.

  58. Problems with the script!

    I downloaded the latest Raspbian and latest script and run into a big number of problems!

    First I was wrong with the answer to give the PI3 the name of myboard. There is something wrong with the name setting. I did not want to change the name to myboard. I ended up with an empty name field in the preferences. OK I wanted the name to be raspberrypi and changed that after installing Raspbian. Also chosing the options in the menu is confusing there is no mentioning how to choose options other than the hint on top of the menu to choose(how???) I made the mistake by pushing a return after the namesetting to and the script started running, which was not the intention. Choose with arrow keys and Tab or space , unclear especially when you are not doing this regularly.

    I get a lot of messages about email system not being alright, probably nothing because I do not use this but annoying!

    I was a bit attending the flow of the script and I have seen an error when installing Webmin. As a result I cannot login to Webmin. Can I install it afterwards?

    I backed up the files in .node-red from my flows and the database from my MySQLite data, but no flows appear in node-red. Then i tried stopping and starting node-red by node-red stop and node-red start but this gives errors because of a faulty package.json. I did that by overwriting it with the backup files . I only need to set back floss_*.json or also flows_*_cred.json?

    I cannot login to my Root account although I have given exact answers when asking for my username and passwords in the script.

    It does not seem to be so easy as the first time I ran the script on my Pi3.

    Have there been introduced problems because of all adaptions for other than Pi3(/2?)

    Are there any logs of the script to see if there were errors, I have no idea?

    Repeating the procedure is the best option I think!

    1. Situation update: 2nd time I did it alright and updated the ROOT an pi passwords which I unintentionally skipped the first time. Importing flows through clipboard by copying the JSON text from the flow* file. Only fill in username and password for the broker and we are going again.
      odd and even are there of course.

    2. space to select/deselect options, enter to confirm, easy peasy, you're the first to complain about this... webmin complained about missing user and password, probably... if you screw up things, don't try to patch a dead system, just restart flashing a clean distro... and if using latest testing version, you'll find logs in the same folder of the script...

  59. Ran the new and improved, experimental script last night and it ran without a hitch. Ran it over Real VNC, 2 hours and 19 minutes later, it was done.

    Nicely put together.

      1. unfortunately that distro is a mix of dietpi on top of armbian based on jessie, a bit of a mess... mosquitto is on version 1.3.4 so the problems in it not starting were related to websockets, in the end... other problem is that the mosquitto_passwd file works only interactively, so no scripting for that part... i could add a switch to check this specific version on this specific board, but is it worth it? If someone wants to go this route and use this dietpi, wait for the updated version Peter will publish and then you have to comment out the websockets part completely, so you end up with mosquitto passwordless but at least starting... too much of a hassle for now...

    1. Because 7 gave some other boards issues - and because I had no particular use for 7. Previous versions work just fine for the likes of PHPLiteAdmin etc... nothing to stop anyone upgrading...

            1. Hi
              Well it's probably very nice - but twice I tried - the first time nothing - the second time looked promising but no data... the point of this was? We know all of those work together? Or were you just proud of the site?

              1. Yes, they all work together.
                But if I had to do it now, I would develop it in another way.
                Probably with NGIX and skipping all the graphics.
                You know, putting an ssh file in the boot before booting and doing it from the beginning via ssh.
                Surely it is how you end up doing it.
                By the way, I saw on your profile that you spent some time in Spain.
                Do you understand Spanish?
                I develop in English with difficulties.

                1. I am indeed in Spain right now and for the rest of the summer - that is if we don't all die from the heatwave. Sorry can't help you with Spanish as my Spanish language is atrocious.

    2. because raspbian is just... OLD... still based on Jessie, and as far as debian is done (and raspbian on it), they have a release time of about 3 years... Stretch is out, let's hope in soon adoption, and support for the 64 bits specific of the rpi3... if you want php7 (which is faster than 5, but in our scenario is used only for phpliteadmin and little more, nothing vital to a iot setup), then go for an ubuntu 16.04 distro like armbian or dietpi, i think

        1. problems with github, not script... unfortunately their files are served in a round-robin fashion, or the like, and sometimes you end up in a server that's not aligned with the others and have missing files... that's what happens with that bcrypt file in your logs... i tried about 10 times to go to that url, 2 times i downloaded the file, 8 time i got a 404 page (not existing url)... so, just cross finger and retry...

          1. I thank you Dr....

            How do I then "reset" to start flawlessly from the top? eg, does not complain about directories, etc?

            Is there a way to know I have d/l the GOOD script?

            I will keep trying.
            Again, thank you.

  60. I see you made an update for The Script.
    As I think I am behind in updating my existing Pi3 and Raspbian Jessie I thought let's try that VM method to create a new image with the latest version.
    Then the question what OS to use! I got lost. You sort of advertised Raspbian Stretch, but when I read the comments of your masked friend that is not latest and best technology to use ?????????????? A little advice would be appreciated here.

    1. No advertising - simply mentioned that it exists (and hence WILL be with us) and so we modified the script accordingly. Which one you use is up to you. I'm upgrading my PIs. I just did the install on a Pi3 - had to modify a separate Grafana setup to get it to work but most other stuff seems just fine. Backed up the SD and now I have a Pi2 working on the same setup.

      1. So at last xenial is a little advice! So I take the desktop version as I am a linux illiterate. So I got this one: ubuntu-16.04.3-desktop-amd64.iso

      2. Sorry to say. In VMWare Workstation 12 I am stuck at the globe asking for timezone. The flatted globe does not fit into my little window "Where are you". I only see a part of a unactivated button and that's where I am no further actions possible ESC ctrl-c Tab nothing! What is the trick to get out of that situation? see also png
        There is a message which asks for Install Tools and that seems the only way out circumventing all following settings.

          1. Cannot resize window. Something to do with display adapters not yet initialised. I must update Tools. I will start from scratch again. I took away 3D feature.

  61. I followed creation of pi. HOwever there is a problem!
    >>>

    root@Ubuntu-VM:/home/hoekbrwr# adduser --quiet --disabled-password --shell /bin/bash --home /home/pi --gecos "User" pi echo "pi:password" | chpasswd usermod pi -g sudo echo "pi ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/pi chmod 0440 /etc/sudoers.d/pi
    chpasswd: invalid option -- 'g'

    <<<

    What is going on?
    I have created pi user as administrator through menu, but cannot let it login automatically. Slow boat to China....

    1. honestly, don't know what the h... are you doing... why create the pi user from menu, if script creates it itself? and why not just creating it during install... and, repeat, customizing the graphic part of ubuntu is beyond this blog, there are TONS of tutorials online on how to do... why are you running those lines by hand and not by script? Restart from scratch, do a textual install, install the script, then do whatever you want with gui, not relevant for the script functionalities

  62. Took away the invalid option -g., next problem

    >>>

    root@Ubuntu-VM:/home/hoekbrwr# adduser --quiet --disabled-password --shell /bin/bash --home /home/pi --gecos "User" pi echo "pi:password" | chpasswd usermod pi sudo echo "pi ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/pi chmod 0440 /etc/sudoers.d/pi
    adduser: Only one or two names allowed.

    <<<

    I think I already had to define the pi user during installation of the OS!

    That's it for today... Tomorrow more tries.
    It is not as easy as it looks. All the details in the text are useless because with every other OS installed things look different to the blog text.

    1. last comment, then i surrender: WHY ARE YOU RUNNING THOSE LINES BY HAND??? Instructions are CLEAR: login with whatever user you want, create a file, paste the script, save, give permissions, run it... if logged in user it's not pi it will be created, if it's pi it will be updated with needed permissions, then you have to login as pi and rerun the script and answer some VERY basic questions, PERIOD! DON'T OPEN THE SCRIPT, LEAVE IT THERE AS IF IT IS A BLACK BOX! RUN IT, NOTHING ELSE!

      1. Sorry but this is still in the blog with a lot of other stuff not useful because this was for one specific system accurate. I will run The Script and then it should be OK.

        1. You've lost me Leo - the script works well for a bunch of different systems - most of which are noted in the comments at the start. Don't deviate from simply running the script - or we can't help.

          1. NOTHING BIG WRONG WITH THE SCRIPT! Rest assured it is almost flawless. unattended in the VM in 18 minutes on my 6 year old PC(a bit pimped SSD 8GB RAM)
            I was following the instructions in the blog and it is not in sync with Ubuntu UI messages. When I start installing Ubuntu I am asked to make a user account and there I used my standard username. I think one should use pi as a user and set the pw at that point. Then the script will start without the extra round because there is a pi user. I tried this big command(found in your text) to create a pi user and I got all kind of strange errors. I am not a Linux guy and I tried to create the user with the User menu in the settings and that was not clear to me. Then Antonio got irritated by my complaints and I realised I had seen this earlier with another try that the script creates the pi user in the first round and then can be started and run to the end.
            The text just above Bash script just did not fit for Ubuntu. I think it was written with another linux version when you first made the blog for VM installation. All things very confusing root, pi user admin(later on in the script)
            Now I have run The Script and have a VM with Ubuntu.
            How should I produce an image for my pi3 from that? Convert .vmdk to .iso? Can I boot it then?

              1. I am lost. What is the purpose of running on a VM? I made a wrong conclusion it can be used to flashed to use as home controller on the tested hardware?

  63. Peter and Mr Shark, Thank you for the 'script' and the set up of the VM, I used virtual box. I had one issue and that was when I went to install Ubuntu I could only see 32 bit files despite me having a 64 bit Windows 10 machine. This problem is caused by hyper-V being disabled in the BIOS. There's quite a lot of information about that on 'Google' and once I found the correct location in the bios and enabled that functionality I could see the 64 bit files in the virtual box set. I think where Leo is getting confused, as was I, is that Peter's excellent video describes the Virtual box install whereas the blog text describes the VMWARE version, which of course it states. In my case once I had the issue of the missing 64 bit files solved I followed the video and everything installed perfectly and after installation everything runs perfectly. I put a BigTimer flow on the virtual machine and used MQTT to send the data to MQTT-spy running on my Windows 10 desktop PC, in fact the same physical machine that also has the virtual machine. Windows and Linux in harmony!
    Incidentally it 16 minutes to run the 'script' and load everything onto the virtual machine. A super job in getting the script to be so easy to use and so versatile. I am very new to Linux so it is a big help to install so much software easily.

      1. Now I can see how many wrong turns I took in my run on the VM installation. It is very nice and helpful. You have that so fast great work!

  64. Hi , I could not install the script in ubuntu-16.04.3-server VM.
    This is the error:
    E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
    E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? .Thank your help.

    Marcelo

  65. Hi Peter,

    Firstly, Heartfelt gratitude for sharing this wonderful piece of script. Works smoothly for everything. However, I am facing trouble accessing the MQTT broker. I am trying the access the MQTT broker running on my Raspberry pi (raspbian stretch) from my Node MCU. The connection can't be established. The Node MCU connects to the Wifi but returns an error (rc=5) connecting to the MQTT server. It says connection denied and I am not authorized to connect.

    I need some help. May be this is not the right blog but a little pointer in the right direction would be great.

    Thanks,
    Aritra

Leave a Reply

Your email address will not be published. Required fields are marked *