RPI4 Network Lesson

RJ45 Network

This blog entry was last updated mid-September 2019 and covers a wide range of issues I’ve had and resolved since the introduction of the Raspberry Pi 4 single-board computer (SBC) and the “Buster” update to the (Linux Debian-based) Raspbian operating system, in particular in setting up the networking and hostname which has changed somewhat since STRETCH on the RPi-3. Enjoy, technical readers using the RPi may find this very useful...

On the subject of rpi-clone

RPI-CloneI recently had an update From BillW of rpi-clone “fame”. I use this tool several times a week on my Raspberry PIs before doing updates – works a treat to make seamless clones of working systems – and despite concerns of some I’ve never knowingly lost data. However, the move to RPi-4/Buster caused some concerns as rpi-clone does not (did not) resize the boot partition.. and the new operating system version uses a larger boot partition than predecessors.

I have tested the latest revision of rpi-clone (https://github.com/billw2/rpi-clone) numbered 2.0.22, changes made only recently to add a “–p” option (without the quotes) – this will clone an RPi-3 SD and increase the boot partition to 256M, the standard for RPi-4.  I’m now using that size for RPi-2, RPi-3 and RPi-4

I use the unattended flag (-U) and in order to do the boot resize you need to do a complete copy (slow) the first time to force the 256M boot partition.

sudo rpi-clone -f -p 256M -U sda

If you wish to set up a new hostname while you are there, then add the –s option followed by a hostname. For example:

sudo rpi-clone -f -p 256M -s myclone -U sda

Now then – you MAY think it is just as easy to use the “sudo nmtui” command line utility to change the host name – WRONG – as I just found out – that utility changes only the /etc/hostname file, NOT the /etc/hosts file (don’t ask why). Using nmtui to do the job, next time you reboot after a hostname change – you’ll find a complaint on the command line.  Rpi-clone to the rescue as it updates BOTH /etc/hostname and /etc/hosts file.

When doing a “normal clone” it all gets simpler (and faster)

sudo rpi-clone -U sda

All this of course is assuming you are making a clone on SDA, typically an SD or USB stick mounted in a USB socket on the RPi..  I already cloned an RPi-3 using the new rpi-clone update – to a USB stick – and noted that the boot sector was indeed enlarged. I’ll come back on this when I’ve done the same to an SD as you cannot currently boot an RPi-4 from USB. I guess I should have done an SD backup first really.

On the utility “nmtui” and cloning

I’ve now realised that the hostname of a cloned pi should preferably be set in rpi-clone – not nmtui – as yesterday I had an issue with an RPi4 clone – I wanted to change the static IP address I’d originally set with nmtui, as the new clone was to be used on an RPi-3.

I noted on fitting the SD into the destination RPi-3, that “nmtui” had put in a temporary IP address as the original address was not working – turns out that for some reason just under the fixed IP address field, somehow the MAC address of the original board had been inserted… which rendered the entry unusable on a new board, hence the temporary address. After some head-scratching. using nmtui I removed the MAC address and saved, without removing the temporary entry. By the time I rebooted, all was well.

End of Mid-September update

In the past, I’ve set up countless RPi-2 and 3 SBCs over time, usually headless and using the file /etc/dhcpcd.conf to set up a static IP address, It worked every time. Continue reading for some hopefully useful info…

Raspberry Pi 4 upgraded from a working Stretch to BusterWhen I got my new RPi-4, I upgraded all my older systems to BUSTER, not one of the easiest things to do and indeed not recommended on the RPI-forums. (Use a clean installation, they say. GREAT unless you happen to have years’ worth of complex, only partly documented code on your RPi-2 or 3).

Well, I did upgrade, the change is all documented on the blog – and of course when it came to setting a static IP address I never thought twice about it. I used my Buster-upgraded installation several times on the RPi-4 without issue.

NetworkThat was fine until recently when, frustrated with a few packages of little relevance being “held back” by “sudo apt-get upgrade” or to use the commonly accepted abbreviation “sudo apt upgrade”.

It turns out that if you use “sudo apt dist-upgrade” you get all the packages available. Following that by “sudo apt autoclean” gets rid of any old versions lying around and you’re all set.

I checked out two of my RPi-3 boards with this thoroughly up to date setup and all was well until the first time I tried this on the RPi-4 – no IP address. I don’t use the graphical desktop and hence could not even see if the board was working.

Reluctantly I fitted a screen, keyboard and mouse to the RPi-4 – and as user PI at a command prompt I could then play with the board (like I have nothing better to do). But nothing I did would bring up the Ethernet connection. I even checked /etc/network/interfaces as recommended by one chap on the web… No.

Then my friend Antonio reminded me that “the script” has a couple of commented out lines that allow for permanent setting of graphical mode and vice-versa. I put the RPi-4 into graphical mode and reset. That helped but still no Ethernet. In the setup on the graphical desktop you can set a fixed IP address. I tried that – that seemed to work – but setting up by the desktop is not ideal for me.

The dist-upgrade seemed to have enabled WICD – despite (according to Steve Lenehan in here) it being no longer supported. That being the case I was happy to stick with dhcpcd.conf – I promptly disabled then totally removed WICD. Antonio (Mr Shark concurs that WICD is now old hat and is not supported.

Solved:

A file called “/etc/wicd/wired-settings.conf” – that existed in my older images but with  IP etc set to “none” – Antonio thought that it might override “/etc/dhcpcd.conf” – well, it took over from dhcpcd.conf  - I got rid of it when I removed the /etc/wicd folder and all was well on my new RPi-4 except that…. in /etc/dhcpcd.conf, I gave the RPi the fixed address of 192.168.1.30 – that worked a treat, but sadly so did 192.168.1.20 – an address I could find no reference to in any file in /etc/ – yet there it was… I could SSH into either address.

According to the graphical desktop I’ve disabled WIFI so that wasn’t it. Turns out (thank you Antonio for spotting this) that while my manual update of /etc/dhcpcd.conf was of course working and setting up the static address as it always has, with the combination of RPi-4 and Buster somehow, nmtui (Network Manager Text User Interface) had silently added in the second address (192.168.1.20) on DHCP. For now I’ve disabled Network Manager but it looks like the real answer, as nmtui has been around for a long time, would seem to be to comment out any manual mods to /etc/dhcpcd.conf and set a static address (192.168.1.30) in nmtui.

That didn’t work well at least using an eth0 entry. What DID work well was to put a manual address into /etc/dhcpcd.conf as has always worked and killing Network Manager (once only) like this:

sudo systemctl disable network-manager

sudo apt remove -purge network-manager

sudo rm -rf /etc/network-manager

I did this remotely using Mobaxterm on my PC, after which a reboot and all was wonderful.

Ok, as an alternative this seems to work and can be done from SSH – i.e. remotely.

With Network Manager un-installed - and running on dhcp.conf, I installed Network Manager (sudo apt install network-manager) – I used nmtui to set up eth0 with a static 192.168.1.30 address…. and then:

sudo systemctl disable dhcpcd.service

sudo reboot

That STILL left me with 30 and 20 (keep up) so THIS time I deleted eth0, left “Wired Connection 1” in but made it static 192.168.1.30, then booted. All works a treat, no more 192.168.1.20

Unbelievable. So now I have Buster on RPi-4, upgraded from Stretch without starting from scratch and it seems, all working well.

From the same program, while setting up your IP address, you can set up the hostname OR you can do that during a clone.

Incidentally, those two lines selecting command line or desktop, commented out… below..

# you may want to use these on an RPi or elsewhere to force either
# a graphical or command line environment
#sudo systemctl set-default multi-user.target
#sudo systemctl set-default graphical.target

Ways of changing network settings

According to Mr Shark:

EducationThere are at least 5 different methods of setting up networking in Linux, each has its own config file, each can manage static AND dynamic ip, each is independent and can cause the collisions we’ve seen above... Let’s try to clarify, I'll just list them with basic details, and give proper online GOOD documentation... a RTFM hasn’t ever harmed anyone...

Choose ONE of these, and be sure the others are disabled, not with just lines commented out in their config file... we had issues because of commenting out lines in /etc/dhcpcd.conf file while using Network Manager... but commenting out lines in that file will just revert its daemon to its standard behaviour, which is requesting IPs via dhcp... so the double ones, fixed by nmtui, and dynamic by dhcpcd...

In double quotes the service/daemon name, which you should enable/disable and/or start/stop via the usual "sudo systemctl enable|disable|start|stop|status SERVICENAME" command using:

1) the good old "networking" service.
It is configured via the /etc/network/interfaces config file, info: https://wiki.debian.org/NetworkConfiguration

2) the "wicd" service, which was installed by previous (to Buster) Raspbian editions if you used the GUI version, and deprecated by Buster (you end up having it only if you upgrade from previous editions and did not follow full instructions on Raspberry site to remove it and other stuff after upgrading...) This is configured via the /etc/wicd/*.conf config folder/files, info: https://wiki.archlinux.org/index.php/Wicd

3) the usual "dhcpcd" Raspbian suggested method which is configured via the /etc/dhcpcd.conf file, info: https://wiki.archlinux.org/index.php/Dhcpcd

4) the almost standard method in every modern distribution, "network-manager" - it is configured via various tools, in console you can use "sudo nmtui" (Network Manager Text User Interface), very easy to do, info: https://wiki.archlinux.org/index.php/NetworkManager

5) the newer method, the "netplan" way uses the format which you'll find in MANY other services today (Docker and Home Assistant in primis), which is the YAML file format, that implies you format and indent the file in a PRECISELY spaced style, otherwise you'll get errors...
managed via the /etc/netplan/* files, info: https://help.ubuntu.com/lts/serverguide/network-configuration.html

My own preference after experiencing the alternatives is to use (4) nmtui and I always use a static IP address for the Raspberry Pi, mainly because I cannot get ESP8266s to talk to my MQTT broker based on host name.

We hope this is clear. Now, and if you want more details, as suggested above, RTFM.

Facebooktwitterpinterestlinkedin

20 thoughts on “RPI4 Network Lesson

  1. a huge thanks for putting this in writing here Pete, and sorry to read you learned about it the hard way. at least it will save some folks the headaches.
    i consider myself an experienced unix/linux admin, but this is also new to me.

    i do read release updates etc, but it gets harder to keep up with the swelling releases/versions

    so a big thank you sir!

    i prefer though to let my ubiquity router handle the dhcp fixed ip’s, easier to maintain a single list, but that is just personal preference i suppose

  2. Thanks Peter, for slogging it out for us all. Had a similar problem with Octoprint a while ago. They had mored where to put the static IP address. Update need uniformly across the linux spectrum.

  3. I have tripped and fallen on my nose a couple of times due to the 5 different methods. I have always managed to make it work, but I’m never certain I’ve done it the best way or even completely right. This post will be bookmarked so I can re-read it when next I have network problems with a Pi. Thanks.

    PS: I don’t regard RTFM as aggressive, it’s solidly good advice that I give to people on an almost daily basis in my day job. I just wish people would actually read the fine manual…

  4. Hello Pete

    just feedback info.

    Just did a new buster lite sd card etcher 10/07/2019. script run no problems

    Ran Mr Sharks Nginx script no problem except pear not found

    Now overwriting my old cards boot(45Mb) with rpi-clone .

    Is a newer version 2.0.22

    #!/bin/bash

    # rpi-clone is Copyright (c) 2018-2019 Bill Wilson
    #
    # Redistribution and use in source and binary forms, with or without
    # modification, are permitted under the conditions of the BSD LICENSE file at
    # the rpi-clone github source repository:
    # https://github.com/billw2/rpi-clone

    version=2.0.22

    Clone my card to usb stick

    pi@rpi4buster:~ $ sudo rpi-clone sda -f -U

    Booted disk: mmcblk0 31.9GB Destination disk: sda 30.8GB
    —————————————————————————
    Part Size FS Label Part Size FS Label
    1 /boot 256.0M fat32 — 1 43.5M fat32 —
    2 root 29.5G ext4 — 2 28.6G ext4 —
    —————————————————————————
    == Initialize: IMAGE partition table – forced by option ==
    1 /boot (39.0M used) : MKFS SYNC to sda1
    2 root (2.9G used) : RESIZE MKFS SYNC to sda2
    —————————————————————————
    Run setup script : no.
    Verbose mode : no.
    ———————–:
    ** WARNING ** : All destination disk sda data will be overwritten!
    ———————–:

    Initializing
    Imaging past partition 1 start.
    => dd if=/dev/mmcblk0 of=/dev/sda bs=1M count=8 …
    Resizing destination disk last partition …
    Resize success.
    Changing destination Disk ID …
    => mkfs -t vfat -F 32 /dev/sda1 …
    => mkfs -t ext4 /dev/sda2 …

    Syncing file systems (can take a long time)
    Syncing mounted partitions:
    Mounting /dev/sda2 on /mnt/clone
    => rsync // /mnt/clone with-root-excludes …
    Mounting /dev/sda1 on /mnt/clone/boot
    => rsync /boot/ /mnt/clone/boot …

    Editing /mnt/clone/boot/cmdline.txt PARTUUID to use 33ac1de7
    Editing /mnt/clone/etc/fstab PARTUUID to use 33ac1de7
    ===============================
    Done with clone to /dev/sda
    Start – 22:08:45 End – 22:13:32 Elapsed Time – 4:47
    unmounting /mnt/clone/boot
    unmounting /mnt/clone
    ===============================

    The usb stick sda has a boot of 45Mb at start of clone

    at end of clone both have 256Mb boot. I didnt use -p command

    sudo fdisk -l
    Device Boot Start End Sectors Size Id Type
    /dev/mmcblk0p1 8192 532480 524289 256M c W95 FAT32 (LBA)
    /dev/mmcblk0p2 540672 62333951 61793280 29.5G 83 Linux

    Disk /dev/sda: 28.7 GiB, 30752000000 bytes, 60062500 sectors
    Disk model: Ultra Fit
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0x33ac1de7

    Device Boot Start End Sectors Size Id Type
    /dev/sda1 8192 532480 524289 256M c W95 FAT32 (LBA)
    /dev/sda2 540672 60062499 59521828 28.4G 83 Linux

    I repeated with second card with the same result

    regards Brian

      1. As long as he leaves me here. I have no desire to have anything to do with the UK but – needs must. Have to go back over the winter.

        lol, good thing that Boris wil put an end to tha soon then. #spain4Us
        😛

  5. Hello Pete

    my fault I have a problem with eyesight, sometimes having to

    read a word at a time with a looper.

    Your point was plainly made.

    Alex point on Brexit

    press release

    Millions of Germans and other continental Europeans will settle down in front of their TV sets tonight for what has become a New Year’s Eve ritual – the showing of an aged British comedy sketch that is unknown in the UK.

    Norddeutscher Rundfunk (NDR), the Hamburg-based channel which first recorded and broadcast Dinner for One in 1963, will show it no less than five times during the course of the evening in various versions

    They also did this every night for the coverage of the house of commons comedy with Mrs May. I think they think its a sequel to Freddie Frinton

    I can not fault the judgement.

    regards Brian

  6. Hi,
    thankyou for the info you provided here.
    I have an issue on RPI4: the network speed.

    The sfpt tranfer of a 250mb file in the micro SD of both
    the raspy is very slow:
    RPI3 (+/-) 650kb/sec
    RPI4 (+/-) 400kb/sec !!!

    I installed the RPI4 OS from the latest headless image
    as in past I made for RPI3+ (the RPI3 is used as NAS,
    and I bought the RPI4 to improve the file transfer speed)

    I have both RPI3 + usb3-to-GBE dongle (of course limited in speed)
    and RPI4 connected to the router (GBE interface) and the SD of
    the RPI4 is brand new, same type (ultra) and was verified.

    If I use only wireless interface on the RPI4, at a distance of 20cm
    from the router, the network speed decreases further (…)

    Do you have any Idea of the cause for theese poor performances
    in RPI4 networking?

    Thankyou in advance.

    Lore

    1. wifi at 20cm from router is TOO much near, you risk to have bad performances, indeed… but overall EVERY raspberry is bad in nas setup… they can say whatever they want, but just because you can does not mean you should (use it as a nas)… an odroid hc1 is a way better alternative, i can saturate gigabit bandwidth using an hybrid WD 2.5 hdd, which is cheap and has both performances (useless to go beyond what you can reach via ethernet if gigabit, an ssd would be just wasted…) and size… https://www.hardkernel.com/shop/odroid-hc1-home-cloud-one/

      1. Hi, thankyou for the answers. But any Idea about the poor performances in RPI4 troughtput?
        DrFragle and MrShark, 20 cm is the shortest distance I tried, but I tried with one meter far too, without obstacles.
        I will use ethernet only, i any case.
        Please let me know if you get in mind something or other info.
        I will try asap to test the connectivity with iperf3 too…

        Thanks

        Lore

  7. I have been banging my head against the wall with rpi-clone. Should be as simple as “sudo rpi-clone –U sda” when i want to clone an exact image right? I am trying to clone form a 32gb to a 64gb microsd. For some reason it just isnt working for me. I have Ubuntu Server 18.04 installed on RP3b+ and have a usb to microsd adapter plugged in. Everytime I image and test in another pi, I get ” Failed to start Remount Root and Kernel File System” when services are booting up and it all basically falls apart form there. The clone works clearly as ubuntu is installed on the new SD Card, but it doesnt work properly. Can anyone help? Love to get this sorted and working.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.