RPi Buster and RPI-Clone

BusterSee the later blog entry about RPI4 and Buster as issues I had here have been resolved.

No doubt there are many of you out there with Raspberry Pi Single Board Computers who have for some time, like me been using “rpi-clone” to successfully clone/backup their PI to known good SDs like Sandisk or Samsung SDs… maybe initially with Raspbian Stretch or even earlier versions… and who have now moved to the “Buster” upgrade as recommended by RPI. Well – there is (was) a potential problem using RPI-Clone if you took the upgrade (as against fresh start) approach as many of us needed to do in order to preserve months or years of work. Read on over the break…

I've now cloned my RPI systems more often than I could possibly count.

In creating the  clone, the procedure varies depending on whether you are cloning to a new SD/USB or one previously used on the same (partition) setup.  When making a new clone where the destination was not previously initialised with the same setup, it is necessary to reboot  then use the -f for FORCE option in RPI-CLONE. That option forces resizing of the ROOT (file) partition to handle differing SD sizes (larger and smaller) and it does it well and is only needed for SDs freshly introduced. Up to now this has worked as the size of the BOOT partition (small) has been consistent but now on BUSTER it has grown. If you attempt to clone from an original which was upgraded from Stretch without expanding the BOOT partition, then the clone may well fail. At the time of writing this blog entry, RPI-clone does not know about changing the size of the BOOT partition. By September 2019 this had been resolved so see the later blog entry who's permalink is here.

So, my 16GB original SD had an upgrade to Buster on it but without a change to the BOOT partition size. Good enough to run BUSTER but NOT a large enough BOOT partition to handle the cloning procedure as well. I initially thought that earlier (upgraded Buster) clones using -f were forcing the new SD to resize the (FAT) BOOT sector to the larger BUSTER standard of 120MB, as against the earlier 44MB of Stretch, in fact the clones STILL had only 44 meg BOOT sectors.

As RPI-Clone did not (at the time of this blog entry) handle the boot partition resizing, I ended up putting my original (well, a clone) clone into a new RPI4 USB socket while running the RPI4 from a new Buster installation on SD. I had to manually add (using the RPI terminal program – easy aside from the pain of attaching a screen and keyboard to the RPI4) GPARTED to the SD installation then running that with my SD in it’s USB adaptor – and CAREFULLY using GPARTED to firstly take off 200 MEG from the end of partition 2, then moving P2 right by the same amount. I then expanded P1 from 44MB to around 130MB without shifting it’s left position.  I then saved that new modified Buster clone, powered off, put my SD into the PI4 SD slot and powered up. From there I could make a NEW first clone using the RPI-clone –f option. Having now prepared a clone with the right BOOT partition size I could now make a test change to my original clone and use the second clone to re-clone WITHOUT a reboot or –f option.

Here in more detail from the start with a desktiop RPI on a virgin Buster…. in the terminal…

  1. sudo apt-get install -y gparted
  2. sudo gparted

I then put into the USB socket a working upgrade clone – which had the partition SDA1 at 44236 blocks size (1k SD blocks hence 44 MB BOOT partition)

and told GPARTED to process SDA rather than it’s working operating system on mmcblk0

I then used the PARTITION – UNMOUNT command to remove the little keys from the partitions, i.e. to ensure the SDA partitions were not mounted, After that I shrunk SDA partition 2 (SDA2) by a roughly a couple of hundred meg and shifted it’s start position right accordingly.  I moved the right side of SDA1 right to increase SDA1 to fill up the gap without shifting it’s left side.

When I set this off going, SDA2 resized, took about half an hour, but when it came to SDA1, it failed, maybe I was supposed to leave a gap or maybe I inadvertently shifted the left of SDA1. I tried resizing SDA1 again (very quick) and this time ensured I didn’t move the left side AT ALL and reduced the size of SDA1 to 134 Meg to leave a gap. When I set it going, everything just worked. i.e. I turned off the Pi4, swapped SDs and the newly size-changed SD with all my data still intact just worked.

I rebooted and made a clone of the clone using –f option. So now I am able able to use THAT clone to make another….. I powered off, swapped SDs, cloned again with no –f option…. this would be quick, but I’d rebooted. 5 minutes later I had my clone – but could I do it again, this time without the –f option again AND without without a reboot or –f option? YES.

I now had a complete upgrade from Stretch, using Buster and able to clone, without losing any of my data or setup.

there is a lot of arrogant and often incorrect or out-dated nonsense out there about backing up/cloning Raspberry Pi. My subscribers deserve better. The informatiuon you see here is how I handled the boot partition resizing until BillW (rpi-clone) added the option to handle this to rpi-clone itself.

Having now read the RPI-CLONE README properly (not easy) it turns out there is an –s option you can put on the far right of the rpi-clone command to alter /etc/hosts and /etc/hostname if you want to change the hostname of your clone/copy. Please note that an alternamive way to do this - nmtui - does NOT update the hosts file.

In coming to the conclusions above, I tried the commands suggested below: on my Pi3 with a known good SD which only the previous day had made a “good” backup on that same Pi3:

df –i

and the result of the test was:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 971040 268781 702259 28% /
devtmpfs 123728 379 123349 1% /dev
tmpfs 124880 10 124870 1% /dev/shm
tmpfs 124880 556 124324 1% /run
tmpfs 124880 3 124877 1% /run/lock
tmpfs 124880 14 124866 1% /sys/fs/cgroup
/dev/mmcblk0p1 0 0 0 - /boot
tmpfs 124880 13 124867 1% /run/user/997
tmpfs 124880 13 124867 1% /run/user/1000

which seemed fine – then this test:

sudo tune2fs -l /dev/sda

Second test failed (for reference) as it looks like you can only test partitions, not the complete SDA.

SO I completely shut down my PI (sudo shutdown now) then turned off then turned back on…

I am of course doing all of this via my PC, using Mobaxterm in an ssh session with the headless Pi as usual…

In case anyone thinks this means a lot of typing... here’s my latest list of aliases… in /etc/bash.bashrc go to /etc and edit with “sudo nano bash.bashrc” - this lot are on the end of the file.

alias mqttlog='tail -f /var/log/mosquitto/mosquitto.log | ccze -A'
alias nrlog='sudo journalctl -f -n 50 -u nodered -o cat | ccze -A'
alias space='df -h|grep -v udev|grep -v tmpfs|grep -v run'
alias cls='python /home/pi/cls.py'
alias stop='sudo shutdown now'
alias boot='sudo reboot'
alias clone='sudo rpi-clone -U sda'
alias cleanclone='sudo rpi-clone -f -U sda'

I’m about to change clone and cleanclone to add the –s flag on the right with new manually added hostnames just as soon as I figure out how to add arguments to aliases.

When I make clones now I see, using my PARTITIONS alias…

pi:~:09:13[0]> partitions
major minor  #blocks  name

1        0       4096 ram0
1        1       4096 ram1
1        2       4096 ram2
1        3       4096 ram3
1        4       4096 ram4
1        5       4096 ram5
1        6       4096 ram6
1        7       4096 ram7
1        8       4096 ram8
1        9       4096 ram9
1       10       4096 ram10
1       11       4096 ram11
1       12       4096 ram12
1       13       4096 ram13
1       14       4096 ram14
1       15       4096 ram15
179        0   15558144 mmcblk0
179        1     131072 mmcblk0p1
179        2   15261184 mmcblk0p2
8        0   31260672 sda
8        1     131072 sda1
8        2   30963712 sda2

See mmcblk0p1 and sda1 – at 131072 – well, under STRETCH those blocks WERE 44,000 (or 44MB) or thereabouts, no good for Buster and I kept cloning the OLD boot size to the NEW copy – NOT big enough. Not any more.

Facebooktwitterpinterestlinkedin

25 thoughts on “RPi Buster and RPI-Clone

  1. Pete,

    Next time you see the problem, try doing a “df -i” and “tune2fs -l /dev/XXXX” for that specific device both before and after the reboot, saving the results somewhere permanent so that you can compare the results.

    You’re probably most interested in the “Free Blocks” and “Free Inodes” counts, but possibly the reserved block counts, too.

    It sounds as though your backups are somehow creating too many fragments and running the filesystem out of available inodes (although I haven’t ever come across any back-up system that writes to the filesystem it’s backing up from).

    It might also be worth going to the Buster release notes and checking whether the filesystem defaults changed.

    -John-

    1. Right – BEFORE: I did the first one.

      Filesystem Inodes IUsed IFree IUse% Mounted on
      /dev/root 971040 268781 702259 28% /
      devtmpfs 123728 379 123349 1% /dev
      tmpfs 124880 10 124870 1% /dev/shm
      tmpfs 124880 556 124324 1% /run
      tmpfs 124880 3 124877 1% /run/lock
      tmpfs 124880 14 124866 1% /sys/fs/cgroup
      /dev/mmcblk0p1 0 0 0 – /boot
      tmpfs 124880 13 124867 1% /run/user/997
      tmpfs 124880 13 124867 1% /run/user/1000

      The second one – you didn’t mention it needs SUDO so I prefixed it with sudo.

      sudo tune2fs -l /dev/sda

      and the result was:

      tune2fs 1.44.5 (15-Dec-2018)
      tune2fs: Bad magic number in super-block while trying to open /dev/sda
      Found a dos partition table in /dev/sda

      But that was a GOOD backup SD? Help… That was an SD I’d used last night and made a “successful” backup with.

      Surely it WOULD find a DOS partition (first one) on a PI…..

      1. Pete,

        Yup, tune2fs will only work with Unix filesystems (partition level, ie:- /dev/sda1 not device level, so not /dev/sda) and definitely not DOS.

        The inode problem only applies to Unix filesystems, so it looks like you’ve got something else going on there.

        -John-

        1. Well, for sure, both SDA1 and 2 are created by the PI…. as you saw, second time around, the SAME SD made what seems like a perfect copy. The only differnce being, this time I used -f and did a power cycle. Occasionally, the power cycle is not needed, sometimes, the -F is not needed and more rarely none of that helps. That last one might be a red herring. VERY occsionally under STRETCH I would get an unrecoverable SD, most of the time it all just worked. I’m sure there is something else here I and others are missing.

  2. This article was a gift only just for the fact it mentioned MobaXterm. I was just in the process of rebuilding my W10 system after a crash. I did not know that app and it is a sort of Swiss knife for remote controlling your PI.

  3. Hello Pete

    Here is info from my RPI4 buster light sd card
    might be of help (Ibelieve their is no such thing as useless info)

    pi@rpi4buster:~ $ lsblk
    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    mmcblk0 179:0 0 29.7G 0 disk
    ├─mmcblk0p1 179:1 0 256M 0 part /boot
    └─mmcblk0p2 179:2 0 29.5G 0 part /

    pi@rpi4buster:~ $ sudo fdisk -l /dev/mmcblk0
    Disk /dev/mmcblk0: 29.7 GiB, 31914983424 bytes, 62333952 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0x91da3615

    Device Boot Start End Sectors Size Id Type
    /dev/mmcblk0p1 8192 532480 524289 256M c W95 FAT32 (LBA)
    /dev/mmcblk0p2 540672 62333951 61793280 29.5G 83 Linux

    As to the hospital they stuck a catheter into my main artery at the groin and went up to my heart to have a look (in full colour display).

    Not something I would like to repeat!

    regards Brian

    1. Far from useless info Brian… I was always led to belive that PIs used 4k sectors until I realised mine all use 1K sectors (probably as I’ve upgraded and upgraded)… and yours appears to be using 512 byte sectors… now my interest is up, I wonder for a PI in typical control system (non-desktop) use, what the ultimate sector size would be…. and I never did get to the bottom of the exact start location and size of the boot partitionj should be for Buster…. and what if any gap is actually needed between boot and root…. What I have works very well, but it would be nice to be precise – if GPARTED will actually let you change destination sector size as well (I just changed the size of the boot part after moving up and shrinking the root part) then RBOOT handling the clone/copy without losing source data. All sounds a little tricky.

      I’m glad to say I had nothing invasive today unless you call experimenting on the brain with flashing lights invasive.

  4. hello Pete

    The sd card sandisk 32gb brand new

    the interesting bit is boot partition

    /dev/mmcblk0p1 8192 532480 524289 256M c W95 FAT32 (LBA)

    start sector 8192

    end sector 532480

    size 256M

    I think you can use this I cant as I dont know enough about Linux

    This was a virgin card no cloning
    buster lite latest
    script run no errors
    Flows exported from my original pi3 as json
    imported to programm

    regards Brian

  5. hello Pete

    forgot to mention the gap to next partition perhaps the hospital can look at
    my brain (if Iv got one) next time!

    /dev/mmcblk0p2 540672 62333951 61793280 29.5G 83 Linux

    next partition starts at sector 540672

    I do regard flashing lights as invasive they give me migraine

    regards Brian

  6. Brian – I dont understand why your BOOT needs to be 256 meg, even Buster apparently only needs 128 meg. So, I’m seing the start at 8192… mine is that, but is that bytes or blocks and if blocks, what size? And is there a reason for the gap between P1 and 2?

  7. Hello Pete

    Good questions exactly what I thought.
    I can ask for bytes thus lsblk -b

    I get on a Virgin SD card with busterlite dated 10/7/2019 burnt by etcher
    lsblk -b the -b forces byte info

    pi@raspberrypi:~ $ lsblk -b
    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    mmcblk0 179:0 0 31914983424 0 disk
    ├─mmcblk0p1 179:1 0 268435968 0 part /boot
    └─mmcblk0p2 179:2 0 31638159360 0 part /
    pi@raspberrypi:~ $

    boot partition is 268,435,968 bytes
    !! maybe something major with booting is planned!!
    There is a gap to the start of the work partition.

    if you enter fdisk -l
    at the end you get these are sectors of 512KB

    Device Boot Start End Sectors Size Id Type
    /dev/mmcblk0p1 8192 532480 524289 256M c W95 FAT32 (LBA)
    /dev/mmcblk0p2 540672 62333951 61793280 29.5G 83 Linux
    pi@raspberrypi:~ $

    If you calculate 532480 -8192 = 524289 sectors of 512KB for boot
    524289 * 512 = 268435968 bytes in boot partition

    This agrees with the figure for bytes from putting in lsblk -b as above

    hope this helps

    regards Brian

  8. Hello Pete
    https://www.raspberrypi.org/blog/buster-the-new-version-of-raspbian/

    PhilE says:
    25th Jun 2019 at 12:50 pm

    Provided there are no unnecessary files on your card, an old 56MB /boot FAT partition should be sufficient for an upgrade, but we would strongly recommend moving to the new 256MB FAT partition when possible.
    Reply

    spotted this official boot is 256MB

    so start sector end sector is correct as is the gap to next partition

    regards Brian

  9. hello Pete

    rpi-clone buster to buster

    busterlite to new sd 32Gb card useing etcher

    rpi-clone sd 32gb to this card.

    The running prog buster rpi 4 all flows nginx

    NOT upgrade from Stretch

    pi@rpi4busternginx:~ $ sudo mount /dev/sda1 /media/usb

    pi@rpi4busternginx:~ $ lsblk -b
    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    sda 8:0 1 31914983424 0 disk
    ├─sda1 8:1 1 268435968 0 part
    └─sda2 8:2 1 1920991232 0 part
    mmcblk0 179:0 0 31914983424 0 disk
    ├─mmcblk0p1 179:1 0 268435968 0 part /boot
    └─mmcblk0p2 179:2 0 31638159360 0 part /

    Boot partition is 256Mb on both

    pi@rpi4busternginx:~ $ sudo rpi-clone sda -f -U

    Destination disk partition /dev/sda1 is mounted on /media/usb.
    The clone cannot proceed unless it is unmounted.

    Booted disk: mmcblk0 31.9GB Destination disk: sda 31.9GB
    —————————————————————————
    Part Size FS Label Part Size FS Label
    1 /boot 268.3MB fat32 — 1 268.3MB fat32 —
    2 root 31.6GB ext4 rootfs 2 1.9GB ext4 rootfs
    —————————————————————————
    == Initialize: IMAGE mmcblk0 partition table to sda – forced by option ==
    1 /boot (40.4MB used) : IMAGE to sda1 FSCK
    2 root (3.1GB used) : RESIZE(31.6GB) MKFS SYNC to sda2
    —————————————————————————
    Run setup script : no
    Verbose mode : no
    ———————–:
    ** WARNING ** : All destination disk sda data will be overwritten!
    : The partition structure will be imaged from mmcblk0.
    ———————–:

    Initializing
    Imaging past the start of /boot partition 2.
    => dd if=/dev/mmcblk0 of=/dev/sda bs=1M count=268 …
    Resizing last partition to end of disk …
    Resize success.
    Changing destination Disk ID …
    Delaying so partprobe can update /dev entries …
    => fsck -p /dev/sda1 …
    => mkfs -t ext4 /dev/sda2 …

    Syncing file systems (can take a long time)
    Syncing mounted partitions:
    Mounting /dev/sda2 on /mnt/clone
    => rsync // /mnt/clone with-root-excludes …
    Mounting /dev/sda1 on /mnt/clone/boot
    => rsync /boot/ /mnt/clone/boot …

    Editing /mnt/clone/boot/cmdline.txt PARTUUID to use 779bc80b
    Editing /mnt/clone/etc/fstab PARTUUID to use 779bc80b
    ===============================
    Done with clone to /dev/sda
    Start – 22:58:26 End – 23:02:20 Elapsed Time – 3:54
    unmounting /mnt/clone/boot
    unmounting /mnt/clone
    ===============================

    pi@rpi4busternginx:~ $

    Note the time taken 3mins 54secs

    tested cloned sd card all appears to be running ok

    regards Brian

  10. hello Pete

    just noticed I didnt comment on the gap between boot and data partition

    on buster in my comments Aug 7th to Aug 11

    The gap is also 8192 sectors of 512KB = 256MB

    This is from the busterlite official img burnt on Etcher

    Reagards Brian

  11. hello Pete

    I havent the foggeist idea why,but it is the same

    as the gap from the start of sd card to the boot partition.

    This is with no changes by me. Just the busterlite official img

    from the raspi site

    Raspbian Buster Lite

    Minimal image based on Debian Buster

    Version: July 2019

    Release date: 2019-07-10

    Kernel version: 4.19

    Size: 426 MB

    The image burnt by Etcher

    Its irritating not to know why!, and suspicious

    that both gaps are the same size.

    regards Brian

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.