See the later blog entry about RPi-4 and Buster as issues I had here have been resolved.
No doubt there are many of you out there with Raspberry Pi single-board Computers who have for some time, like me been using “rpi-clone” to successfully clone/backup their Pi to known good SDs like Sandisk or Samsung SDs… maybe initially with Raspbian Stretch or even earlier versions… and who have now moved to the “Buster” upgrade or beyond as recommended by RPI. Well – there is (was) a potential problem using RPI-Clone if you took the upgrade (as against fresh start) approach as many of us needed to do in order to preserve months or years of work. Read on over the break…
I’ve now cloned my RPi systems more often than I could possibly count.
Here is how, for example my THE SCRIPT grabs rpi-clone:
cd sudo git clone https://github.com/billw2/rpi-clone.git cd rpi-clone sudo cp rpi-clone /usr/local/sbin cd sudo rm -r rpi-clone
In creating the clone, the procedure varies depending on whether you are cloning to a new SD/SSD/USB or one previously used on the same (partition) setup. When making a new clone where the destination was not previously initialised with the same setup, it is necessary to reboot then use the -f for FORCE option in RPi-CLONE. That option forces resizing of the ROOT (file) partition to handle differing SD/SSD sizes (larger and smaller) and it does it well and is only needed for SD/SSDs freshly introduced. Up to now this has worked as the size of the BOOT partition (small) has been consistent but now on BUSTER and beyond it has grown. If you attempt to clone from an original which was upgraded from Stretch without expanding the BOOT partition, then the clone may well fail. At the time of writing this blog entry, RPi-clone does not know about changing the size of the BOOT partition. By September 2019 this had been resolved so see the later blog entry who’s permalink is here.
So, my 16GB original SD had an upgrade to Buster on it but without a change to the BOOT partition size. Good enough to run BUSTER but NOT a large enough BOOT partition to handle the cloning procedure as well. I initially thought that earlier (upgraded Buster) clones using -f were forcing the new SD to resize the (FAT) BOOT sector to the larger BUSTER standard of 120MB, as against the earlier 44MB of Stretch, in fact the clones STILL had only 44 meg BOOT sectors.
As RPI-Clone did not (at the time of this blog entry) handle the boot partition resizing, I ended up putting my original (well, a clone) clone into a new RPI4 USB socket while running the RPI4 from a new Buster installation on SD. I had to manually add (using the RPI terminal program – easy aside from the pain of attaching a screen and keyboard to the RPI4) GPARTED to the SD installation then running that with my SD in it’s USB adaptor – and CAREFULLY using GPARTED to firstly take off 200 MEG from the end of partition 2, then moving P2 right by the same amount. I then expanded P1 from 44MB to around 130MB without shifting it’s left position. I then saved that new modified Buster clone, powered off, put my SD into the PI4 SD slot and powered up. From there I could make a NEW first clone using the RPi-clone –f option. Having now prepared a clone with the right BOOT partition size I could now make a test change to my original clone and use the second clone to re-clone WITHOUT a reboot or –f option.
Here in more detail from the start with a desktop RPI on a virgin Buster…. in the terminal…
sudo apt-get install -y gparted sudo gparted
I then put into the USB socket a working upgrade clone – which had the partition SDA1 at 44236 blocks size (1k SD blocks hence 44 MB BOOT partition)
and told GPARTED to process SDA rather than it’s working operating system on mmcblk0
I then used the PARTITION – UNMOUNT command to remove the little keys from the partitions, i.e. to ensure the SDA partitions were not mounted, After that I shrunk SDA partition 2 (SDA2) by a roughly a couple of hundred meg and shifted it’s start position right accordingly. I moved the right side of SDA1 right to increase SDA1 to fill up the gap without shifting it’s left side.
When I set this off going, SDA2 resized, took about half an hour, but when it came to SDA1, it failed, maybe I was supposed to leave a gap or maybe I inadvertently shifted the left of SDA1. I tried resizing SDA1 again (very quick) and this time ensured I didn’t move the left side AT ALL and reduced the size of SDA1 to 134 Meg to leave a gap. When I set it going, everything just worked. i.e. I turned off the Pi4, swapped SDs and the newly size-changed SD with all my data still intact just worked.
I rebooted and made a clone of the clone using –f option. So now I am able able to use THAT clone to make another….. I powered off, swapped SDs, cloned again with no –f option…. this would be quick, but I’d rebooted. 5 minutes later I had my clone – but could I do it again, this time without the –f option again AND without without a reboot or –f option? YES.
I now had a complete upgrade from Stretch, using Buster and able to clone, without losing any of my data or setup.
there is a lot of arrogant and often incorrect or out-dated nonsense out there about backing up/cloning Raspberry Pi. My subscribers deserve better. The informatiuon you see here is how I handled the boot partition resizing until BillW (rpi-clone) added the option to handle this to rpi-clone itself.
Having now read the RPi-Clone README properly (not easy) it turns out there is an –s option you can put on the far right of the rpi-clone command to alter /etc/hosts and /etc/hostname if you want to change the hostname of your clone/copy. Please note that an alternative way to do this – nmtui – does NOT update the hosts file.
In coming to the conclusions above, I tried the commands suggested below: on my RPi-3 with a known good SD which only the previous day had made a “good” backup on that same RPi3:
df -i
and the result of the test was:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 971040 268781 702259 28% /
devtmpfs 123728 379 123349 1% /dev
tmpfs 124880 10 124870 1% /dev/shm
tmpfs 124880 556 124324 1% /run
tmpfs 124880 3 124877 1% /run/lock
tmpfs 124880 14 124866 1% /sys/fs/cgroup
/dev/mmcblk0p1 0 0 0 – /boot
tmpfs 124880 13 124867 1% /run/user/997
tmpfs 124880 13 124867 1% /run/user/1000
which seemed fine – then this test:
sudo tune2fs -l /dev/sda
Second test failed (for reference) as it looks like you can only test partitions, not the complete SDA.
SO I completely shut down my PI (sudo shutdown now) then turned off then turned back on…
I am of course doing all of this via my PC, using Mobaxterm in an ssh session with the headless Pi as usual…
In case anyone thinks this means a lot of typing… here’s my latest list of aliases… in /etc/bash.bashrc go to /etc and edit with “sudo nano bash.bashrc” – this lot are on the end of the file.
alias mqttlog='tail -f /var/log/mosquitto/mosquitto.log | ccze -A' alias nrlog='sudo journalctl -f -n 50 -u nodered -o cat | ccze -A' alias space='df -h|grep -v udev|grep -v tmpfs|grep -v run' alias cls='python /home/pi/cls.py' alias stop='sudo shutdown now' alias boot='sudo reboot' alias clone='sudo rpi-clone -U sda' alias cclone='sudo rpi-clone -f -U sda'
I’m about to change clone and cclone (clean clone) to add the –s flag on the right with new manually added hostnames just as soon as I figure out how to add arguments to aliases.
When I make clones now I see, using my PARTITIONS alias…
pi:~:09:13[0]> partitions
major minor #blocks name1 0 4096 ram0
1 1 4096 ram1
1 2 4096 ram2
1 3 4096 ram3
1 4 4096 ram4
1 5 4096 ram5
1 6 4096 ram6
1 7 4096 ram7
1 8 4096 ram8
1 9 4096 ram9
1 10 4096 ram10
1 11 4096 ram11
1 12 4096 ram12
1 13 4096 ram13
1 14 4096 ram14
1 15 4096 ram15
179 0 15558144 mmcblk0
179 1 131072 mmcblk0p1
179 2 15261184 mmcblk0p2
8 0 31260672 sda
8 1 131072 sda1
8 2 30963712 sda2
See mmcblk0p1 and sda1 – at 131072 – well, under STRETCH those blocks WERE 44,000 (or 44MB) or thereabouts, no good for Buster and I kept cloning the OLD boot size to the NEW copy – NOT big enough. Not any more.
Update 2023 for RPI4: I now make my BOOT partitions 256 Meg…
Hi Guys, I got here because I did a search for ‘Raspbian Superblock problem’. I want to backup my SD/uSD cards and it keeps failing because of a superblock problem. One system was probably stretch upgraded to buster, and the other *may* have been buster from the start. Is it possible that the sector sizes on these cards may be relevant?
I can’t clone because I don’t have any spare cards.
Your thoughts appreciated,
PeterM
If the original did not have the same boot sector size you have to do a CLEAN clone…
Here are my definitions I put into /etc/bash.bashrc in case it helps. I use “clone” normally but if using a fresh SD or going from an older Raspbian to the current one – I use “cclone” for clean clone.
#optional hostnames in 4 functions below
BLACK=’\033[0;30m’
RED=’\033[0;31m’
GREEN=’\033[0;32m’
BROWN=’\033[0;33m’
BLUE=’\033[0;34m’
PURPLE=’\033[0;35m’
CYAN=’\033[0;36m’
LIGHTGRAY=’\033[0;37m’
DARKGRAY=’\033[1;30m’
LIGHTRED=’\033[1;31m’
LIGHTGREEN=’\033[1;32m’
YELLOW=’\033[1;33m’
LIGHTBLUE=’\033[1;34m’
LIGHTPURPLE=’\033[1;35m’
LIGHTCYAN=’\033[1;36m’
WHITE=’\033[1;37m’
NC=’\033[0m’
clone () {
printf “${LIGHTBLUE}Creating a quick clone on SDA${NC}\n”
touch /home/pi/clone-date
bashCmd=(sudo rpi-clone -U sda)
if [ -n “$1” ]; then
bashCmd+=(-s “$1”)
fi
“${bashCmd[@]}”
}
cclone () {
printf “${LIGHTRED}Creating a full clone on SDA${NC}\n”
touch /home/pi/clone-date
bashCmd=(sudo rpi-clone -f -U sda)
if [ -n “$1” ]; then
bashCmd+=(-s “$1”)
fi
“${bashCmd[@]}”
}
cloneb () {
printf “${LIGHTBLUE}Creating a quick clone on SDB${NC}\n”
touch /home/pi/clone-date
bashCmd=(sudo rpi-clone -U sdb)
if [ -n “$1” ]; then
bashCmd+=(-s “$1”)
fi
“${bashCmd[@]}”
}
clonem () {
printf “${LIGHTBLUE}Creating a quick clone on MMCBLK0${NC}\n”
touch /home/pi/clone-date
bashCmd=(sudo rpi-clone -U mmcblk0)
if [ -n “$1” ]; then
bashCmd+=(-s “$1”)
fi
“${bashCmd[@]}”
}
ccloneb () {
printf “${LIGHTRED}Creating a full clone on SDB${NC}\n”
touch /home/pi/clone-date
bashCmd=(sudo rpi-clone -f -U sdb)
if [ -n “$1” ]; then
bashCmd+=(-s “$1”)
fi
“${bashCmd[@]}”
}
cclonem () {
printf “${LIGHTRED}Creating a full clone on mmcblk0${NC}\n”
touch /home/pi/clone-date
bashCmd=(sudo rpi-clone -f -U mmcblk0)
if [ -n “$1” ]; then
bashCmd+=(-s “$1”)
fi
“${bashCmd[@]}”
}
Hi Steve
Having just set up your script and sorted my node-red imported flows (hard work) the need to backup is important.
Cab I just check, do you use – sudo rpi-clone sdb every time you backup a working rpi4?
Thanks Brian
(I know this was meant for Steve – for the record I made a function in /etc/bash.bashrc (using sudo nano /etc/bash.bashrc) and putting them at the end…
#optional hostnames in functions below
clone () {
bashCmd=(sudo rpi-clone -U sda)
if [ -n “$1” ]; then
bashCmd+=(-s “$1”)
fi
“${bashCmd[@]}”
}
cclone () {
bashCmd=(sudo rpi-clone -f -U sda)
if [ -n “$1” ]; then
bashCmd+=(-s “$1”)
fi
“${bashCmd[@]}”
}
cloneb () {
bashCmd=(sudo rpi-clone -U sdb)
if [ -n “$1” ]; then
bashCmd+=(-s “$1”)
fi
“${bashCmd[@]}”
}
ccloneb () {
bashCmd=(sudo rpi-clone -U sdb)
if [ -n “$1” ]; then
bashCmd+=(-s “$1”)
fi
“${bashCmd[@]}”
}
The functions are used by user PI at the command line just as is (no sudo) – ie “clone” clones sda, cloneb clones to sdb – the cclone and ccloneb variations do clean cloning – wiping everything first for new chips or ones with incompatible partitioning. I hope someone finds this useful – I use them on a regular basis before any significant changes to ensure I have a back up case of disasters. Priceless.
Shouldn’t the ccloneb option have a -f flag in the rpi-clone command. Otherwise cloneb and ccloneb appear identical?
Good job you’re on the ball, Steve, I’ve been making backups of that mistake. Now corrected, thanks.
Many thanks Steve. This would make a great step by step video for us noobees.
BrianD
Strange, its backed up all of my flows but MQTT wont connect. I have checked the mqtt in and out nodes and they appear to have all of the correct connection data. Any idea what has gone wrong.
I used the cclone option.
No idea, cloning works for me every time.
hello Pete
here’s some information about the boot eeprom
https://www.raspberrypi.org/documentation/hardware/raspberrypi/booteeprom.md
regards Brian
hello Pete
I havent the foggeist idea why,but it is the same
as the gap from the start of sd card to the boot partition.
This is with no changes by me. Just the busterlite official img
from the raspi site
Raspbian Buster Lite
Minimal image based on Debian Buster
Version: July 2019
Release date: 2019-07-10
Kernel version: 4.19
Size: 426 MB
The image burnt by Etcher
Its irritating not to know why!, and suspicious
that both gaps are the same size.
regards Brian
hello Pete
just noticed I didnt comment on the gap between boot and data partition
on buster in my comments Aug 7th to Aug 11
The gap is also 8192 sectors of 512KB = 256MB
This is from the busterlite official img burnt on Etcher
Reagards Brian
Thanks Brian. That’s a BIG gap, more than I left.. Any idea why?
hello Pete
rpi-clone buster to buster
busterlite to new sd 32Gb card useing etcher
rpi-clone sd 32gb to this card.
The running prog buster rpi 4 all flows nginx
NOT upgrade from Stretch
pi@rpi4busternginx:~ $ sudo mount /dev/sda1 /media/usb
pi@rpi4busternginx:~ $ lsblk -b
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 31914983424 0 disk
├─sda1 8:1 1 268435968 0 part
└─sda2 8:2 1 1920991232 0 part
mmcblk0 179:0 0 31914983424 0 disk
├─mmcblk0p1 179:1 0 268435968 0 part /boot
└─mmcblk0p2 179:2 0 31638159360 0 part /
Boot partition is 256Mb on both
pi@rpi4busternginx:~ $ sudo rpi-clone sda -f -U
Destination disk partition /dev/sda1 is mounted on /media/usb.
The clone cannot proceed unless it is unmounted.
Booted disk: mmcblk0 31.9GB Destination disk: sda 31.9GB
—————————————————————————
Part Size FS Label Part Size FS Label
1 /boot 268.3MB fat32 — 1 268.3MB fat32 —
2 root 31.6GB ext4 rootfs 2 1.9GB ext4 rootfs
—————————————————————————
== Initialize: IMAGE mmcblk0 partition table to sda – forced by option ==
1 /boot (40.4MB used) : IMAGE to sda1 FSCK
2 root (3.1GB used) : RESIZE(31.6GB) MKFS SYNC to sda2
—————————————————————————
Run setup script : no
Verbose mode : no
———————–:
** WARNING ** : All destination disk sda data will be overwritten!
: The partition structure will be imaged from mmcblk0.
———————–:
Initializing
Imaging past the start of /boot partition 2.
=> dd if=/dev/mmcblk0 of=/dev/sda bs=1M count=268 …
Resizing last partition to end of disk …
Resize success.
Changing destination Disk ID …
Delaying so partprobe can update /dev entries …
=> fsck -p /dev/sda1 …
=> mkfs -t ext4 /dev/sda2 …
Syncing file systems (can take a long time)
Syncing mounted partitions:
Mounting /dev/sda2 on /mnt/clone
=> rsync // /mnt/clone with-root-excludes …
Mounting /dev/sda1 on /mnt/clone/boot
=> rsync /boot/ /mnt/clone/boot …
Editing /mnt/clone/boot/cmdline.txt PARTUUID to use 779bc80b
Editing /mnt/clone/etc/fstab PARTUUID to use 779bc80b
===============================
Done with clone to /dev/sda
Start – 22:58:26 End – 23:02:20 Elapsed Time – 3:54
unmounting /mnt/clone/boot
unmounting /mnt/clone
===============================
pi@rpi4busternginx:~ $
Note the time taken 3mins 54secs
tested cloned sd card all appears to be running ok
regards Brian
Hello Pete
https://www.raspberrypi.org/blog/buster-the-new-version-of-raspbian/
PhilE says:
25th Jun 2019 at 12:50 pm
Provided there are no unnecessary files on your card, an old 56MB /boot FAT partition should be sufficient for an upgrade, but we would strongly recommend moving to the new 256MB FAT partition when possible.
Reply
spotted this official boot is 256MB
so start sector end sector is correct as is the gap to next partition
regards Brian
hello Pete
interesting I just noticed in system information they list boot partition as 250MB
as well
regards Brian
Hello Pete
Good questions exactly what I thought.
I can ask for bytes thus lsblk -b
I get on a Virgin SD card with busterlite dated 10/7/2019 burnt by etcher
lsblk -b the -b forces byte info
pi@raspberrypi:~ $ lsblk -b
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0 179:0 0 31914983424 0 disk
├─mmcblk0p1 179:1 0 268435968 0 part /boot
└─mmcblk0p2 179:2 0 31638159360 0 part /
pi@raspberrypi:~ $
boot partition is 268,435,968 bytes
!! maybe something major with booting is planned!!
There is a gap to the start of the work partition.
if you enter fdisk -l
at the end you get these are sectors of 512KB
Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 532480 524289 256M c W95 FAT32 (LBA)
/dev/mmcblk0p2 540672 62333951 61793280 29.5G 83 Linux
pi@raspberrypi:~ $
If you calculate 532480 -8192 = 524289 sectors of 512KB for boot
524289 * 512 = 268435968 bytes in boot partition
This agrees with the figure for bytes from putting in lsblk -b as above
hope this helps
regards Brian
Brian – I dont understand why your BOOT needs to be 256 meg, even Buster apparently only needs 128 meg. So, I’m seing the start at 8192… mine is that, but is that bytes or blocks and if blocks, what size? And is there a reason for the gap between P1 and 2?
hello Pete
forgot to mention the gap to next partition perhaps the hospital can look at
my brain (if Iv got one) next time!
/dev/mmcblk0p2 540672 62333951 61793280 29.5G 83 Linux
next partition starts at sector 540672
I do regard flashing lights as invasive they give me migraine
regards Brian
hello Pete
The sd card sandisk 32gb brand new
the interesting bit is boot partition
/dev/mmcblk0p1 8192 532480 524289 256M c W95 FAT32 (LBA)
start sector 8192
end sector 532480
size 256M
I think you can use this I cant as I dont know enough about Linux
This was a virgin card no cloning
buster lite latest
script run no errors
Flows exported from my original pi3 as json
imported to programm
regards Brian
Hello Pete
Here is info from my RPI4 buster light sd card
might be of help (Ibelieve their is no such thing as useless info)
pi@rpi4buster:~ $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0 179:0 0 29.7G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 29.5G 0 part /
pi@rpi4buster:~ $ sudo fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 29.7 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x91da3615
Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 532480 524289 256M c W95 FAT32 (LBA)
/dev/mmcblk0p2 540672 62333951 61793280 29.5G 83 Linux
As to the hospital they stuck a catheter into my main artery at the groin and went up to my heart to have a look (in full colour display).
Not something I would like to repeat!
regards Brian
Far from useless info Brian… I was always led to belive that PIs used 4k sectors until I realised mine all use 1K sectors (probably as I’ve upgraded and upgraded)… and yours appears to be using 512 byte sectors… now my interest is up, I wonder for a PI in typical control system (non-desktop) use, what the ultimate sector size would be…. and I never did get to the bottom of the exact start location and size of the boot partitionj should be for Buster…. and what if any gap is actually needed between boot and root…. What I have works very well, but it would be nice to be precise – if GPARTED will actually let you change destination sector size as well (I just changed the size of the boot part after moving up and shrinking the root part) then RBOOT handling the clone/copy without losing source data. All sounds a little tricky.
I’m glad to say I had nothing invasive today unless you call experimenting on the brain with flashing lights invasive.
a good guy wrote about it 🙂
This article was a gift only just for the fact it mentioned MobaXterm. I was just in the process of rebuilding my W10 system after a crash. I did not know that app and it is a sort of Swiss knife for remote controlling your PI.
Ho Leo – I’ve been using Mobaxterm for ages now, I love it. Free is great, paid allows more sessions… https://mobaxterm.mobatek.net/download.html
I need more SDs to experiment with – indeed just took delivery of a pair of 32GB Samsung EVO+ 32GB SDs.
Pete,
Looks like /tmp/rpi-clone-ooutput might have a lot more details on the actual failure, too.
-John-
and check /var/log/messages and/or /var/log/syslog
Pete,
Next time you see the problem, try doing a “df -i” and “tune2fs -l /dev/XXXX” for that specific device both before and after the reboot, saving the results somewhere permanent so that you can compare the results.
You’re probably most interested in the “Free Blocks” and “Free Inodes” counts, but possibly the reserved block counts, too.
It sounds as though your backups are somehow creating too many fragments and running the filesystem out of available inodes (although I haven’t ever come across any back-up system that writes to the filesystem it’s backing up from).
It might also be worth going to the Buster release notes and checking whether the filesystem defaults changed.
-John-
Well done, John, at least that gives me a starting point.
Right – BEFORE: I did the first one.
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 971040 268781 702259 28% /
devtmpfs 123728 379 123349 1% /dev
tmpfs 124880 10 124870 1% /dev/shm
tmpfs 124880 556 124324 1% /run
tmpfs 124880 3 124877 1% /run/lock
tmpfs 124880 14 124866 1% /sys/fs/cgroup
/dev/mmcblk0p1 0 0 0 – /boot
tmpfs 124880 13 124867 1% /run/user/997
tmpfs 124880 13 124867 1% /run/user/1000
The second one – you didn’t mention it needs SUDO so I prefixed it with sudo.
sudo tune2fs -l /dev/sda
and the result was:
tune2fs 1.44.5 (15-Dec-2018)
tune2fs: Bad magic number in super-block while trying to open /dev/sda
Found a dos partition table in /dev/sda
But that was a GOOD backup SD? Help… That was an SD I’d used last night and made a “successful” backup with.
Surely it WOULD find a DOS partition (first one) on a PI…..
Pete,
Yup, tune2fs will only work with Unix filesystems (partition level, ie:- /dev/sda1 not device level, so not /dev/sda) and definitely not DOS.
The inode problem only applies to Unix filesystems, so it looks like you’ve got something else going on there.
-John-
Well, for sure, both SDA1 and 2 are created by the PI…. as you saw, second time around, the SAME SD made what seems like a perfect copy. The only differnce being, this time I used -f and did a power cycle. Occasionally, the power cycle is not needed, sometimes, the -F is not needed and more rarely none of that helps. That last one might be a red herring. VERY occsionally under STRETCH I would get an unrecoverable SD, most of the time it all just worked. I’m sure there is something else here I and others are missing.
I’m guessing that DOS message is something to do with the first partition on any RASPBIAN system (the boot partition) being DOS compatible – i.e. FAT16 or 32.