Easy Backup 2

Having satisfied myself that RPI-CLONE does a good job of creating backups of Raspberry Pi (no guarantees you won’t lose data but it hasn’t happened to me yet) I started the search for a solution for other boards. I find the Orange Pi +2E to be a particularly nice piece of hardware, but useless without an easy backup.  armbian-config will let you copy from SD to EMMC but not the other way around – so no backup capability for eMMC users. So I've been working on this...

I wrote THIS for SD and the FriendlyArm NanoPi boards ages ago

syncto()
{
if [[ $1 = "" ]]; then
echo "Sorry - no IP specified. Use syncto ip_address"
else
sudo rsync -aAXv / --delete --exclude={"/run","/etc/hosts","/etc/hostname","/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} root@$1:/
fi
}

After ensuring the backup SD has been given a basic Armbian and the SD resized, well, it turns out that given a backup SD with a basic Armbian Xenial installation on the OPI +2E, it is easy to back up for example an SD Armbian installation to the backup SD. the "/boot/*" exclude is only needed for eMMC to SD backup and it IS essential for that. Took me several hours to come to that conclusion in my ignorance. Note that on setups with sizeable traffic and/or database action going on you could lose data. Well, just don't do it.

sudo mkdir /mnt/sda1

sudo mount /dev/sda1 /mnt/sda1

sudo rsync -aAXv  / --delete --exclude={"/boot/*","/run","/etc/hosts","/etc/hostname","/dev/*","/proc/*","/sys/*","/tmp/*",”/run/*”,"/mnt/*","/media/*","/lost+found"} -q /mnt/sda1

Miss out the hosts and hostname entries if you want to replicate the original including name. For sd to sd copies miss out the boot exclude.

HOWEVER when it comes to backing up an eMMC installation, I’m getting no-where. It LOOKS like the backup has done without error but whereas you’d expect an SD installation to take priority over an eMMC installation, in this case, plugging the new SD in results in the board booting from eMMC. Not 100% yet as only occasionally I get errors in the copy with the message "structure needs cleaning".

Facebooktwittergoogle_pluspinterestlinkedin

19 thoughts on “Easy Backup 2

  1. I don't know anything about the OPI +2E, but does it perhaps have a kernel command line like in Raspberry's /boot/cmdline.txt or Grub's config on a PC? Where it has a parameter root=/dev/... or root=PARTUUID=... to tell the kernel where the root filesystem is?

    1. Not like the Pi AT ALL apparently. Pi has partition 0 for boot, partition 1 for file system. Armbian people make diparaging remarks about Pi version even though it has outsold the rest by a large margin.a Pi does hoever have limited RAM fore no good reason and VERY limited serial ports - hence my interest in sOPI S+2E but up to now I cannot take advantage of eMMC on OPI +2E as I cannot restore a backup.

      1. > Armbian people make diparaging remarks about Pi version

        Pete, seriously if you don't want to understand why a VideoCore SoC is different than an ARM SoC literally no one can help you.

        I'm trying to explain to you since quite some time now why the RPi is different and why every ARM SBC out there doesn't boot the same way (not searching for some files with specific names on a FAT partition but loading a secondary boot loader from a fixed address somewhere on the boot media and then u-boot. Both not living in or requiring any partitions or filesystems).

        The reason why understanding this is important: you then will also understand why rsync won't help you getting a bootable clone on a new SD card. Since rsync acts above the filesystem layer and inside partitions and neither SPL nor u-boot live there.

        That's why tools that only take care about the RPi and work there flawlessly will not result in creating a bootable clone on any ARM SBC (unless you already flashed an OS image to the SD card prior to that since then u-boot+spl are already there -- but most probably outdated)

      2. well, what can i say, i use as well various small boards. the pi has beat them all in terms of versatility.

        however, there are some things the pi is not as well equipped as other boards. mind you, it was worse a year or two ago. i started my foray into arm-mini-servers with an NSLU2 and two seagate dockstars, mainly to have an cheap way of leaving a server on 24/7 as use as a NAS, name- and dhcp-server for my local home network.

        for the rest, there were AVR's and arduinos ( never got into the pic side of things ).

        when i wanted to make my own mailserver, the pi1 was not able to perform properly, so i bought a banana pi, which performed way better because of its dual core CPU, and the NAS was moved to a cubox i4-pro with it's e-sata port and gbit network interface. the pi's are used for kodi, volumio, security cams, etc, with the latest development being:

        odroid xu4 as NAS, dns-server, dhcp-server, running as well as mosquitto, node-red, telegraf, influxdb and grafana. the mail-server is now a tinkerbox. sensors are mostly esp8266 ( wemos D1 pro ) and an esp32 drives a waveshare 7" e-paper display, the latter working, but the actual layout of the displayed information leaves alot to be desired 🙂

        the main reason why the pi's are used for other things is, the USB3 ports on the odroid xu4, which make it a really good NAS, and the processing power of the tinkerbox, altho this could be a 3B+ pi as well, the pi definitely is good for a small mail-server now, definitely.

        a pi is as well used for the latest project (industrial mobile data collection), it runs as well influxdb, grafana, bind, dhcp-server, accesspoint, connecting an ultrasonic flow-meter, differential pressure sensors and power meter directly over USB, and several PT100 sensors over wifi/mqtt from esp8266 modules. as well as in my home project, node-red is responsible for collecting and transforming ( when needed ) the data, to place them in the influxdb, for later processing.

      3. Hi Pete,

        I've got an OrangePi +2E but it's in live use right now and I don't have a spare to 'play' with but I guess you might have to temporarily change some boot settings to undo the 'emmc' partition mapping so that it then looks at your SD card when restoring your image. Then, once restored, you can transfer it back to EMMC as per how you originally moved from SD to EMMC.

        At your own risk (I've not tried this), maybe edit stuff in

        /boot/boot.cmd

        and/or

        /boot/armbianEnv.txt

        Then rebuild the image using 'mkimage' for it to take effect

        mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr

        Then reboot?

        Someone else like Tkaiser may be able to advise whether the above is the correct advice. It's a topic that I'm interested in too as I have an +2E running Armbian from EMMC and don't currently have a backup/restore solution in place for that machine unlike all my SD card based Raspberries, Oranges and FriendlyARMs.

        My plan B would be to blat/format the EMMC so it can't boot from it and maybe then tries the inserted SD card that has your backup on it?

        If I hadn't just deployed my +2E, I'd try all this out for you first but maybe there's some food for thought in my suggestion and someone else can jump in and confirm whether or not I'm pointing you in the right area...?

        1. In fact, any SD card image in place should take precedence over the eMMC image anyway. All of this seems like a good reason not to use the eMMC. I get no such issues with Raspberry pi. The Orange Pi boards really need an easy eMMC backup.

        2. With Allwinner boards SD card has always higher boot priority than eMMC (but that's not everywhere like this, e.g. I recently managed to brick my Rockchip NanoPC-T4: https://forum.armbian.com/topic/7498-nanopc-t4/?do=findComment&comment=62192 )

          When booted from eMMC you can insert an SD card, then write u-boot+spl to the card, set up partitions, transfer the partition contents with rsync or whatever tool you like, then check UUIDs with blkid, then adjust UUID on the SD card in /boot/armbianEnv.txt. Done.

          For writing u-boot+spl check the contents of nand-sata-install and there the write_uboot_platform function (relying on the contents of /usr/lib/u-boot/platform_install.sh which are different for each and every single ARM SoC out there since their boot mechanisms are all incompatible).

          You need to source the functions from /usr/lib/u-boot/platform_install.sh and then call them with appropriate $DIR variable and pointing to the SD card device node (which is not always fixed since u-boot developers chose to switch SD card and eMMC nodes last year)

          When accepting the challenges (every ARM SBC out there using a slightly incompatible boot mechanism which requires to handle them all separately when you want to do it correctly) and when you're fine with 'cloning gone wrong' then it should be pretty easy to use the above information to create an eMMC backup script.

          Since I know that live cloning can't work reliably with such SBC distros as Raspbian or Armbian my only advise is still shutting down the board and doing an offline clone via USB as explained already. I do this regularly with my important Allwinner boards booting from eMMC. The clones go to a btrfs filesystem with deduplication functionality so I get not just one clone but incremental updates not needing much space. So I can go even back into time to restore an older clone -- directly to eMMC via USB since why harming SD cards in this process? 🙂

        3. The best way to clone the contents of the eMMC or to refresh it on an OPi Plus 2E is to shut the board down, take a Micro USB cable, connect it to a PC or other SBC and then holding down the FEL button when powering on the board again. When using our little tools the internal eMMC will then simply appear as an USB drive on the other machine that can be simply backed up (at least with Linux and macOS -- on Windows it's more complicated, just see the two links in another post of mine)

          BTW: In https://github.com/armbian/build/tree/master/config/sources all the boot differences of at least the ARM SoC platforms Armbian supports can be studied. It's the write_uboot_platform function that is essential and checking the other u-boot relevant functions uboot_custom_postprocess or setup_write_uboot_platform is a great way to get an idea how different those ARM thingies boot.

          1. At last, simple, good advice, some of the more in-depth stuff gets too much for those of us who use operating systems as a means to either use or write applications. Thank you. For the T4, I have two of those but took the easy option of using them as Android TV Boxes. The T4 does that well though it would be even better if screen overscan could be tackled at a high level, simply. More expensive than, but more flexible than "Android TV boxes which usually have a crippled Play Store.

            I just recently reviewed such a box -awful - and I see them all the time out here in rural Spain - Brits buy them for the comfort of TV just like at home and usually pay over the top. I am getting excited about an Android 8.1 box on it's way to me with full Play Store and 4K handling as I have a 4K TV (I'll grant you there is very little 4K material out there).

          2. Sadly, experience shows that Windows (I'm a life-long and enthusiastic Windows user) is pretty crap at handling Linux partitioning and related. I used to throw SD boards away thinking they were bust until I twigged recently to partitioning boards with errors (I only use good Samsung and Sandisk) with a Raspberry Pi. Like many others I do not have any Linux PCs nore do I plan to get one so the Pi has proven to be indispensible. If only they'd put more RAM in them. That and eMMC is why I'm so keen to get OPI +2E with working backup.

            1. to recover an sd on windows

              diskpart
              list disk
              # BE SURE to choose the right disk number, 1st column of previous command
              select disk ### <-- replace with the number above
              clean
              exit

              the "clean" command will NOT ask for a confirmation (in pure unix BOFH style), and clears every partition on the selected disk... then you can use the windows disk utility to create new partitions, graphically, or continue using diskpart... or move to linux and do from there

          3. Be wary about over-assuming my knowledge or that of others. So this sounds good at first glance. OPI +2E - shut it down, connect toRaspberry Pi USB via USB cable, hold down the FEL button on the OPI +2E (between the microphone and 3-way UART connector) on power up....

            Got that far..."using our little tools" - are you able to elaborate? Which tools - if the OPI is in some form of reset mode at this time I guess you mean some tool we install on a Pi or smiilar - you say Windows is not a good idea for this, I concur. If you have time to elaborate,please do remember we're not all heavily into operating systems - and hence may not follow terms you thing as common sense. My attempts with OPI +2E and SD seem to work but do not always bootable SDs all the time - and I don't know why.

            1. > "using our little tools" - are you able to elaborate? Which tools

              https://tech.scargill.net/easy-backup-2/#comment-45212

              Just tried it out on my MacBook. It's as 'complex' as

              - Double click on my app
              - connect the board with an USB cable to my MacBook
              - press FEL button on the board
              - power on

              The eMMC is available as an USB drive now and can be read and written.

              If you don't use macOS or Linux where everything works out of the box you might need additional drivers. The situation with Windows is described in the other link and there's a Windows GUI available that could cope with more Allwinner SoC families: https://tech.scargill.net/easy-backup-2/#comment-45214

              1. I think it is far to say that very few of us (percentage wise) are using MACs and probably just as few using Linux PCs. I'm using a pretty standard Window 10 64-bit PC. I'll take a look once I've resolved another unrelated problem- Node 8 and Node-Red on the Pi. Thanks for the feedback.

                Is this a stupid question? You said "double-click on my app" - where? And for clarification- press and HOLD FEL button, power up, then release ?

                1. I honestly don't care about which OS who is using and how many users prefer this over that. I only said that this is stuff that works out of the box with Linux and macOS and requires some extra steps with Windows.

                  Since I personally do not use Windows (only when being paid dealing with Windows servers here and there) I can neither provide help nor recommend any steps.

                  So if you're Windows user at least I can't help you with the necessary steps but the link I shared https://github.com/zador-blood-stained/fel-mass-storage should provide the details (e.g. installing something called Zadig to install additional USB drivers for whatever reasons)

                  Also again: https://forum.armbian.com/topic/2454-fel-mass-storage-or-writing-images-directly-to-emmc/?do=findComment&comment=42145 -- since I'm no Windows user I never tried this GUI and can't comment on whether it works or not. Just spotted its existence and sharing as a possibly nice way to ease to process.

                  Wrt FEL button (on some boards called u-boot button). You press this button and then the SoC enters FEL mode: http://linux-sunxi.org/FEL/Protocol so it can boot over USB. As soon as FEL mode is entered (the the SoC appears as a new USB device on the connected host) the button can be released.

                  The tools we (linux-sunxi and Armbian community) provide then load in a single step a kernel to the SoC which presents then eMMC or an SD card as USB Mass Storage devices so the board will simply appear as an external USB 'thumb drive' exposing the eMMC's contents this way.

                  And double clicking my app is not possible on Windows since the app only runs on macOS. For Windows see the other links and the necessary (more complex) steps there.

  2. As explained already in various comments you need to take care about SPL and u-boot on any real ARM SBC (the RPi being not a real ARM SBC but a hybrid and the boot variant there is 'loading a proprietary RTOS called ThreadX from first FAT partition' and as such incompatible to any real ARM SBC out there).

    If you want a bootable clone (you're always talking about cloning BTW -- backup needs versioning) you transfer to copy u-boot+SPL too. This can be done with a full device clone *below* the partition and filesystem layer, e.g. by using dd.

    Cloning a live system can't work reliably (doesn't matter whether this is understood or not, it's technically not possible to 100% reliably do this without special precautions) so the best way to clone the eMMC on an Allwinner board like OPi 2E is

    - shut the board down
    - Use a Micro USB cable to connect the board to a PC or Mac
    - Start our tools that will 'FEL boot' the OPi 2E later so that the internal eMMC will appear as an USB drive to the PC (or Mac)
    - boot the OPi 2E holding down the FEL button
    - create a disk image. With macOS and Linux use ddrescue for example, in Windows WinDiskImager32

    This way you get a full clone of the eMMC installation able to burn to an SD card (or directly to eMMC again using the below mentioned tools)

    - For all OS visit https://github.com/zador-blood-stained/fel-mass-storage
    - For macOS when wanting more convenience visit https://github.com/ThomasKaiser/sunxi-armbian-flasher-osx

    1. TKaiser - I know you felt an urgent need to jump in there, maybe you didn't read everything or more likely I've not updated my notes. I'm aware of the differences between the bona-fide Raspberry Pi (best selling SBC) and others, many of which I've tested and played with as well as deployed.

      I also know that backing up a live system can result in data loss though this hasn't happened to me and after hundreds of backups I'd still rather have rpi-clone than nothing.

      I also realised the partition structures are very different on othe boards. I just didn't know HOW different until recently. Hence I've a master OPI +2E here with the latest updates, Node-Red, SQLITE, Grafana, MQTT snd so much more.

      I have both SD and eMMC images (the eMMC image was made from the SD by Armbian-config - and that is, apparently in this case ALL it will do, Other options are greyed out.. Using an SD with only a minimal not-updated Armbian system on it, I put that onto a USB slot and use rsync as in this artcle (slightly updated as I had for gotten to exclude the boot folder and it was THAT whixh was preventing eMMC to SD copying, I've now made a master SD from the eMMC and from there I can use Armbian-config to program eMMC (from scratch) oe rsync to copy onto a basic Armbian installation, all my stuff.

      Clearly within the limits of (arrarently possibly) losing live data. this is demonstrably working. I have a SLITE database in there with umpteen tables, an Influxdb database feeding Grafana from Node-Red, boards talking to the sytem via MQTT and in all cases the copies work. If I can focus you into helping me figure out how to minimise or eliminate data losses on buser systems instead of keep saying is doesn't work - that would be great. Hey, it is better than losing data and the othe "solutions" out there either take an age or need a Linux degree.

  3. Hello Pete

    First hope this finds you in good health.

    Just done my first backup from Orangepi + 2E eMMc to SD card .
    Works perfectly- the
    only problem resizing the image (my lack of Linux), but managed in the end.
    One question should I freeze the kern against updating or can I update it?
    (You mentioned that the newer version of Armbian stretch would not work with the script) I have backups of all the versions anyway.

    Regards

    Brian

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.