Raspberry Pi RPI-CLONE Issues

Peter ScargillGeneral62 Comments on Raspberry Pi RPI-CLONE Issues

As I normally rave about RPI-CLONE – I  thought I’d share the dirty laundry this time.

** Update Jan 13 2019

It seems pretty obvious in hind-sight that the issues I was having below were power supply related and one bad SD card. I’ve just finished my 7th successful backup-clone since removing the intermediate battery pack to the Pi and shortening the USB lead. I neglected to note that I’d recently added an Amazon Echo generation 2 DOT into the mix. It is possible that is relevant. Good timing as I’ve just moved to Blynk local server and cracked ESP8266 host names on Windows and I do NOT want to lose that.

** end of update

I have backed up both Pi 2 and Pi 3 hundreds of times, generally without issue, but the last few days I’ve had some cloning problems – generally along the lines of failure – read-only.

Here is a photo of one of my Pi3 units  – 192.168.14.70 – the main controller sits at 192.168.14.71 but is otherwise identical).

Raspberry Pi 3 with OLED display

Incidentally, don’t make the mistake I did and fit an OLED up permanently to the PI. The digits on this 128×32 display are already starting to fade after less than a month of constant use. I suggest turning the display on only when needed. I had the same issue with a pair of NanoPi units I left running in Spain, after 3 months the OLED displays were all but shot. These are typical OLED displays you’d find cheaply on Ebay or AliExpress for example. But I digress.

This Pi (and my main Pi) are kitted out with Raspbian and the contents of “the script” and control real world stuff via Node-Red and MQTT (not a lot as I’m still setting up the UK).

Here is the screen output of a successful backup/clone using:

sudo rpi-clone –f –U sda

A variation of the above misses out the –f flag for SDs which have already backed up the same master (i.e. partitions are the same).

Screenshot of rpi-clone

Above I backed up an 8GB working master to a 16GB card. A similar backup to a 32GB card (all Sandisk or Samsung, no rubbish) then failed with a read-only message. I wiped the partial clone files in /mnt and rebooted.

I then cloned the 16GB SD to the original 8GB without the –f flag (which took very little time as almost no changes) and this worked perfectly.

Over the weekend after NYE, I saw the kind of error we saw on the 32GB version above a few times but a retry usually got it, occasionally I’ve had to bin the SD as I no longer trust it. It isn’t as yet entirely clear why this happens after several months of no issues so comments invited. I don’t THINK it is power supply related as these two boards have been backed up countless times in identical circumstances. I do backups before any updates, particularly to Node-Red which has a habit of trashing my nodes. A few days later, I’m having no issues at all, back to normal, it seems.

Which makes it all the more concerning when I get a day like at the weekend where backups were failing en-masse.

62 thoughts on “Raspberry Pi RPI-CLONE Issues”

  1. I wonder do I have Pete’s issues with rpi-clone. I have cloned 3 times from a 16G to a 32G sd card (both properly tested) and the process shows no errors. The result is that all seems fine except that root html page does not show nor does mqtt seem to be running. Node red seems ok. Very strange!

  3. So now we’re WAY off subject, talking about using these mini PCs as media players… however, Android media players are cheaper…

    Antonio you confused two of us at least, daying simultaneoudly that Debian and one of those PCs – No WiFi on stock 18.10

    “Decent on 18.04 modded iso I linked, on Netgear router – Shitty on 18.04 modded on my main ISP router”

    So, is 18.04 working with WIFI on the units – or not and if yes, how much work/time did you have to spend modifying. Still not seeing a compelling reason to move from the PI…

    And also we have discussed no wear levelling – recall our conversations with our Armbian friend – it seems the better SDs DO have some kind of wear levelling. Personally I only use Samsung and Sandisk – not Chinese clone SDs and generally, since cracking power requirements have no problems (except last weekend – still unresolved).

    1. lol, yes, we’re a bit OT 🙂
      but these are NOT mediaplayers, they’re x86 mini PCs… there’s no decent linux support on cheap mediaplayers, so they’re out for domotic usage, while these x86 pc can run standard linux (ubuntu in my case), without serious problems (using wired networking), and with acceptable power consumpion

      for wifi, i mean, on my device i tested both 18.04 modded iso i gave links to, and 18.10, on same hardware… 18.10 had no wifi support at all, while 18.04 modded iso has support, but i tested on my 2 wifi access points and had mixed results: connecting to a netgear one, wifi was stable (i installed and updated full system), while moving in my room where’s my main ISP router+wifi (different SSID), i had disconnects…

      so, don’t know if it’s the wifi of the pc or the router, if it’s the pc wifi it’s easily fixable using a well supported external usb wifi dongle, i don’t need it as i want wired, so my tests halt here…

      i’m moving from rpi3b+ to this because it’s faster, with equivalent costs (i paid 74€, while an rpi3b+ +good sdcard +good psu +nice case is quite as expensive and less reliable, sd cards are not meant to be used 24/7 (me too have sandisk and samsung), ssd and emmc yes

      i don’t need rpi gpio, i can always hookup sensors to esp8266 around the house, or attach an usb serial adapter if i need it to the minipc

      1. You guys are going to cry when I tell you this, I have ignored all best practice as I’m an idiot but guess what, it all works flawlessly and aside from power cuts hasn’t missed a beat in 4 years.

        I run Domoticz and Node Red on a Raspi-A (no case, no fan) attached to an unpowered USB hub attached to a desktop, I have an Arduino Uno plugged into the USB of the Raspi that deals with RF in/out, everything is stored on an 8+ year old 8Gb SD card I got from an old camera.

        Don’t worry, I’ve been expecting it to fail for 2 years so have regular backups, it’d take me a max of 5 mins to restore to a new Pi but it keeps astounding me with it’s reliability.

        On a side note – I have an old MacMini running Win10, that’s on 24/7 365 – turn off automatic updates, runs Kodi in the bedroom no worries.

        1. Aside from that burst of failures the other day and one failure on a Pi I constantly mess with a year or more ago, this stuff works for me. I just use decent SDs (no doubt with the odd Chinese copy thrown in which might account for all of my issues above). The setup I’ve left in Spain worked all summer when I was there and has done so since we left in October – despite no doubt umpteen 5 minute power cuts since. Now if it will just hold on until spring… please. For the ESPs I use my own ESP-GO but I’m willing to give Tasmota a go, especially for Sonoffs.

  4. As another alternative to the Pi and low power x86 – look on ebay at second hand desktops – people are getting rid for low prices. Just picked up a HP Pavilion i5 base unit with 8gb and 1TB disk for £50 for my daughter to play Minecraft. It even comes with a full Win10 installed – but she hasnt realised yet I’ve partitioned the disk to use for Hassio and storage.

    1. Elements to consider when looking a 24/7 up platform is also power consumption, noise and UPS.
      An Core i5 refurbished desktop would exceed my standards in those.

      Looking into Pete’s great work to get an UPS for it’s Pis, I must admit that even low power Atom/Celeron based PC would require some works to solve this topic too.

      1. that’s why i didn’t take more than high end atom/celeron or the likes, like z83 or ap34… the pc i use mostly is a n2820 intel nuc (named nuc-azzo, italian for “one dick”, yes, lol 😀 ), with a 128gb ssd and 4gb ram, with win10… it’s on 24/7 for years, it’s connected to my big tv so i usually just take my logitech remote keyboard/touchpad and i’m online… from there i go to a headless 10 years old eeebox with just 1gb ram and 500gb hd with ubuntu 16.04, which is my “download” (you know what i mean) machine, via ssh+vnc… at my parents home i’ve an i5, but i use it just 2 days every 2 weeks i go there… my notebook is an i7, i’ve not turned it on since last summer…

        i could just take an i3 or i5 and virtualize both my win10 and ubuntu, and add a new vm for hassio, but then what? I through away my nuc+eeebox, which both work flawlessly? To have what? More cpu power i don’t use or need?

        the ups side is easy addressable for these devices, though: a manageable one with usb connection and software is quite cheap now, so if mains go off it will shut down the system… it will not turn on again when restored, though, usually… doing something like this video by Ralph Bacon, you can maybe turn it on via a little arduino, for example, if battery charge is enough… https://www.youtube.com/watch?v=cDuObAR42Nw

        look at this 8bit guy video for some interesting info on mods for ups to have a longer lifetime, bypassing the inverter: https://www.youtube.com/watch?v=-Ur-D37-juo

        1. Only Barbudor has mentioned power use. 24/7/365 power… Pi uses half or less than the alternatives.

          1. that’s not relevant to me… they’re just about 7 watts max, nothing problematic, an rpi consumes half of that, we’re talking of 10€/year for the rpi or 20€/year for the other, you should add to the balance the stability, the speed, and many other factors

  5. This is what I had said in a differen t forum on a similar subject:
    One ought to understand the difference in between general purpose storage (e.g., offline, mainly) vs. active, production, reliability sensitive primary storage devices. Flash memory in latter use case usually has a built-in storage controller that is doing wear-leveling, bad sector isolation/remapping, (and often times cache controller.)
    SD card is the simplistic form of flash that missing most of these advanced features.
    Check most recent Android (offload apps and RAM cache) and Win OS restriction on that, it should tell a story.
    BTW, I actually hear people b***hing some low-end SBC using microSD as its boot/OS device (especially the swap slice is also on it) that work is lost due to no longer bootable, files lost, storage performance slowed down, etc.

  6. OK!
    Is there a route from the standard RPI installed with the script to HA combined with all those blocks in Node-RED to stay active and doing the heavy lifting?
    I have a small PC with W10 being workless(was for my wife’s activities, but she does everything on the phone now).
    I have second thoughts on my setup with RPI and SD cards. A lot more people will be interested in this conversion.

  7. Hi guys,
    Happy New Year,

    A RPi may looks cheap but when you look at all you need to add to make it reasonnable reliable :
    – A box (+10€)
    – A good power supply (+10€)
    – Tricks to run it from USB instead of (SD (+40..60€)
    – the time you are spending to try to make all these hacks together

    Just to run software ? No even take advantage of the HAT connector ?

    You have pretty good fanless PC like Z83 available from 80€ (gearbest) to 130€ (Amz) with :
    – Standard x86-64 core supported by standard debian/ubuntu/…
    – Up to 4GB DDR and 64GB mSATA
    – True ethernet (no USB hack)
    – True USB 3.0

    From my point of view, RPi3 has lost all its interest except on the workbench.
    The Pi Zero W still has some value when HW interface is required as it is easier to develop C/C#/python… (vs AVR/ARM/ESP)
    Only very low power sensors and hard-real-time may still requires mcu.

    Considering the time you are spending on it, isn’t it time to “industrialize” your setup ?

    All the best

      3. look on amazon, weekly there’s some z83 with 2gb ram+32gb emmc for 75€… i paid such, just look for the offers… i took an ap34, too (newer cpu), with 6gb ram and 64gb emmc and external antenna for just 131€ (normally costs 220)…

        you can even look for some more expensive intel NUC, which is natively supported by HASS.IO distro (aside from being possible to install on top of docker manually on a standard linux distro, of course)

        1. Currently $84 (US) on GearBest for the Z83-II :-

          https://www.gearbest.com/tv-box-mini-pc/pp_562950.html

          (there may be an extra $4 shipping charge, depending upon where you are).

          The Z83-II is the one I got a couple of years back and, after a bit of trouble getting the BIOS/UEFI combination to recognize anything except Windows, it’s been working flawlessly ever since. It currently has roughly 10TB of disks hanging off the USB3 port and backs up all of the other machines on the network, using two of its four cores to run xz compression real-time on the incoming data stream(s).

          It certainly isn’t a perfect machine, but it is pretty good value for the price (do make sure you get the correct power lead for your area though, as the plug and PSU is a one piece molding).

          See my (sometimes less than flattering) remarks on getting an open-source OS booted:-

          Z83-II Mini-PC (Quad core 8350)

          -John-

          1. Good morning John

            Ok, got the link – also EU plug available. The questions I have are; would we still expect issues running other than Windows 10? and what about 1-step backup if that is do-able without lots of R&D?

            I use Win10 every day and love it but I would not expect my PC to stay up, un-interrupted 24-7-365 under any circumstances. My PCs want to update something most weeks and Windows updates seem to require a reboot most times. So even if the Z83 is easy to set up with Debian and reliable, what about 1 step backup… one of my reasons for using RPi after playing with most other SBCs out there, apart from iffy support for the others, is in fact rpi-clone, now back to working fine incidentally. Meanwhile, over in Spain, my RPi (2 I recall) is operating flawlessly as it did all summer and has continued to since we left early October, running Node-Red, SQLITE and much more as well as Pi-VPN, PIHOLE etc… I regularly check in remotely. Thats a Pi with a £3 box and £5 PSU + battery, driving a bunch of ESP8266 boards and my Nano-peripheral. Should it eventually fail there’s an pi-cloned SD sitting on top of it, possibly a non-technical friend could replace… though I’d prefer an automated fall-over… not there yet..

            1. The Gearbest AD states that Windows will require manual updates but then we knew that – so using Windows 10 for IOT is out..

            2. through away windows, put ubuntu 18.04 with kvm or proxmox and use docker or virtual machines… so even backups are easy to do, using snapshots for both of them… having something “virtual” is the solution to backups, as you can always take the vm and let it start elsewhere, being hw independent…

              backup MUST be automatic and possibly local and remote…
              3.2.1 rule
              3 copies on at least 2 different media, 1 of which offline

            3. Pete,

              I think that the one-step backup is going to be your main issue. I do have unattended back-ups of the Z83 (although obviously not the 10TB of external disks) across the network to, as it happens, a Raspberry-Pi, but restores are a pain in the jacksie. You wouldn’t want to clone machines that way.

              For me, the z83 fits my needs and has been a good, reliable machine (far more reliable than the ARM-based predecessor was). It also runs Docker containers just fine …but it does have some problems with trying to run virtual machines (like I said, it’s not perfect, by any means).

              -John-

            1. Antonio,

              No, sorry. I wasn’t interested in the WiFi when I got it (I’ve got plenty of WiFi connected devices as it is), but one of the reasons I got it was the Gigabit ethernet. I’ve read elsewhere that the Gbe on this machine is limited, speed wise, but it was still a major improvement for me. I did run a couple of versions of Ian Morrison’s spins on various Linux flavours when I was still trying to get it to boot reliably, but I never tried the wireless connection, as I never planned to use either it or the bluetooth.

              Unfortunately, it’s also too busy nowadays to take down for experimenting. With the single exception of back-ups, I do have other systems that will pick up the load for all of the other services which the z83 provides, but other users tend to complain that “the internet is slow” when DNS has to timeout to one of the secondary servers.

              -John-

              2. so, installed ubuntu 18.10 (why stay back in release if i don’t need software compatibility as i plan to run docker and hassio on it, right?), to have better hardware support… installed with no issues, works everything EXCEPT wifi, i don’t even see the device in lspci or lsusb (don’t know how it’s connected, i tried both)… bluetooth is working, too… it seems very snappy with ubuntu desktop, better than windows 10… now i’m searching for wifi, but just as a test as i want my control systems to run wired…

                everybody can do this, as it’s just F7 on boot, select a usb dongle with ubuntu on it (burned using rufus with default settings)

                1. to be clear: you need F7 only while just powered the z83 on to get access to the boot menu that allows to boot from usb, to install ubuntu or just start whatever live system you want, you don’t need it once installed…

                  for my purposes, i could stay with ubuntu 18.10 even if wifi is not working, but if you buy a Z83 F or W model, on these links you’ll find ubuntu/lubuntu/xubuntu/mate (they only change the graphical interface, google them, lubuntu is my preferred as it’s windows like and very light, too) with all you need already up and running, included wifi: just pass them on a flash drive with rufus or etcher, boot from them, install, done, all working:
                  http://www.minisforum.com/showthread.php?tid=201
                  http://www.minisforum.com/showthread.php?tid=204

                  anyway, wifi is not that good, so you better buy an external dongle if you want more reliability… read here: http://www.minisforum.com/showthread.php?tid=411

                  this board is NOT for using the script, as not working on ubuntu > 16.04

                  2. confirmed dodgy wifi connection using built-in device… so just use latest ubuntu 18.10 if you want to be bleeding edge, or use one of the 18.04 iso in previous links… right now just installed docker on lubuntu 18.04 and installing hassio…

                    edit: holy cow, it’s FAST! Hassio installed in 2 or 3 minutes, against more than 15 on a RPi3+!

                    edit2: HOOOHOOO! Nodered addon installed and deployed in 2 minutes! Tomorrow i’ll test sqlite node in nodered and other stuff

                    1. @Barbudor

                      The AP34 looks like a bargain. According to a couple of reviews, the EMMC internal drive is slow, but there’s an M.2 slot inside, so you can add an M.2 drive internally, or just add an external HDD/SSD on one of the four USB3 ports (if you’re a Windows user, you apparently can’t easily activate Windows on either of those [M.2 or HDD], though).
                      CNX-Software say that it plays 4K video okay, but not 8K. Otherwise it looks really good at that price.

    3. indeed it’s what i’m going to do, i’ll use my sbcs for other stuff or just sell them, sick of limitations… i don’t use the 40 pin gpio so much, prefer remote wifi or rf sensors here and there, so even that point is not critical to me…

      docker is a beauty and makes all this stuff a breeze, and with hassio you don’t even need to know it exists… just need to choose if using a z83 or my udoo x86 which is doing nothing somewhere here…

      1. Hi Antonio,
        Are you publishing about what you are doing with docker?
        I started to work a bit with it 1.5y ago in my previous job but no more since I moved. I’d like to restart but lacking a real project to motivate myself.
        Thanks

        1. details here: https://tech.scargill.net/work-in-progress-using-docker/#comment-47407
          eventually i’ll add some addon i miss in hassio setup, other than that that’s what i’ll use from now on

          frenck, one of the most advanced home assistant developers, use to do some live streams where he does the addons and shows some docker tips… but they’re too long for me (even 7 hours), and full of disco music i personally hate… i skip those parts…

      1. All good stuff BUT can you backup the entire virtual setup with one click? Or do you have to back it up in stages (no good)?

          1. So download hass.io, 10 mins install, download samba add on, install, restore snapshot.. as against “sudo rpi-clone -U sda” click.

            1. you’re missing the crucial point: portability… rpi-clone works only on raspberry, and with sd storage, as you said… ha snapshots work in whatever setup you can run hassio onto, whatever storage, so you can even pass to a more efficient platform if needed, without having to reinstall and reconfigure every service… that’s the beauty of docker: abstract your data from the underlying hardware 🙂

    1. Been there, done that as you know – problem is rpi-clone won’t clone SATA drives so you’re back to square 1.

      1. well, then it’s time to move on and look for alternatives, like the home assistant docker setup: as i showed you, you can move your full setup from a raspberry to a virtual machine (completely different hw architecture) in a matter of half an hour… it’s pointless insisting in using a raspberry as it’s now, if it’s mined at its roots by an unreliable storage option, not to mention the memory limits (only 1gb) and connectivity one (shared bus between usb and eth)… a docker setup is virtually portable on every platform supported, it’s hardware agnostic…

        https://www.youtube.com/watch?v=vnie-PJ87Eg

Comments are closed.