Raspberry Pi hanging on reboot

Peter ScargillUncategorized14 Comments on Raspberry Pi hanging on reboot

PiOk, this is a favour for my pal David Miles. He’s struggling with a PI which stops right at the last minute when powering down or rebooting – as per the image here.

I’ve never had this trouble so I’m not a good one to comment – anyone struggled and overcome this issue? If so, if you leave comments in here I’m sure he’ll spot them and respond accordingly.

And before you ask, no – still no sign of the little ESP8266/Nextion boards we ordered from China to play with.  I think they’re all on holiday over there.

14 thoughts on “Raspberry Pi hanging on reboot”

  1. I ran into a problem and found this site. David could you tell me if you left the pi in that state for over 12 hours to see if it would “work itself out”?

  2. Just to bring this story to a close…..

    I decided yesterday that the easiest/quickest solution was simply to rebuild the Pi from scratch. Downside is I’ll never know what caused the problem but on the plus side it only took a couple of hours and everything now works perfectly.

    Thanks again to those who offered advice.

  3. Okay, good; that means we are still looking at something in the shutdown sequence.

    There are two things in your listing. The obvious one is, as you say, the SIGRTMIN line. This is (I think!)
    systemd itself handling its own thread processes. From the manual page for systemd:-
    SIGRTMIN+24 Immediately exits the manager (only available for --user instances).

    I don’t have Raspbian and I’m not running systemd, so I can’t really help with this one. Please do check with the Raspbian forum for expert knowledge (I doubt that very many people are viewing this blog topic any longer).

    The other issue in your listing is that the NTP daemon indicates that it’s starting, which is bad. It should be stopping. You might want to try stopping the NTP daemon manually from the command line and then do a shutdown afterwards to see whether that is indeed the culprit. Try using:-
    sudo systemctl list-units
    to find the correct name for NTP (it could be “ntp” or “ntpd”, depending on your system).

    Once you know the name, you can do:-
    sudo systemctl stop NTP-NAME

    That should give you some output confirming that NTP has been shut down. After that, try shutting down the system and see if it still hangs. If it does hang, take careful note of the point at which it stopped this time round.

    You can also try running the command systemd --test when your system is running normally (you may have to use sudo for this, too). It should (according to the manual page) give you a listing of the start-up sequence. The shutdown sequence should be the reverse of that listing, which should indicate what process would be next in line from NTP and give you another clue.

    If you’re still not getting anywhere after that, your best bet is to float the question on the Raspbian and/or Rasberry-Pi forums to get answers from people with actual experience of this OS.

    Good luck!

    -John-

    1. Thanks, John. Really helpfil. I’ll give all that a try tomorrow or next weekend and see what happens.

      I did post this on the Pi forums at the outset as well but have had far more feedback here than on there.

      Will post there again if your ideas don’t get me anywhere.

      Thanks again.

  4. David,

    Again, just to make sure that we’re all on the same page, you say “…around the time I had another failed shutdown”, which implies that it doesn’t fail every time you try to shut down. Is that the case?

    -John-

  5. Ok. Glad we’re on the same wavelength now. 😉

    I’ve just done a shutdown and after it hung and I powered off/on I looked at the daemon.log file and found this section which I assume covers the incomplete shutdown and then the first part of the manual restart:

    Feb 14 17:47:25 raspberrypi systemd[11116]: Stopping Default.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Stopped target Default.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Stopping Basic System.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Stopped target Basic System.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Stopping Paths.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Stopped target Paths.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Stopping Timers.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Stopped target Timers.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Stopping Sockets.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Stopped target Sockets.
    Feb 14 17:47:25 raspberrypi systemd[1]: Stopping Multi-User System.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Starting Shutdown.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Reached target Shutdown.
    Feb 14 17:47:25 raspberrypi systemd[11116]: Starting Exit the Session…
    Feb 14 17:47:25 raspberrypi systemd[1]: Stopping Login Prompts.
    Feb 14 17:47:25 raspberrypi systemd[1]: Stopped target Login Prompts.
    Feb 14 17:47:25 raspberrypi systemd[1]: Stopping Serial Getty on ttyAMA0…
    Feb 14 17:47:25 raspberrypi systemd[1]: Stopping LSB: triggerhappy hotkey daemon…
    Feb 14 17:47:25 raspberrypi systemd[1]: Stopping LSB: Start NTP daemon…
    Feb 14 17:47:25 raspberrypi systemd[11116]: Received SIGRTMIN+24 from PID 11173 (kill).
    Feb 14 17:47:38 raspberrypi systemd-modules-load[59]: Inserted module ‘fuse’
    Feb 14 17:47:38 raspberrypi systemd-modules-load[59]: Inserted module ‘i2c_dev’
    Feb 14 17:47:38 raspberrypi systemd[1]: Mounted Configuration File System.
    Feb 14 17:47:38 raspberrypi systemd[1]: Mounted FUSE Control File System.
    Feb 14 17:47:38 raspberrypi systemd[1]: Started Create Static Device Nodes in /dev.
    Feb 14 17:47:38 raspberrypi systemd[1]: Started Apply Kernel Variables.
    Feb 14 17:47:38 raspberrypi systemd[1]: Starting udev Kernel Device Manager…
    Feb 14 17:47:38 raspberrypi systemd[1]: Started udev Kernel Device Manager.
    Feb 14 17:47:38 raspberrypi systemd[1]: Starting Copy rules generated while the root was ro…
    Feb 14 17:47:38 raspberrypi systemd[1]: Starting LSB: Set preliminary keymap…
    Feb 14 17:47:38 raspberrypi systemd[1]: Starting LSB: Tune IDE hard disks…
    Feb 14 17:47:38 raspberrypi systemd[1]: Started Copy rules generated while the root was ro.
    Feb 14 17:47:38 raspberrypi systemd-fsck[54]: e2fsck 1.42.12 (29-Aug-2014)
    Feb 14 17:47:38 raspberrypi systemd-fsck[54]: root: clean, 160477/3751936 files, 1243521/14995456 blocks
    Feb 14 17:47:38 raspberrypi fake-hwclock[58]: Sun 14 Feb 17:47:27 UTC 2016
    Feb 14 17:47:38 raspberrypi hdparm[94]: Setting parameters of disc: (none).
    Feb 14 17:47:38 raspberrypi systemd[1]: Started LSB: Tune IDE hard disks.
    Feb 14 17:47:38 raspberrypi keyboard-setup[93]: Setting preliminary keymap…done.

    So do you agree that this is the final item prior to the crash?

    Feb 14 17:47:25 raspberrypi systemd[11116]: Received SIGRTMIN+24 from PID 11173 (kill).

    If so, what does that mean and how do I fix it?

    To complicate matters further I found something in the log from last weekend which I think is around the time I had another failed shutown, and that time the last message in the log was

    Stopping D-Bus User Message Bus…

    which would mean it is hanging at a different point each time…..

    Any ideas please?

  6. David,
    This is normally caused by an “rc” script which isn’t returning control to the parent process (on start-up it’s not going into background and on shutdown it isn’t terminating).
    The first and easiest check …what did you just add to the rc queue (either directly or indirectly – ie:- adding a new package).
    The next easiest is to boot into single user mode and check the modification times in the rc tree.
    Either way, you need to boot into single user to fix the issue.

    P.S. – For a more specific answer, more information would be useful. What are you running on your Pi and what specific revision is it. I’m sure there’s someone who checks in here who probably has an identical set up and can give more specific troubleshooting tips.

    1. Thanks for this.

      I’m still very new to Linux so a lot of what you’ve said is going over my head I’m afraid.

      I don’t know exactly when this problem started but I think it may have coincided with me adding a USB Ethernet adapter to the Pi Zero.

      Can you tell me how to boot into single user mode (never heard of that at all) and how to check the rc tree please?

      The config is a Pi Zero with all latest updates applied via apt-get update and upgrade, and the latest firmware applied with raspi-update. Software-wise it’s got Jessie with Node Red. And nothing else apart from Weaved.

      2. David,
        Applying the normal “what has changed?” troubleshooting logic, lets assume that Weaved (whatever it is) was the last thing you added and that’s what’s causing the problem. Use the info from the stackexchange link which Kris provided (below) to get into single user mode, then cd into /etc. I’m not familiar with Raspbian, but you should see either an rc.d or an init.d directory in /etc. These directories contain the start-up scripts for processes which are started automatically at boot time.
        You can list the entries in whichever of those directories you have with ls -alrt to give a listing by update time (the last item in the list was the last file which was updated (added or edited). The chances are very, very good that the last file in the list is the one which is causing you problems (and also that it will be something to do with Weaved). If the last file in the listing has a modified date which is consistent with all of the other files in the directory, then all bets are off and you probably shouldn’t change anything. However, if the modification time is markedly more recent than the others (ie:- from the day or day before your problem first appeared), then you’re probably on safe ground and can move that file to one side.
        Use something like mv filename ._filename_ORIG to move the offending file to a name with a leading “.” character, so that it isn’t recognized as a valid script file by the start-up process (you could also move the file out of that directory completely to, say, /root if you want to …but -don’t- move it to the /tmp directory, as /tmp is automatically emptied on reboots).
        Go back to multi-user config and reboot.

        A further note on rc.d/init.d – Generally the files in those directories were executed in simple, alphabetical listing order. However, nowadays most operating systems have some sort of method to ensure dependencies are met (ie:- the network needs to be running before some scripts will execute), so an alphabetic listing may not give you any help (anyone want to chime in here with info on Raspbian?). If you look through the files though, it shouldn’t take you too long to find which one is the last one to be successfully executed (egrep Volatile * is a good starting point) and then you’ve probably narrowed down the list of culprits to a file with a modification time which is newer than the file containing the “Create Volatile” text. Deerstalker on, pipe clamped firmly between teeth… 🙂

        1. I’ll deal with the easy bit first – it definitely is not Weaved that is causing the problem as that was installed ages ago and log before the shutting down problem began.

          I haven’t booted in single user mode yet as the articles didn’t seem to agree on the correct way of doing it. Also it sounded like that was only required for systems that would not boot up.

          So what I have done so far is gone into /etc/init.d and run ls -alrt.

          The ONLY files modified this year (all on 31 Jan) are:

          .depend.start
          .depend.stop
          .depend.boot

          Prior to that the last changed file was raspi-config back in October.

          The .depend.stop file contains the following:

          TARGETS = dhcpcd fake-hwclock triggerhappy avahi-daemon lightdm plymouth urandom alsa-utils sendsigs rsyslog umountnfs.sh rpcbind nfs-common hwclock.sh networking umountfs umountroot halt reboot
          sendsigs: plymouth avahi-daemon lightdm alsa-utils triggerhappy
          rsyslog: avahi-daemon sendsigs
          umountnfs.sh: plymouth avahi-daemon rsyslog lightdm alsa-utils sendsigs triggerhappy
          networking: umountnfs.sh rpcbind
          umountfs: plymouth avahi-daemon dhcpcd networking urandom umountnfs.sh hwclock.sh rpcbind lightdm alsa-utils triggerhappy
          umountroot: umountfs fake-hwclock
          halt: umountroot
          reboot: umountroot

          On the basis of all that, is it still worth trying to boot in single user mode? Would that make any difference to the contents of the init.d directory?

          Does what is in the .depend.stop file help narrow down what is causing the problem?

          1. Ah-hah! Delete all of the above! 🙂 One slightly ambiguous sentence sends us all off the tracks. I took “when powering down or rebooting” to mean stopping -and- starting. I didn’t realize that you could boot up normally.

            Forget trying to boot into single user mode.

            Look in the file /var/log/daemon.log to get a chronological listing of start-up and shut-down. There may well be an error message in there from the offending process, but it will show you the exact shut-down sequence from previous sessions too.

            If that doesn’t give you any clues then you might want to try the Raspbian forum for people who are more familiar with that OS and systemd.

Comments are closed.