The joy that is SysRq
I’m constantly surprised when I come across long-time Linux users who don’t know about SysRq. The Linux Magic System Request Key Hacks are a magic set of commands that you can get the Linux kernel to follow no matter what’s going on (unless it has panicked or totally deadlocked).
Why is this useful? Well, there are many situations where you can’t shut a system down properly, but you need to reboot. Examples:
- You’ve had a kernel OOPS, which is not quite a panic but there could be memory corruption in the kernel, things are getting pretty weird, and quite honestly you don’t want to be running in that condition for any longer than necessary.
- You have reason to believe it won’t be able to shut down properly.
- Your system is almost-locked-up (i.e. the above point)
- Your UPS has about 10 seconds worth of power left
- Something is on fire (
lp0possibly?) - …Insert other esoteric failure modes here…
In any of those situations, grab a console keyboard, and type Alt+SysRq+s (sync), Alt+SysRq+u (unmount), wait for it to have synced, and finally Alt+SysRq+b (reboot NOW!). If you don’t have a handy keyboard attached to said machine, or are on another continent, you can
In my books, the useful SysRq commands are:
- b
- Reboot
- f
- Call the oom_killer
- h
- Display SysRq help
- l
- Print a kernel stacktrace
- o
- Power Off
- r
- Set your keyboard to RAW mode (required after some X breakages)
- s
- Sync all filesystems
- u
- Remount all filesystems read-only
- 0-9
- Change console logging level
In fact, read the rest of the SysRq documentation, print it out, and tape it above your bed. Next time you reach for the reset switch on a Linux box, stop your self, type the S,U,B sequence, and watch your system come up as if nothing untoward has happened.
Update: I previously recommended U,S,B but after a bit of digging, I think S,U,B may be correct.
Comments
Raising Elephants
You can actually do a bit better by pressing Alt+SysRq+e, which sends SIGTERM to all processes. Processes which are still working can still terminate gracefully instead of being killed. There is a mnemonic to help remember the sequence.
I always thought it was
I always thought it was supposed to be S,U,B?
re: S, U, B
You want to sync() the unmounted filesystem. If you sync it before it's unmounted, then the version on disk isn't guaranteed to be consistent.
S,U,B
Surely though if you wanted to sync data, you'd have to be able to write to the disk?
Yep, that's the way I know
Yep, that's the way I know it. You can't sync to an unmounted disk, as far as I'm aware. Once it's unmounted, it's totally unusable. You have to sync first. REISUB.
The only cases where I ever think of needing this, it turns out I can't use them anyway because it's panicked.
S, U vs U, S
I knew about "Raising Skinny Elephants Is Utterly Boring" for quite a while, and it does S before U. So are all the other similar elephant mnemonics listed on the Wikipedia page.
Are you sure data can be synced to a read-only filesystem?
REISUB
I've been a devotee of the R-E-I-S-U-B sequence for some time now, but it seems I'll have to change it to the less-pronounceable REIUSB. Amusingly some videos, when played in vlc, consistently cause my system to lock up so badly that the whole sequence goes unheeded.
Looks like I could be wrong here
I need to do some research on this. From my understanding of the sync() call, it syncs all write-cached data, not necessarily filesystems.
I guess I need to dig deeper / do some testing...
Research
I’ve been digging into the kernel (thanks Linux Cross Reference). While I’m a kernel newbie, and don’t know much about the structures Linux uses for this, it looks to me like you should sync first, yes.
emergency_remount() calls fsync_super(), which calls sync_inodes_sb(), which looks like it may sync the file-system.
However, emergency_sync() calls sync_filesystems() which won’t work if they’ve been remounted RO (it checks).
I’m not a kernel developer.
Sync
According to sysrq.txt,
's' - Will attempt to sync all mounted filesystems.which means that it doesn't do anything for unmounted filesystems. Also from sysrq.txt: "I generally 'S'ync, 'U'mount, then re'B'oot when my system locks." I would guess that remounting a filesystem read-only includes syncing everything to disk.
U
Please note that 'u' doesn't actually unmount any filesystems, it just remounts them R/O. That said, it certainly seems like an emergency_sync() after emergency_remount() is useless; it might also be useless before-hand, but at least it'll actually do something that way.
Post new comment