We had quite a successful Ubuntu Global Jam on Saturday. Thanks Yola for hosting us. It was only 5 of us, and I was the only one with any uploads under my belt, but the nice small team meant I had time to help everyone fix some build failures. We had intended to focus on Scientific packages, but it was easier to just pick arbitrary build failures from the most recent archive rebuild.

Maia visited us briefly at lunch time and took some photos, thanks :) Sorry you couldn’t stay longer.

Some stats from the jam

Examples that we fixed together:

• sync libmatio to fix FTBFS
• fix FTBFS in pidgin-twitter

Kyle Williams:

• sync kcollectd to fix FTBFS
• sync k9copy to fix FTBFS
• fix kx11grab FTBFS
• fix kon2 FTBFS

Marco Gallotta:

• fix pysatellites FTBFS
• fix ratpoison FTBFS

Michiel Baird:

• Sync camlimages to fix advi FTBFS
• Sync libsdl-sound1.2 to fix asc FTBFS

I think that’s pretty good for a day’s work :P

In other news

Eep, haven’t blogged an ages. Since I last blogged, I’ve become a Debian Developer, made an Ibid point release, been to (my first) UDS and a Debconf (which were both fantastic, and very different), spent a week sailing in Croatia with my brother, and been pulled into the Ubuntu release team. All this has meant little progress in my MSc, though :/

Our Ubuntu LoCo release parties always end up being part-install-fest. Even when we used to meet at pubs in the early days, people would pull out laptops to burn ISOs for each other and get assistance with upgrades.

As the maintainer of a local university mirror, I took along a mini-mirror to our Lucid Release party and will be doing it for the Maverick party tomorrow. If anyone wants to do this at future events, it's really not that hard to organise, you just need the bandwidth to create the mirror. Disk space requirements (very rough, per architecture, per release): package mirror 50GiB, Ubuntu/Kubuntu CDs 5GiB, Xubuntu/Mythbuntu/UbuntuStudio CDs 5GiB, Ubuntu/Kubuntu/Edubuntu DVDs: 10GiB.

I took the full contents of our ubuntu-archive mirror, but you can probably get away with only i386 and amd64 for the new release people are installing and any old ones they might be upgrading from. You can easily create a partial (only selected architectures and releases) Ubuntu mirror using apt-mirror or debmirror. It takes a while on the first run, but once you have a mirror, updating it is quite efficient.

The CD and DVD repos can easily be mirrored with rsync. Something like --include '*10.04*'.iso' --exclude '*.iso' will give you a quick and dirty partial mirror.

As to the network. I took a 24port switch and a pile of flyleads. A laptop with 1TB external hard drive ran the mirror. At this point, you pick between providing Internet access as well (which may result in some poorly-configured machines upgrading over the Internet) or doing it all offline (which makes sense in bandwidth-starved South Africa). For lucid, I chose to run this on a private network - there was a separate WiFi network for Internet access. This slightly complicates upgrades because update-manager only shows the update button when it can connect to changelogs.ubuntu.com, but that's easily worked around:

Instead of getting people to reconfigure their APT sources (and having to modify those meta-release files), we set our DNS server (dnsmasq) to point all the mirrors that people might be using to itself. In /etc/hosts:

Dnsmasq must be told to provide DHCP leases /etc/dnsmasq.conf: :

Then we ran an Apache (all on the default virtualhost) serving the Ubuntu archive mirror as /ubuntu, the meta-release files in the root, and CDs / DVDs in /ubuntu-releases, /ubuntu-cdimage. There were a couple of other useful extras thrown in.

We could have also run ftp and rsync servers, and provided a netboot environment. But there was a party to be had :)

For the maverick party, I used this script to prepare the mirror. It's obviously very specific to my local mirror. Tweak to taste:

We'll see how it works out tomorrow. Looking forward to a good party.

In line with my SysRq Post comes another bit of assumed knowledge, SMART. Let’s begin at the beginning (and stick to the PC world).

Magnetic storage is fault-prone. In the old days, when you formatted a drive, part of the formatting process was to check that each block seemed to be able to store data. All the bad blocks would be listed in a “bad block list” and the file-system would never try to use them. File-system checks would also be able to mark blocks as being bad.

As disks got bigger, this meant that formatting could take hours. Drives had also become fancy enough that they could manage bad blocks themselves, and so a shift occured. Disks were shipped with some extra spare space that the computer can’t see. Should the drive controller detect that a block went bad (they had parity checks), it could re-allocate a block from the extra space to stand in for the bad block. If it was able to recover the data from the bad block, this could be totally transparent to the file-system, if not the file-system would see a read-error and have to handle it.

This is where we are today. File-systems still support the concept of bad blocks, but in practice they only occur when a disk runs out of spare blocks.

This came with a problem, how would you know if a disk was doing ok or not? Well a standard was created called SMART. This allows you to talk to the drive controller and (amongst other things) find out the state of the disk. On Linux, we do this via the package smartmontools.

Why is this useful? Well you can ask the disk to run a variety of tests (including a full bad block scan), these are useful for RMAing a bad drive with minimum hassle. You can also get the drive’s error-log which can give you some indication of it’s reliability. You can see it’s temperature, age, and Serial Number (useful when you have to know which drive to unplug). But, most importantly, you can find out the state of bad sectors. How many sectors does the drive think are bad, and how many has it reallocated.

Why is that useful?

In the event of a bad block, you can manually force a re-allocation. This way it happens under your terms, and you’ll know exactly what got corrupted.

Next, Google published a paper linking non-zero bad sector values to drive failure. Do you really want be trusting known-non-trustworthy drives with critical data?

Finally, there is a nasty RAID situation. If you have a RAID-5 array with say 6 drives in it and one fails either the RAID system will automatically select a spare drive (if it has one), or you’ll have to replace it. The system will then re-build on the new disk, reading every sector on all the other disks, to calculate the sector contents for the new disk. If one of those reads fails (bad sector) you’ll now be up shit-creek without a paddle. The RAID system will kick out the disk with the read failure, and you’ll have a RAID-5 array with two bad disks in it — one more than RAID-5 can handle. There are tricks to get such a RAID-5 array back online, and I’ve done it, but you will have corruption, and it’s risky as hell.

So, before you go replacing RAID-5 member-disks, check the SMART status of all the other disks.

Personally, I get twitchy when any of my drives have bad sectors. I have smartd monitoring them, and I’ll attempt to RMA them as soon as a sector goes bad.

I’m constantly surprised when I come across long-time Linux users who don’t know about SysRq. The Linux Magic System Request Key Hacks are a magic set of commands that you can get the Linux kernel to follow no matter what’s going on (unless it has panicked or totally deadlocked).

Why is this useful? Well, there are many situations where you can’t shut a system down properly, but you need to reboot. Examples:

• You’ve had a kernel OOPS, which is not quite a panic but there could be memory corruption in the kernel, things are getting pretty weird, and quite honestly you don’t want to be running in that condition for any longer than necessary.
• You have reason to believe it won’t be able to shut down properly.
• Your system is almost-locked-up (i.e. the above point)
• Your UPS has about 10 seconds worth of power left
• Something is on fire (lp0 possibly?)
• …Insert other esoteric failure modes here…

In any of those situations, grab a console keyboard, and type Alt+SysRq+s (sync), Alt+SysRq+u (unmount), wait for it to have synced, and finally Alt+SysRq+b (reboot NOW!). If you don’t have a handy keyboard attached to said machine, or are on another continent, you can

In my books, the useful SysRq commands are:

b

Reboot

f

Call the oom_killer

h

Display SysRq help

l

Print a kernel stacktrace

o

Power Off

r

Set your keyboard to RAW mode (required after some X breakages)

s

Sync all filesystems

u

Remount all filesystems read-only

0-9

Change console logging level

In fact, read the rest of the SysRq documentation, print it out, and tape it above your bed. Next time you reach for the reset switch on a Linux box, stop your self, type the S,U,B sequence, and watch your system come up as if nothing untoward has happened.

Update: I previously recommended U,S,B but after a bit of digging, I think S,U,B may be correct.

I’ve written about this before, but Ctrl-Alt-workspace switching key-presses nail me routinely.

Let’s go through some history:

We have Ctrl-Alt-Delete, the “three-fingered-salute”, meaning reboot, right? That combination was designed to NEVER be pressed by accident. And it never used to be.

The X guys needed a kill-X key-press, as things can sometimes get broken in X. So they chose Ctrl-Alt-Backspace, which is also a pretty sensible combination. It’s very similar to Ctrl-Alt-Delete, so we remember it, and backspace has milder connotations than delete, so we understand it to mean that it’ll only kill a part of the system.

X also has some other Ctl-Alt- shortcuts. Some of these are also suitably obscure, i.e. NumPad+ and NumPad-. Others like Ctrl-Alt-F1 mean change to virtual console 1. That one might do by accident, if you are an old WordPerfect user, but should be safe enough otherwise. They were designed to look like big brothers of (and even work as) standard VT-changing behaviour.

For changing workspaces, Alt-F1 style key-presses were used, mimicing VT-changing key-presses. This is great for *nix users, but people coming from Windows expect Alt-F4 to close a program, not take them to workspace 4.

So GNOME came along, and decided that instead, they’d use Ctrl-Alt-Arrow key-presses to change workspace. That’s fine, but it’s a pretty common action, so I’m often holding down Ctrl-Alt without even thinking about it. If I start editing something and press delete/backspace, before I’ve released Ctrl-Alt, boom! And I run screaming and write a blog post.

Now, I know that Ctl-Alt-{Delete,Backspace} can be disabled (even if the latter is a little tricky to do), but I’d really like to change them. I like to be able to kill X without using another machine and ssh, I just don’t like this to happen by accident. And no, the solution isn’t for me to change my workspace-changing keys, because this problem must affect every GNOME user, not just me.

Dangerous key-presses should be really unlikely key-presses. Alt-SysRq- key-presses are good in this regard, they’ll always be unlikely. (Oh, and they are insanely useful.)