Archives: October 2007

An update on Laptop Hard Drives & Linux

Launchpad bug 59695 has been gathering a huge amount of activity since I wrote about this issue. The issue seems to be that the hardware manufacturers (BIOS and HDD firmware) set very aggressive values for power management. And every other OS (Windows & Mac OSX) override these values to something more sane. The manufacturers only test their equipment in Windows, so they don’t see any problems :-)

This kind of thing seems to happen to Linux quite regularly - we all remember the ACPI debacle caused by manufacturers using Microsoft’s broken ASL compiler, which worked in Microsoft’s broken ACPI environment (or was overridden with driver updates).

So in my opinion, Ubuntu (and every other distributor) has to step in and override these aggressive settings. And, by the look of the bug report, Gnome Power Manager should provide the user with a slider to set the balance between power savings and hardware lifetime.

There is already the obligatory ubuntu is killing your Hard Drive blog, and a closed for cooling off Ubuntu forums thread, if you want get involved and start swinging your battle-axe.

Oh, and the workaround I posted last week obviously doesn’t cover the case of the machine resuming from suspend. You have to use an ACPI event script for that.

A serious warning to Linux Laptop users

I’ve just come across a rather scary, and worryingly old launchpad bug, which talks about real hardware damage. There is more on the problem here. But basically, by default, Linux is far too optimistic with spinning laptop hard drives down, and you can reach number of spin-up/downs that your drive is rated for over it’s entire life-time, in a few months.

My laptop (3 months old), is already at 160000 Load/Unloads: Around half it’s rated life.

The easy solution is

# hdparm -B 180 /dev/sda

Or, the following in /etc/hdparm.conf

/dev/sda {
    apm = 180
}

Lets hope that this gets resolved soon, or the problem isn’t as bad as it appears.

I see that Matthew Garrett (the Ubuntu Laptop Tzar) is subscribed to this bug, but doesn’t seem to have commented on it. I find that a little odd, considering its seriousness.

N800 (overdue) review

I was lucky enough to win a Nokia N800 at LugRadio Live 2007, because I’d come from South Africa. I’ve had it for 3 months now, and the successor was announced last week, so I think it’s time to blog about it, properly.

Hardware

N800

The hardware is pretty decent. The screen is very bright and vivid (great for photos), and just big enough to read websites comfortably. There are enough buttons to fulfil the basic tasks without reaching for the stylus, but you generally work it with the stylus (or a finger). It has quite a few hardware features:

  • A periscope webcam (640x480) that pops out the left side, and can swivel 200°
  • 2 SD sockets (one in battery compartment, for more permanent storage, and one external)
    • It ships with a 128MiB µSD card and SD adaptor. I wish manufacturers would ship with decent sized (say 2GiB) cards…
  • A built-in FM radio (requires headphone lead for antenna)
  • WiFi (802.11g)
  • Bluetooth
  • Stereo Speakers (very tinny)
  • Microphone
  • Headset socket (ships with a wired stereo hands-free set)
  • Standard Nokia charger (I wish it charged by USB)
  • mini-USB-B port. It has the hardware to be used in USB OTG mode, but this requires a custom kernel and custom cables
  • A (slightly flimsy) fold-out standing-arm
  • A spare stylus

The battery life is comperable to my cellphone (about a week of non-use, or a day or two of use), and I’m happy with most of the hardware, but I do have a few issues:

The power button isn’t sufficiently recessed for it’s soft carrying case. This means that I can’t carry it off in my bag, it’ll turn itself on eventually, and run it’s battery flat from spurious touch-screen clicks. The solution is to always leave it on, and screen-locked (it supports an auto-lock). For a cellphone manufacturer, I’d have thought they’d get this right.

The webcam’s position at the far left means it gets a good view of the left side of your face. This can be a little disconcerting.

The USB port is (without some serious hackery) only in peripheral mode. I’d really like to be able to plug a USB keyboard into this device (bluetooth keyboards are way too expensive).

There’s a bottleneck in the system (processor presumably), that stops it playing youtube videos at full framerate. Mplayer seems to just be able to handle QVGA video at 24fps, but nothing more.

# cat /proc/cpuinfo
Processor   : Some Random V6 Processor rev 2 (v6l)
BogoMIPS    : 320.37
Features    : swp half thumb fastmult vfp edsp java
CPU implementer     : 0x41
CPU architecture: 6TEJ
CPU variant : 0x0
CPU part    : 0xb36
CPU revision        : 2
Cache type  : write-back
Cache clean : cp15 c7 ops
Cache lockdown      : format C
Cache format        : Harvard
I size              : 32768
I assoc             : 4
I line length       : 32
I sets              : 256
D size              : 32768
D assoc             : 4
D line length       : 32
D sets              : 256

Hardware    : Nokia N800
Revision    : 24202524
Serial              : 0000000000000000

Software

The software stack of the N800 is everything I could desire. It runs a Debian derivative of Linux, Maemo. Maemo uses proper Debian package management, the GTK widget set (with addons), Telepathy for IM, and Gstreamer for media. This makes it a doddle to port existing X applications to the N800.

When you first turn on the N800, after unboxing (or reflashing), you go through an install wizard. It sets the hostname, timezone, and pairs with your bluetooth cellphone. The bluetooth phone pairing is well thought out, and beats anything I’ve ever come across on any platform. Unfortunately it suffers from the same problem as Windows - it doesn’t set your Home Town, etc. based on the time-zone.

The default web browser is Opera-based, but a Gecko engine is available, and a WebKit one remoured to be on the way. It has a (proprietary) flash plugin, so you can watch YouTube, and the CACert.org root certificate is pre-installed :-)

The device has 4 input modes:

  • Bluetooth keyboard (or something like Russ Nelson’s chording bluetooth keyboard, which I came across at OSCON, and is truly a sight to behold)
  • Stylus on-screen keyboard. I find this frustrating, but at least it has predictive-text.
  • Full screen, thumb on-screen keyboard. This is better, but switching between pages (case, numerals, symbols) gets tiring. It’s launched by pressing on a text field with a finger, this only works about 60% of the time.
    • A user has prototyped an iPhone-like keyboard for the N800 (in python), which works very nicely. But clearly the Maemo input system doesn’t allow plugins, so it remains a prototype.
  • Handwriting recognition. I’m a P910i user, so this fits well with me, but I took a while to get used to writing inside the input area, rather than all over the screen. The handwriting recognition is good, and trainable, which is very nice. But I find the input area too small to reliably input in the correct case. It can’t decipher cursive (but then nobody can decipher my cursive), and some symbols don’t input easily. All this is forgiven by the ability to correct the handwriting recognition, by writing over the incorrect letter.

I find the input fine, although slow, for everything except passwords. Multi-case, symbol-laden passwords really bring out the worst of all the input systems.

The default software selection is passable, but not great. The device really ships with “internet tablet” software, with a few decent games thrown in. The major problems are the media player doesn’t play oggs or video, and the e-Mail program’s IMAP support, which is a joke, at best (It uses IMAP as if it was POP).

But I enabled a few extra repositories and soon my N800 became a really cool device:

  • Camera (by default, the web-cam is only used by the “Internet Call” software)
  • There is a port of Claws, which is a real IMAP client
  • Fmradio (yes it isn’t usable by default)
  • ipython (the maemo Hildon GUI library has python bindings)
  • maemo-mapper. It’s hard to tell the functionality of this without a GPS, but it supports all the popular online maps (for imagery), and uses OpenStreetmap by default, which turns many geeks’ eyes.
  • maemo-recorder - A sound recorder.
  • maemo-wordpy - A WordPress client.
  • Mediastreamer - A UPNP media client. I’ve tried this with MediaTomb and GMediaServer, but in both cases, connections seemed to die prematurely. I think this is a known bug with libupnp.
  • Mplayer - I used my N800 to watch movies on cheap transatlantic flights. It’s battery life is much better than a laptop, although the screen is small.
  • OpenSSH (client and server) - naturally :-)
  • rdesktop and vncviewer - These impress geeks, while also being useful
  • An X terminal is a must. The maemo-hackers edition is the one to get, it has a Ctrl-key…
  • Skype is installable with one click. The N800 makes a great skype phone (assuming you have WiFi, which is rare to find in this country, but in Portland OR it was the perfect device to have in my back pocket)

My repositories:

http://catalogue.tableteer.nokia.com/certified/ bora user
http://catalogue.tableteer.nokia.com/non-certified/ bora user
http://repository.maemo.org/ bora free non-free
http://repository.maemo.org/extras/ bora free non-free
http://www.claws-mail.org/maemo/ bora user
http://maemo-hackers.org/apt bora main

What’s the point

It’s a cool device for a geek to own, no doubt, but what’s the actual point of it?

I can answer that in a few ways:

  • Firstly, it’s screen’s bigger than my cellphone, but smaller than a laptop, so it’s good for goofing off during university lectures, and surfing the web.
  • It plays YouTube. All my machines are AMD64, and don’t run proprietary flash, so they don’t. I can’t say I use this much, but when I’m forced at gunpoint to watch a YouTube video, I can do it.
  • It’s nice for looking at photos. Bigger than a camera / phone, and supporting bluetooth and SD.
  • I don’t own a portable video player, so this is my portable video player.
  • When you can’t get to sleep at night, and suddenly something comes to you that you meant to find out about on Wikipedia or the web, you can grab it from the bedside table and do so. (Yes, I’m an unashamed geek)
  • My cellphone is notoriously unreliable, and this doesn’t make for a good alarm clock. So the N800 got pressed into this service. However, it’s alarm system seems to have a bad bug, that can semi-brick it for a week. I haven’t got to the bottom of it. But sometimes (and only when you have alarms set), it won’t boot for a week. You either have to re-flash it, or wait for it to suddenly wake up and make alarm noises.
  • If I want to see if there’s WiFi somewhere, it’s a lot quicker to check on the N800, then to take out a laptop.

But in the end, all that matters is that it runs Linux , and has Python dammit, so it’s a piece of cake to write any software for it that I want to. I think that reason on it’s own makes it a device worth owning.

And I thought our Ubuntu mirroring was a success

Due to archive.ubuntu.com’s saturated links atm, I switched to using mirror.ac.za as our upstream, but forgot to take their different URL scheme into account. We don’t use rsync’s —max-delete option on debian mirrors, so voom, our entire ubuntu archive mirror vanished. I’m resyncing it UWC, who we have a 10Mbps wireless link to…

Disk Usage Graph

Congratulations on Ubuntu 7.10

Congratulations to the Ubuntu team, on a successful gutsy release. While gutsy is currently a little buggy for me, the millions of eyes will hopefully find all the bugs responsible… (and I must get around to filing the relevant launchpad bugs)

I run a teeny weeny little mirror (by International standards, for South Africa, it’s OK), so I’ve followed the release process, and been hanging out on #ubuntu-mirrors since last night. Preparing an Ubuntu release is quite an undertaking. In total, each mirror needs to carry around 20-30 CD images, and 4 DVD images. That’s reasonable chunk of data, and it takes a lot of coordination for everybody to get it.

I see many distros release, we normally pick them up automatically, and notice their presence the next day. But there’s definitely something special about Ubuntu releases. They have the feel of a release. The tension builds up the night before, and the #ubuntu-release-party channel fills up (when I popped in, around 500 people). Then, a few hours before the release is announced, people start noticing that it’s on the mirrors. Pointing this out on the release-party channel is not allowed (it would disturb the otherwise rowdy party of 500 users pressing F5 every 10 seconds), but pretty quickly, before the release is even announced, all the mirrors that carry Ubuntu CDs get flat-lined. If they don’t, then it’s a sign that their hardware isn’t up to scratch, and they have to find & fix their bottlenecks. On the #ubuntu-mirrors channel, you can see sysadmins from around the world showing off graphs of flat-lined, multi-gigabit links and sharing server tuning tips. If you want any experience in widely-distributing large files, run an ubuntu mirror at release-time, you’ll gain the experience fast.

I don’t know if this massive assault means Ubuntu is the most popular distribution out there. Most Ubuntu users don’t need the new ISO. They rather need click the big “upgrade” button, and hammer their local mirror to the tune of about a gigabyte. Ubuntu is based on Debian’s awesome package management system, that (if used correctly) should never require a re-install. There are Debian systems out there that were installed once, back in the 90s, and have been upgraded (both distro versions and hardware) continuously since then. Do people not know that, or do they want the thrill of booting up with the new Gutsy CD? Debian is widely considered to be the most popular distribution, and a Debian release hardly raises eyebrows (other than somebody saying “Debian released? Has hell frozen over?”, and scheduling an upgrade for some time in the next 6 months). If nothing else, this shows how different distributions’ user-base can be, while being technically very similar.

I’ve done my bit to help out the Ubuntu release:

  • I’ve been running Gutsy since it’s fork, and filed god only knows how many bugs on launchpad.
  • I’ve created my unofficial package DVDs, for people without Internet connections.
  • My mirror & toaster had the CD images in advance, and we’ll have the DVDs by tomorrow morning.
  • I’ve got a UK server with a 100Mbps link seeding Ubuntu DVD torrents. I was the sole seed for many hours, and have uploaded over 200GB, so at least I did some good ;-)

Now, as the release traffic dies down again (i.e. heavy but not quite flat-lining), I hope the sysadmins and release-party-goers sleep well, you all deserve it.

BIOS Recovery

A couple of months ago, I was looking at a friend’s laptop. It had definite motherboard problems and a dead HDD. As part of the service, I updated the BIOS. Unfortunately, it died mid-flash.

I tried the local Fujitsu-Siemens service center, but they said the motherboard had to be replaced. On an aging laptop, that’s just not worth it. New HDD, yes. New Mobo, no.

I asked around UCT everywhere, looking for someone with a PROM programmer, but the only one I could find was an ancient device attached to a 286. The “new” programmer (not much newer) was broken… I waited a couple of months, but it still isn’t fixed.

So, I tried a mail-order BIOS flashing service, biosflash.com. They found a compatible chip, programmed it (with the update I’d been trying to install), and put it in the (registered) post within 24hrs. 2 weeks later, I’ve got it, and it installed it in the laptop.

It works!

Usually this kind of kind of thing means the laptop is written off. (Desktops normally have some kind of bad-flash recovery procedure, but it’s rare in laptops, and not foolproof anyway). I’m very impressed with biosflash.com: for only €15, the machine is no longer a brick.

Now to replace that HDD

Automated backups to external disk

I remember somebody asking how to do this on the CLUG lists a while back. But here’s the problem:

You’ve got an automated backup system, but you want offsite backups. DVDs are too small, external hard drives are the only option. You want the user to be able to plug in the firewire disk, have the backup start automatically, and let them know when it’s done.

Here’s how I implemented it:

The backups are implemented with backup-manager, they backup into /mnt/backup-tmp/

The external hard drive connects by firewire. Running udevinfo -a -p /sys/block/sdd on it showed me it’s ID:

ATTRS{ieee1394_id}=="0090a9787b339de6:1:0"

I created this UDEV rule file /etc/udev/rules.d/local-backup.rules:

ATTRS{ieee1394_id}=="0090a9787b339de6:1:0", SYMLINK="backupdisk", RUN+="/usr/local/sbin/backup-to-external.sh"

And the relevant fstab entry:

/dev/backupdisk /mnt/backup-disk vfat   sync                    0       0

And the backup script /usr/local/sbin/backup-to-external.sh:

#!/bin/sh -e

LOCKFILE=/var/run/backup-to-external.lock

logger "Backup disk detected"

# Test for expired locks
if [ -e "$LOCKFILE" ]; then
  if ! kill -0 `cat "$LOCKFILE"`; then
    rm "$LOCKFILE"
  fi
fi

lockfile -r0 "$LOCKFILE"
echo $$ > "$LOCKFILE"

sleep 5s

logger "Backup to external begun"

mount /mnt/backup-disk
rsync -a /mnt/backup-tmp/ /mnt/backup-disk/
umount /mnt/backup-disk

beep -l 1000 -f 3000 -r 5
echo -e "You can disconnect the disk now.\nThank you.\n\nThe backup System." | mail -s "Backup completed" the-secretary@email.address

rm "$LOCKFILE"
logger "Backup to external completed"

CLUG Talk videos

CLUG is one of the more active LUGs around, and has had fortnightly meetings with technical talks consistently since at least 1999 (the extent of our records, and my involvement).

While this heritage is a testament to a strong LUG, there are only about 20 people who regularly attend meetings, and thus get the benefit of these talks. With a LUG membership of around 800 (my best estimate from mailing list figures), this is a tiny fraction of our community. People who can’t make it have been requesting videos of talks for a while, and recently Jonathan Carter brought his camera, and we started playing with videoing them. He has been stuck in Johannesburg for a few weeks and left me his camera, so I’ve been playing around with encoding.

So far, the lessens learned:

  • Video quality: A camera, high up at the back of the room produces reasonable, compressible video without having to have someone pan around following the presenter.
  • But the built-in microphones on a cheap camera just aren’t effective at that range (especially when you’ve got a quiet speaker).
  • Audience questions are hard to record.
  • Audio editing is do-able, and necessary. But so far, we haven’t ventured into video editing (on linux, this isn’t trivial).
  • You can’t normally read all the slides (or demos) with SD video. At full PAL resolution, most slides are legible. Screen-capture is an option, but most presenters make their slides available, which is far more bandwidth efficient.

Our progress so far is these procedures, and these videos: UK, ZA. We are using Ogg/Vorbis/Theora, and 3 different qualities of video. The three qualities are overkill, but I’m still experimenting with settings. I’d like some feedback - especially from a codec expert :-)

We lost Wikimania 2008

We lost Wikimania 2008, quite badly (by points). I’m obviously sad, because I was hopeful, and had invested the odd weekend into comparative flight pricing, bid editing, and sitting on the phone…

According to the published criteria, we should have done well. Somehow we managed to score lower than Alexandria on “rotation” which is odd because Wikimania has never been held in the Southern Hemisphere.

I’d have liked too see the IRC log from the judges meeting, to see where we lost our points, but of course that isn’t going to happen, so we’ll have to work it out ourselves, by comparing the bids.

In retrospect, our bid wasn’t that great. We were pinning more hopes on our location, and Cape Town’s attractions, then on the bid itself. It was expensive, last-minute, and lacking sponsership & good accommodation. By losing, we got our chance to get it right next time.

The good news, is bidding opens for Wikimania 2009 almost immediately. I don’t know what the chances are of 2009 being in Africa again, but I’m assuming we are going to bid, if everyone can pick themselves up and get going on it. To do it, w e are going to need firm sponsors. Find any friends you have corporate contacts, and twist their arms.

BIOS Flashing with memdisk

I’ve just discovered memdisk. It’s part of the syslinux package on Debian/Ubuntu, and hides in /usr/lib/syslinux/memdisk.

Memdisk lets you boot a floppy image, via grub or pxelinux. In this modern era of computers without floppy drives, it means you can do BIOS updates without having to go through the whole procedure of turning a floppy image into a bootable CD.

In PXELINUX, the config file would look like this:

DEFAULT memdisk initrd=FILENAME.img

In Grub, like this:

title     Bios Flash
kernel    /boot/memdisk
initrd    /boot/FILENAME.img

Thanks ThinkWiki for the idea.

Caveat emptor: apparently some flash tools don’t like memdisk, so YMMV

Dovecot shared mailboxes (the correct way)

I’ve just implemented shared mailboxes in dovecot (which rocks, btw). It isn’t difficult, but I don’t think it’s very well documented

The preferred way to do this is with IMAP Namespaces. My natural approach would be to create something like a Maildir tree /srv/mail/shared, and make this the “public” namespace. Then set filesystem permissions on subtrees of that, to define who can see what. Unfortunately, dovecot uses strict Maildir++, and won’t let you create mailboxes inside each other (on the filesystem) /Foo/Bar is stored as a Maildir called .Foo.Bar, so subtrees don’t exist, so this isn’t an option. The up-comming dbox format should allow something like this, but it isn’t usable yet.

My solution was to create multiple namespaces. One for each shared mailbox. Users are given permission to use them via file-system permissions (i.e. group membership), example:

# Default namespace, needed if you add namespaces
namespace private {
    prefix =
    separator = /
    index = yes
}
# Office inbox, available to receptionists, office managers, and directors:
namespace public {
   prefix = office/
   separator = /
   location = maildir:/srv/mail/office/.Maildir:CONTROL=~/.Maildir/control/office:INDEX=~/.Maildir/index/office
   hidden = no
}
# Umask for shared folders
umask = 0007

Setting CONTROL and INDEX mean that dovecot’s metadata is stored in the user’s personal Maildir, so users who don’t have permission to see the shared mailbox don’t get errors.

The permissions of the mailbox should be done as follows:

# touch /srv/mail/office/dovecot-shared
# chown -R mail.office-mailbox /srv/mail/office
# find /srv/mail/office -type d -print0 | xargs -0 chmod 2770
# find /srv/mail/office -type f -print0 | xargs -0 chmod 660

If you want a common subscription list, you have to manually symlink:

# ln -s /srv/mail/office/subscriptions ~luser/.Maildir/control/office/

Seems to work well. (at least with thunderbird)

Non-free codecs on gutsy

Finally, it’s got easy to install the codecs we can legally use in this country:

Medibuntu now has a non-free-codecs package, an all-in-one virtual package.

Nice job!

Personally, I’m still in favour having them there by default, and the installer removes them if you select a patent-encumbered country as your time-zone, but I can see why people are wary of that approach….