Stefano's World

My post about repositories wasn’t just a little attempt to stave off work, it was part of a larger scheme.

I share the ADSL line in my digs with 3 other people. We do split-routing to save money, but we still have to divide the phone bill at the end of the month. Rather than buy a fixed cap, and have a fight over who’s fault it was when we get capped, we are running a pay-per-use system (with local use free, subsidised by me). It means you don’t have to restrain yourself for the common cap, but it also means I need to calculate who owes what.

For the first month, I used my old standby, bandwidthd. It uses pcap to count traffic, and gives you totals and graphs. For simplicity of logging, I gave each person a /28 for their machines and configured static DHCP leases. Then bandwidthd totalled up the internet use for each /28.

This was sub-optimal. bandwidthd either sees the local network, in which case it can’t see which packets went out over which link. Or it can watch the international link, but then not know which user is responsible.

I could have installed some netflow utilities at this point, but I wanted to roll my own with the correct Linux approach (ulog) rather than any pcapping. ulogd is the easy ulog solution.

Ulogd can pick up packets that you “-j ULOG” from iptables. It receives them over a netlink interface. You can tell iptables how many bytes of each packet to send, and how many to queue up before sending them. E.g.

# iptables -I INPUT 1 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 48 --ulog-prefix input

will log the first 48 bytes of any incoming packet to netlink-group 1. It will tag the packet as being “input”, and send them in batches of 50. 48 bytes is usually enough to catch any data you could want from the headers. If you were only need size, 4 bytes will do, and for source and destination as well, 20.

Now, we tell ulogd to listen for this stuff and log it. Ulogd has a pluggable architecture. IPv4 decoding is a plugin, and there are various logging plugins for “-j LOG” emulation, Text files, pcap-files, MySQL, PostgreSQL, and SQLite. For my purposes, I used MySQL as the router in question already had MySQL on it (for Cacti). Otherwise, I would have opted for SQLite. Be warned that the etch version of ulogd doesn’t automatically reconnect to the MySQL server should the connection break for any reason. I backported the lenny version to etch to get around that. (You also need to provide the reconnect and connect_timeout options.)

Besides the reconnection issue, the SQL implementations are quite nice. They have a set schema, and you just need to create a table with the columns in it that you are interested in. No other configuration (beyond connection details) is necessary.

My MySQL table:

CREATE TABLE `ulog` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `oob_time_sec` int(10) unsigned NOT NULL,
  `oob_prefix` char(4) NOT NULL,
  `ip_totlen` smallint(5) unsigned NOT NULL,
  PRIMARY KEY  (`id`),
  UNIQUE KEY `id` (`id`),
  KEY `oob_prefix` (`oob_prefix`),
  KEY `oob_time_sec` (`oob_time_sec`)
);

My ulogd.conf:

[global]
# netlink multicast group (the same as the iptables --ulog-nlgroup param)
nlgroup=1    
# logfile for status messages
logfile="/var/log/ulog/ulogd.log"    
# loglevel: debug(1), info(3), notice(5), error(7) or fatal(8)
loglevel=5    
# socket receive buffer size (should be at least the size of the
# in-kernel buffer (ipt_ULOG.o 'nlbufsiz' parameter)
rmem=131071    
# libipulog/ulogd receive buffer size, should be > rmem
bufsize=150000
# ulogd_BASE.so - interpreter plugin for basic IPv4 header fields
#             you will always need this
plugin="/usr/lib/ulogd/ulogd_BASE.so"
plugin="/usr/lib/ulogd/ulogd_MYSQL.so"

[MYSQL]
table="ulog"
pass="foo"
user="ulog"
db="ulog"
host="localhost"
reconnect=5
connect_timeout=10

The relevant parts of my firewall rules:

# Count proxy usage (transparent and explicit)
iptables -A count-from-inside -p ! tcp -j RETURN
iptables -A count-from-inside -p tcp -m multiport --destination-ports ! 3128,8080 -j RETURN
iptables -A count-from-inside -s 10.0.0.16/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix sr-p
iptables -A count-from-inside -s 10.0.0.32/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix fb-p
iptables -A count-from-inside -s 10.0.0.128/25 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix gu-p

iptables -A count-to-inside -p ! tcp -j RETURN
iptables -A count-to-inside -p tcp -m multiport --source-ports ! 3128,8080 -j RETURN
iptables -A count-to-inside -d 10.0.0.16/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix sr-p
iptables -A count-to-inside -d 10.0.0.32/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix fb-p
iptables -A count-to-inside -d 10.0.0.128/25 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix gu-p

# Count forwarded traffic (excluding local internet connection - ppp2)
iptables -A count-forward-in -i ppp2 -j RETURN
iptables -A count-forward-in -d 10.0.0.16/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix sr-f
iptables -A count-forward-in -d 10.0.0.32/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix fb-f
iptables -A count-forward-in -d 10.0.0.128/25 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix gu-f

iptables -A count-forward-out -o ppp2 -j RETURN
iptables -A count-forward-out -s 10.0.0.16/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix sr-f
iptables -A count-forward-out -s 10.0.0.32/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix fb-f
iptables -A count-forward-out -s 10.0.0.128/25 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix gu-f

# Glue
iptables -A INPUT -i eth0 -j count-from-inside
iptables -A OUTPUT  -o eth0 -j count-to-inside
iptables -A FORWARD -i ppp+ -j count-forward-in
iptables -A FORWARD -o ppp+ -j count-forward-out

So, traffic for my /28 (sr) will be counted as sr-f or sr-p so I can tally up proxy & forwarded traffic separately. (Yes, I can count traffic with squid too, but doing it all in one place is simpler.) fb is random housemate Foo Bar, and gu guest (unreserved IP addresses).

You can query the usage this month with for example:

SELECT oob_prefix, SUM(ip_totlen) FROM ulog WHERE oob_time_sec > UNIX_TIMESTAMP('2008-04-01 00:00:00') GROUP BY oob_prefix;

Your table will fill up fast. We are averaging around 200 000 rows per day. So obviously some caching is in order:

CREATE TABLE daily (
  id INT UNSIGNED NOT NULL AUTO_INCREMENT,
  time TIMESTAMP,
  oob_prefix CHAR(4) NOT NULL,
  data INT UNSIGNED NOT NULL,
  PRIMARY KEY (id),
  KEY (oob_prefix(4)),
  KEY (time)
);

And every night, run something like:

INSERT INTO daily (time, oob_prefix, data)
SELECT FROM_UNIXTIME(MAX(oob_time_sec)), oob_prefix, SUM(ip_totlen)
FROM ulog
WHERE oob_time_sec >= UNIX_TIMESTAMP('2008-04-01 00:00:00')
  AND oob_time_sec < UNIX_TIMESTAMP('2008-04-02 00:00:00')
GROUP BY oob_prefix;
DELETE FROM ulog WHERE oob_time_sec  >= UNIX_TIMESTAMP('2008-04-01 00:00:00')
  AND oob_time_sec < UNIX_TIMESTAMP('2008-04-02 00:00:00');

Finally, I have a simple little PHP script that provides reporting and calculates dues. Done.

Up to now, whenever I’ve needed a backport or debian recompile, I’ve done it locally. But finally last night, instead of studying for this morning’s exam, I decided to do it properly.

The tool for producing a debian archive tree is reprepro. There are a few howtos out there for it, but none of them quite covered everything I needed. So this is mine. But we’ll get to that later, first we need to have some packages to put up.

For building packages, I decided to do it properly and use pbuilder. Just install it:

# aptitude install pbuilder cdebootstrap devscripts

Make the following changes to /etc/pbuilderrc:

MIRRORSITE=http://ftp.uk.debian.org/debian
DEBEMAIL="Your Name <you@example.com>"

The first, to point to your local mirror, and the second to credit you in the packages.

Then, as root:

# pbuilder create --distribution etch --debootstrapopts --variant=buildd

Now, we can build a package, lets build the hello package:

$ mkdir /tmp/packaging; cd /tmp/packaging
$ gpg --recv-key 3EF23CD6
$ dget -x http://ftp.uk.debian.org/debian/pool/main/h/hello/hello_2.2-2.dsc
dpkg-source: extracting hello in hello-2.2
dpkg-source: unpacking hello_2.2.orig.tar.gz
dpkg-source: applying ./hello_2.2-2.diff.gz
$ cd hello-2.2/
$ debchange -n

dget and debchange are neat little utilities from devscripts. You can configure them to know your name, e-mail address, etc. If you work with debian packages a lot, you’ll get to know them well. Future versions of debchange support --bpo for backports, but we use -n which means new package. You should edit the version number in the top line to be a backport version, i.e.:

hello (2.2-2~bpo-sr.1) etch-backports; urgency=low

  * Rebuild for etch-backports.

 -- Your Name <you@example.com>  Wed,  2 Apr 2008 22:24:30 +0100

Now, let’s build it. We are only doing a backport, but if you were making any changes, you’d do them before the next stage, and list them in the changelog you just edited:

$ cd ..
$ dpkg-source -sa -b hello-2.2-2~bpo/
$ sudo pbuilder build hello_2.2-2~bpo-sr.1.dsc

Assuming no errors, the built package will be sitting in /var/cache/pbuilder/result/.

Now, for the repository:

$ mkdir ~/public_html/backports
$ cd ~/public_html/backports
$ mkdir conf
$ cat > conf/distributions << EOF
Origin: Your Name
Label: Your Name's Backports
Suite: stable-backports
Codename: etch-backports
Version: 4.0
Architectures: i386 all source
Components: main
Description: Your Name's repository of etch backports.
SignWith: ABCDABCD
NotAutomatic: yes
EOF

This file defines your repository. The codename will be the distribution you list in your sources.list. The version should match it. The architectures are the architectures you are going to carry - “all” refers to non-architecture-specific packages, and source to source packages. I added amd64 to mine. SignWith is the ID of the GPG key you are going to use with this repo. I created a new DSA key for the job. NotAutomatic is a good setting for a backports repo, it means that packages won’t be installed from here unless explicitly requested (via package=version or -d etch-backports).

Let’s start by importing our source package:

$ cd /tmp/packaging
$ debsign -kABCDABCD hello_2.2-2~bpo-sr.1.dsc
$ cd ~/public_html/backports
$ reprepro -P optional -S devel --ask-passphrase -Vb . includedsc etch-backports /tmp/packaging/hello_2.2-2~bpo-sr.1.dsc

(There is currently a known bug in reprepro’s command-line handling. -S and -P are swapped.)

Now, let’s import our binary package:

$ reprepro --ask-passphrase -Vb . includedeb etch-backports /var/cache/pbuilder/result/hello_2.2-2~bpo-sr.1_i386.deb

Reprepro can be automated with it’s processincoming command, but that’s beyond the scope of this howto.

Test your new repository, add it to your /etc/apt/sources.list

deb http://example.com/~you/backports etch-backports main

# aptitude update
# aptitude install hello=2.2-2~bpo-sr.1

Enjoy. My backports repository can be found here.

Even before school, my future interests were clear: I tied-up the house with wires and made “electrical gadgets” out of old electrical junk. I remember being given my first battery, light bulbs, and wires. From there it was downhill.

The first PC:

My first computer was a [HP 9816][]. It was a year older than me, had a 6800 Processor, 128k RAM, and an (external) pair of single sided 3.5” floppy drives.

It had a ROM BASIC board, and a set of [VisiCalc][] floppies (with manual shutters), so I spent my time

Quick post. If you have multiple IP addresses (i.e. a range) assigned to you server, and you want to listen on all of them (i.e. multiple SSL sites), then rather than using the ancient eth0:1 syntax, you can hack /etc/network/interfaces to use iproute2 properly.

Assuming the IP 10.2.3.4, with the extra range of 10.5.4.110-10.5.4.118 (yes these extra ranges often ignore class-boundries):

auto eth0
iface eth0 inet static
    address 10.2.3.4
    netmask 255.255.255.0
    network 10.2.3.0
    broadcast 10.2.3.255
    gateway 10.2.3.1
    # Extra IPs:
    post-up for last in `seq 110 118`; do ip addr add 10.5.4.$last/32 dev $IFACE; done || true
    pre-down for ip in `ip addr show dev $IFACE | sed -n 's@.* inet \([0-9.]*/32\) .*@\1@ p'`; do ip addr del $ip dev $IFACE; done || true

Yes, it’s ugly as shit, but I can’t think of a neater way to do it

I remember somebody asking how to do this on the CLUG lists a while back. But here’s the problem:

You’ve got an automated backup system, but you want offsite backups. DVDs are too small, external hard drives are the only option. You want the user to be able to plug in the firewire disk, have the backup start automatically, and let them know when it’s done.

Here’s how I implemented it:

The backups are implemented with [backup-manager][], they backup into /mnt/backup-tmp/

The external hard drive connects by firewire. Running udevinfo -a -p /sys/block/sdd on it showed me it’s ID:

ATTRS{ieee1394_id}=="0090a9787b339de6:1:0"

I created this UDEV rule file /etc/udev/rules.d/local-backup.rules:

ATTRS{ieee1394_id}=="0090a9787b339de6:1:0", SYMLINK="backupdisk", RUN+="/usr/local/sbin/backup-to-external.sh"

And the relevant fstab entry:

/dev/backupdisk /mnt/backup-disk vfat   sync                    0       0

And the backup script /usr/local/sbin/backup-to-external.sh:

#!/bin/sh -e

LOCKFILE=/var/run/backup-to-external.lock

logger "Backup disk detected"

# Test for expired locks
if [ -e "$LOCKFILE" ]; then
  if ! kill -0 `cat "$LOCKFILE"`; then
    rm "$LOCKFILE"
  fi
fi

lockfile -r0 "$LOCKFILE"
echo $$ > "$LOCKFILE"

sleep 5s

logger "Backup to external begun"

mount /mnt/backup-disk
rsync -a /mnt/backup-tmp/ /mnt/backup-disk/
umount /mnt/backup-disk

beep -l 1000 -f 3000 -r 5
echo -e "You can disconnect the disk now.\nThank you.\n\nThe backup System." | mail -s "Backup completed" the-secretary@email.address

rm "$LOCKFILE"
logger "Backup to external completed"

I’ve just discovered memdisk. It’s part of the syslinux package on Debian/Ubuntu, and hides in /usr/lib/syslinux/memdisk.

Memdisk lets you boot a floppy image, via grub or pxelinux. In this modern era of computers without floppy drives, it means you can do BIOS updates without having to go through the whole procedure of turning a floppy image into a bootable CD.

In PXELINUX, the config file would look like this:

DEFAULT memdisk initrd=FILENAME.img

In Grub, like this:

title     Bios Flash
kernel    /boot/memdisk
initrd    /boot/FILENAME.img

Thanks ThinkWiki for the idea.

Caveat emptor: apparently some flash tools don’t like memdisk, so YMMV

I’ve just implemented shared mailboxes in dovecot (which rocks, btw). It isn’t difficult, but I don’t think it’s very well documented…

The preferred way to do this is with IMAP Namespaces. My natural approach would be to create something like a Maildir tree /srv/mail/shared, and make this the “public” namespace. Then set filesystem permissions on subtrees of that, to define who can see what. Unfortunately, dovecot uses strict Maildir++, and won’t let you create mailboxes inside each other (on the filesystem) /Foo/Bar is stored as a Maildir called .Foo.Bar, so subtrees don’t exist, so this isn’t an option. The up-comming dbox format should allow something like this, but it isn’t usable yet.

My solution was to create multiple namespaces. One for each shared mailbox. Users are given permission to use them via file-system permissions (i.e. group membership), example:

# Default namespace, needed if you add namespaces
namespace private {
    prefix =
    separator = /
    index = yes
}
# Office inbox, available to receptionists, office managers, and directors:
namespace public {
   prefix = office/
   separator = /
   location = maildir:/srv/mail/office/.Maildir:CONTROL=~/.Maildir/control/office:INDEX=~/.Maildir/index/office
   hidden = no
}
# Umask for shared folders
umask = 0007

Setting CONTROL and INDEX mean that dovecot’s metadata is stored in the user’s personal Maildir, so users who don’t have permission to see the shared mailbox don’t get errors.

The permissions of the mailbox should be done as follows:

# touch /srv/mail/office/dovecot-shared
# chown -R mail.office-mailbox /srv/mail/office
# find /srv/mail/office -type d -print0 | xargs -0 chmod 2770 
# find /srv/mail/office -type f -print0 | xargs -0 chmod 660

If you want a common subscription list, you have to manually symlink:

ln -s /srv/mail/office/subscriptions ~luser/.Maildir/control/office/

Seems to work well. (at least with thunderbird)

I’ve just spent a few hours brain-haemorrhaging over why my new Postfix server wasn’t allowing me to enter “RCPT TO:” over a STARTTLS connection. Instead it would renegotiate the TLS.

Eventually I found an e-mail by Wietse Venema saying:

Victor Duchovni:
> On Mon, Jan 22, 2007 at 04:31:12PM -0500, Wietse Venema wrote: 
> > RCPT TO:<postmaster>
> > RENEGOTIATING
>
> You got bit by the "s_client" "R" feature... try "rcpt to:" lower case,
> then it hangs up.

What utter brain damage, a non-transparent SSL client program.

Read this and be warned — we are all stupid, in the eyes of the truly mad s_client

Thanks to Debian bug 407171, if you had mod_proxy installed in apache2, and upgrade to etch, it’ll also install mod_disk_cache, which means your /var partition is going to fill up quite quickly.

This happened to 2 CLUG servers.

I don’t think this is the correct behaviour, and I’m even more suprised to see that it the appearance of the bug is documented in a bug-report.

I’ve migrated my teeny-weenie Xen web/mail server to Debian/etch. It hasn’t even been rebooted (it would be a shame to spoil the uptime :-) ):

$ uprecords
     #               Uptime | System                                     Boot up
----------------------------+---------------------------------------------------
->   1   198 days, 06:16:44 | Linux 2.6.16.13-xenU      Thu Oct 12 10:12:51 2006
     2    99 days, 19:25:00 | Linux 2.6.12-xenU         Sun Oct  9 03:58:58 2005

It runs Lighttpd, a small and fast little webserver, popular in the Rails world. Lighttpd with PHP-fastcgi is probably faster than apache, and uses much less RAM.

With etch, I’ve finally been able to get mod_rewrite to work. So my Zapiro archive has nice URLs now :-)

Lighttpd has a very nice configuration style:

# No WWW
$HTTP["host"] =~ "^www\.((.+\.)?rivera\.za\.net)$" {
  url.redirect = ( ".*" => "http://%1$1" )
}
# Add WWW:
$HTTP["host"] =~ "^((foobar|someclient)\.co\.za)$" {
  url.redirect = ( ".*" => "http://www.%1$0" )
}

############################################
# PHP Apps:
$HTTP["host"] =~ "^(zapiro\.rivera\.za\.net)$" {
  url.redirect = ( "^/\?/(.*)" => "http://%1/$1" )
  url.rewrite-once = ( "^/(feed)$" => "/index.php?/$1",
                       "^/([0-9]+/[0-9]+/[0-9]+)$" => "/index.php?/$1" )
}

It’s more logical than apache, but you have to watch out for rewrite->redirect->rewrite loops. So if you change to a clean URL syntax, you can’t put in rewrites from index.php?/uglurl to /uglyurl because /uglyurl rewrites back to /index.php?/uglyurl, and you get a loop :-)