split-routing

Split-Routing on Debian/Ubuntu

Fri, 19/09/2008 - 5:13pm — tumbleweed

My post on split-routing on OpenWRT has been incredibly popular, and led to many people implementing split-routing, whether or not they had OpenWRT. While it's fun to have an exercise as a reader, it led to me having to help lots of newbies through porting that setup to a Debian / Ubuntu environment. To save myself some time, here's how I do it on Debian:

Background, especially for non-South Africa readers: Bandwidth in South Africa is ridiculously expensive, especially International bandwidth. The point of this exercise is that we can buy "local-only" DSL accounts which only connect to South African networks. E.g. I have an account that gives me 30GB of local traffic / month, for the same cost as 2.5GB of International traffic account. Normally you'd change your username and password on your router to switch account when you wanted to do something like an Debian apt-upgrade, but that's irritating. There's no reason why you can't have a Linux-based router concurrently connected to both accounts via the same ADSL line.

Firstly, we have a DSL modem. Doesn't matter what it is, it just has to support bridged mode. If it won't work without a DSL account, you can use the Telkom guest account. My recommendation for a modem is to buy a Telkom-branded Billion modem (because Telkom sells everything with really big chunky, well-surge-protected power supplies).

For the sake of this example, we have the modem (IP 10.0.0.2/24) plugged into eth0 on our server, which is running Debian or Ubuntu, doesn't really matter much - personal preference. The modem has DHCP turned off, and we have our PCs on the same ethernet segment as the modem. Obviously this is all trivial to change.

You need these packages installed:

# aptitude install iproute pppoe wget awk findutils

You need ppp interfaces for your providers. I created /etc/ppp/peers/intl-dsl:

user intl-account@uber-isp.net

unit 1

pty "/usr/sbin/pppoe -I eth0 -T 80 -m 1452"

noipdefault

defaultroute

hide-password

lcp-echo-interval 20

lcp-echo-failure 3

noauth

persist

maxfail 0

mtu 1492

noaccomp

default-asyncmap

/etc/ppp/peer/local-dsl:

user local-account@uber-isp.net

unit 2

pty "/usr/sbin/pppoe -I eth0 -T 80 -m 1452"

noipdefault

hide-password

lcp-echo-interval 20

lcp-echo-failure 3

connect /bin/true

noauth

persist

maxfail 0

mtu 1492

noaccomp

default-asyncmap

unit 1 makes a connection always bind to "ppp1". Everything else is pretty standard. Note that only the international connection forces a default route.

To /etc/ppp/pap-secrets I added my username and password combinations:

# User                     Host Password

intl-account@uber-isp.net  *    s3cr3t

local-account@uber-isp.net *    passw0rd

You need custom iproute2 routing tables for each interface, for the source routing. This will ensure that incoming connections get responded to out of the correct interface. As your provider only lets you send packets from your assigned IP address, you can't send packets with the international address out of the local interface. We get around that with multiple routing tables. Add these lines to /etc/iproute2/rt_tables:

      local-dsl

      intl-dsl

Now for some magic. I create /etc/ppp/ip-up.d/20routing to set up routes when a connection comes up:

#!/bin/sh -e

case "$PPP_IFACE" in

 "ppp1")

   IFACE="intl-dsl"

   ;;

 "ppp2")

   IFACE="local-dsl"

   ;;

 *)

   exit 0

esac

# Custom routes

if [ -f "/etc/network/routes-$IFACE" ]; then

  cat "/etc/network/routes-$IFACE" | while read route; do

    ip route add "$route" dev "$PPP_IFACE"

  done

fi

# Clean out old rules

ip rule list | grep "lookup $IFACE" | cut -d: -f2 | xargs -L 1 -I xx sh -c "ip rule del xx"

# Source Routing

ip route add "$PPP_REMOTE" dev "$PPP_IFACE" src "$address" table "$IFACE"

ip route add default via "$PPP_REMOTE" table "$IFACE"

ip rule add from "$PPP_LOCAL" table "$IFACE"

# Make sure this interface is present in all the custom routing tables:

route=`ip route show dev "$PPP_IFACE" | awk '/scope link  src/ {print $1}'`

awk '/^[0-9]/ {if ($1 > 0 && $1 < 250) print $2}' /etc/iproute2/rt_tables | while read table; do

  ip route add "$route" dev "$PPP_IFACE" table "$table"

done

That script loads routes from /etc/network/routes-intl-dsl and /etc/network/routes-local-dsl. It also sets up source routing so that incoming connections work as expected.

Now, we need those route files to exist and contain something useful. Create the script /etc/cron.daily/za-routes (and make it executable):

#!/bin/sh -e

ROUTEFILE=/etc/network/routes-local-dsl

wget -q http://mene.za.net/za-routes/latest.txt -O /tmp/zaroutes

size=`stat -c '%s' /tmp/zaroutes`

if [ $size -gt 0 ]; then

  mv /tmp/zaroutes "$ROUTEFILE"

fi

It downloads the routes file from cocooncrash's site (he gets them from local-route-server.is.co.za, aggregates them, and publishes every 6 hours). Run it now to seed that file.

Now some International-only routes. I use IS local DSL, so SAIX DNS queries should go through the SAIX connection even though the servers are local to ZA.

My /etc/network/routes-intl-dsl contains SAIX DNS servers and proxies:

25.255.3

25.1.9

25.1.11

43.1.14

43.1.11

43.34.190

43.38.190

43.42.190

43.45.190

43.46.190

43.50.190

43.53.190

43.9.21

Now we can tell /etc/network/interfaces about our connections so that they can get brought up automatically on bootup:

# This file describes the network interfaces available on your system

# and how to activate them. For more information, see interfaces(5).

# The loopback network interface

auto lo

iface lo inet loopback

# The primary network interface

allow-hotplug eth0

iface eth0 inet static

        address 10.0.0.1

        netmask 255.255.255.0

auto local-dsl

iface local-dsl inet ppp

        provider local-dsl

auto intl-dsl

iface intl-dsl inet ppp

        provider intl-dsl

For DNS, I use dnsmasq, hardcoded to point to IS & SAIX upstreams. My machine's /etc/resolv.conf just points to this dnsmasq.

So something like /etc/resolv.conf:

nameserver 127.0.0.1

/etc/dnsmasq.conf:

no-resolv

# IS:

server=168.210.2.2

server=196.14.239.2

# SAIX:

server=196.43.34.190

server=196.43.46.190

server=196.25.1.11

domain=foobar.lan

dhcp-range=10.0.0.128,10.0.0.254,12h

dhcp-authoritative

no-negcache

If you haven't already, you'll need to turn on ip_forward. Add the following to /etc/sysctl.conf and then run sudo sysctl -p:

net.ipv4.ip_forward=1

Finally, you'll need masquerading set up in your firewall. Here is a trivial example firewall, put it in /etc/network/if-up.d/firewall and make it executable. You should probably change it to suit your needs or use something else, but this should work:

#!/bin/sh

if [ $IFACE != "eth0" ]; then

  exit;

fi

iptables -F INPUT

iptables -F FORWARD

iptables -t nat -F POSTROUTING

iptables -A INPUT -i lo -j ACCEPT

iptables -A INPUT -i eth0 -s 10.0.0.0/24 -j ACCEPT

iptables -A INPUT -i ppp+ -m state --state ESTABLISHED,RELATED -j ACCEPT

iptables -A INPUT -j DROP

iptables -A FORWARD -i ppp+ -m state --state ESTABLISHED,RELATED -j ACCEPT

iptables -A FORWARD -i eth0 -o ppp+ -j ACCEPT

iptables -A FORWARD -j DROP

iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o ppp+ -j MASQUERADE

20 comments

Local only DSL

Sat, 01/09/2007 - 11:06pm — tumbleweed

Update: Debian/Ubuntu version

I've finally jumped onto the local only DSL bandwagon. If you haven't done it yet, it's a great way to save some bucks. The idea is that you get a local only account like this, which costs a fraction per GiB compared to normal account. Then you get your router to connect to both simultaneously, and route intelligently between them.

Most ADSL routers won't let you connect 2 concurrent ADSL connections on the same ATM circuit. The solution is to use a separate modem and router. I'm using a basic Billion modem, in bridged mode, and a WRT54GL, running OpenWRT/kamikaze, as the router.

OpenWRT doesn't support 2 PPPoE connections out of the box, but I've found the problems, and got a few changes committed upstream, that solve them:

Configurable default route: changeset 8569
Allow specifying ppp unit numbers: changeset 8563+8564
Support for multiple interfaces in UpdateDD (a DynDNS client): bug 2294

The firewall (/etc/init.d/firewall) needs to be modified with "WAN=ppp+" somewhere near the top, so that it masquerades all the ppp connections. This was a hack, apparently the firewall is being re-written soon.

There is also a bug that resets existing PPPoE connections on a ethernet interface when you fire up a new connection. This will apparently be fixed by the future interface aliasing support. For now, I just hacked around it in /lib/network/config.sh:

prepare_interface() {

        local iface="$1"

        local config="$2"

        #SR: We don't want to reset any pppoe connections

        config_get proto "$config" proto

        [ "$proto" = "pppoe" ] && return 0

and /sbin/ifdown:

config_get ifname "$cfg" ifname

config_get device "$cfg" device

[ ."${proto%oe}" == ."ppp" ] && device=

[ ."$device" != ."$ifname" ] || device=

for dev in $ifname $device; do

        ifconfig "$dev" 0.0.0.0 down >/dev/null 2>/dev/null

done

I got my local routes list from cocooncrash's site (he gets them from local-route-server.is.co.za, aggregates them, and publishes every 6 hours). OpenWRT already has a static routing configuration system, but it's very verbose. So I wrote my own, adding the new configuration option routefile. I used these hotplug scripts to set up routing and source routing, with the help of iproute2:

$ ipkg install ip

$ mkdir /etc/routes

$ wget http://mene.za.net/za-routes/latest.txt -O /etc/routes/zaroutes

You'll probably want to update that route file regularly. I don't run cron on my WRT54GL, so I do it manually. Up to you.

/etc/config/network:

# ...local interfaces...

#### WAN configuration

config interface    wan

        option ifname   "eth0.1"

        option proto    pppoe

        option username "xxxxx@international.co.za"

        option password "xxxxxx"

        option routefile "/etc/routes/exceptions"

        option defaultroute 1

config interface    localdsl

        option ifname   "eth0.1"

        option proto    pppoe

        option username "xxxxx@local.co.za"

        option password "xxxxxx"

        option routefile "/etc/routes/zaroutes"

        option defaultroute 0

/etc/iproute2/rt_tables:

#

# reserved values

#

255     local

254     main

253     default

0       unspec

#

# local

#

1   wan

2   localdsl

/etc/hotplug.d/iface/20-split-routes:

case "$ACTION" in

  ifup)

    . /etc/functions.sh

    include /lib/network

    scan_interfaces

    config_get routefile "$INTERFACE" routefile

    # Does this interface have custom routes?

    if [ -e "$routefile" ]; then

      # Add routes for this interface

      cat "$routefile" | while read route; do

        ip route add "$route" dev "$DEVICE"

      done  

      # Set up source routing

      peer=`ip addr show dev $DEVICE | sed -n '/inet/ s#.* peer \([0-9.]*\)/.*#\1# p'`

      address=`ip addr show dev $DEVICE | sed -n '/inet/ s/.* inet \([0-9.]*\) .*/\1/ p'`

      ip route add "$peer" dev "$DEVICE" src "$address" table "$INTERFACE"

      ip route add default via "$peer" table "$INTERFACE"

      ip rule add from "$address" table "$INTERFACE"

    fi

    # Make sure this interface is present in all the custom routing tables:

    route=`ip route show dev "$DEVICE" | awk '/scope link  src/ {print $1}'`

    awk '/^[0-9]/ {if ($1 > 0 && $1 < 250) print $2}' /etc/iproute2/rt_tables | while read table; do

      ip route add "$route" dev "$DEVICE" table "$table"

    done

    ;;

esac

/etc/hotplug.d/net/20-split-routes:

case "$ACTION" in

  register)

    . /etc/functions.sh

    include /lib/network

    scan_interfaces

    # If this interface doesn't have a link local route, we don't need to bother

    route=`ip route show dev "$INTERFACE" | awk '/scope link  src/ {print $1}'`

    [ ."$route" = ."" ] && return 0

    # Add this interface's route to all custom routing tables

    awk '/^[0-9]/ {if ($1 > 0 && $1 < 250) print $2}' /etc/iproute2/rt_tables | while read table; do

      ip route add "$route" dev "$INTERFACE" table "$table"

    done

    ;;

esac

Now, lastly, it won't bring up both interfaces by default. That will be fixed by aliasing in the future, but for now:

/etc/init.d/network-multiwan:

#!/bin/sh

ifup wan

ifup localdsl

$ chmod 755 /etc/init.d/network-multiwan

$ ln -s ../init.d/network-multiwan /etc/rc.d/S49network-multiwan

That's it, and it's working beautifully :-)

What is source routing, people ask? The problem is that your router now has 2 WAN IP addresses. IP1 is used for local traffic, and IP2 for international. So if somebody in ZA tries to connect to IP2, the reply (local destination) will go out of Interface 1. The ISP on the other end of Interface 1 will drop this reply, because it's not coming from IP1.

Source routing tells the router that replies must go out of the same interface that the request came in on. I'm doing it by creating separate routing tables for traffic origionating from each WAN interface.

Gotchas

If you are using 2 different ISPs (say SAIX international and IS local), you must make sure that DNS queries get routed out the right interface. SAIX won't accept queries on their servers from IS, and vice versa.
SAIX Web proxies, Mail servers, and News servers don't accept traffic from local accounts. (especially from another ISP)

51 comments

Navigation

split-routing

Split-Routing on Debian/Ubuntu

Local only DSL

Gotchas