Split-Routing on Debian/Ubuntu

My post on split-routing on OpenWRT has been incredibly popular, and led to many people implementing split-routing, whether or not they had OpenWRT. While it's fun to have an exercise as a reader, it led to me having to help lots of newbies through porting that setup to a Debian / Ubuntu environment. To save myself some time, here's how I do it on Debian:

Background, especially for non-South Africa readers: Bandwidth in South Africa is ridiculously expensive, especially International bandwidth. The point of this exercise is that we can buy "local-only" DSL accounts which only connect to South African networks. E.g. I have an account that gives me 30GB of local traffic / month, for the same cost as 2.5GB of International traffic account. Normally you'd change your username and password on your router to switch account when you wanted to do something like an Debian apt-upgrade, but that's irritating. There's no reason why you can't have a Linux-based router concurrently connected to both accounts via the same ADSL line.

Firstly, we have a DSL modem. Doesn't matter what it is, it just has to support bridged mode. If it won't work without a DSL account, you can use the Telkom guest account. My recommendation for a modem is to buy a Telkom-branded Billion modem (because Telkom sells everything with really big chunky, well-surge-protected power supplies).

For the sake of this example, we have the modem (IP 10.0.0.2/24) plugged into eth0 on our server, which is running Debian or Ubuntu, doesn't really matter much - personal preference. The modem has DHCP turned off, and we have our PCs on the same ethernet segment as the modem. Obviously this is all trivial to change.

You need these packages installed:

# aptitude install iproute pppoe wget awk findutils

You need ppp interfaces for your providers. I created /etc/ppp/peers/intl-dsl:

user intl-account@uber-isp.net
unit 1
pty "/usr/sbin/pppoe -I eth0 -T 80 -m 1452"
noipdefault
defaultroute
hide-password
lcp-echo-interval 20
lcp-echo-failure 3
noauth
persist
maxfail 0
mtu 1492
noaccomp
default-asyncmap

/etc/ppp/peer/local-dsl:

user local-account@uber-isp.net
unit 2
pty "/usr/sbin/pppoe -I eth0 -T 80 -m 1452"
noipdefault
hide-password
lcp-echo-interval 20
lcp-echo-failure 3
connect /bin/true
noauth
persist
maxfail 0
mtu 1492
noaccomp
default-asyncmap

unit 1 makes a connection always bind to "ppp1". Everything else is pretty standard. Note that only the international connection forces a default route.

To /etc/ppp/pap-secrets I added my username and password combinations:

# User                     Host Password
intl-account@uber-isp.net  *    s3cr3t
local-account@uber-isp.net *    passw0rd

You need custom iproute2 routing tables for each interface, for the source routing. This will ensure that incoming connections get responded to out of the correct interface. As your provider only lets you send packets from your assigned IP address, you can't send packets with the international address out of the local interface. We get around that with multiple routing tables. Add these lines to /etc/iproute2/rt_tables:

1       local-dsl
2       intl-dsl

Now for some magic. I create /etc/ppp/ip-up.d/20routing to set up routes when a connection comes up:

#!/bin/sh -e

case "$PPP_IFACE" in
 "ppp1")
   IFACE="intl-dsl"
   ;;
 "ppp2")
   IFACE="local-dsl"
   ;;
 *)
   exit 0
esac

# Custom routes
if [ -f "/etc/network/routes-$IFACE" ]; then
  cat "/etc/network/routes-$IFACE" | while read route; do
    ip route add "$route" dev "$PPP_IFACE"
  done
fi

# Clean out old rules
ip rule list | grep "lookup $IFACE" | cut -d: -f2 | xargs -L 1 -I xx sh -c "ip rule del xx"

# Source Routing
ip route add "$PPP_REMOTE" dev "$PPP_IFACE" src "$address" table "$IFACE"
ip route add default via "$PPP_REMOTE" table "$IFACE"
ip rule add from "$PPP_LOCAL" table "$IFACE"

# Make sure this interface is present in all the custom routing tables:
route=`ip route show dev "$PPP_IFACE" | awk '/scope link  src/ {print $1}'`
awk '/^[0-9]/ {if ($1 > 0 && $1 < 250) print $2}' /etc/iproute2/rt_tables | while read table; do
  ip route add "$route" dev "$PPP_IFACE" table "$table"
done

That script loads routes from /etc/network/routes-intl-dsl and /etc/network/routes-local-dsl. It also sets up source routing so that incoming connections work as expected.

Now, we need those route files to exist and contain something useful. Create the script /etc/cron.daily/za-routes (and make it executable):

#!/bin/sh -e
ROUTEFILE=/etc/network/routes-local-dsl

wget -q http://mene.za.net/za-routes/latest.txt -O /tmp/zaroutes
size=`stat -c '%s' /tmp/zaroutes`

if [ $size -gt 0 ]; then
  mv /tmp/zaroutes "$ROUTEFILE"
fi

It downloads the routes file from cocooncrash's site (he gets them from local-route-server.is.co.za, aggregates them, and publishes every 6 hours). Run it now to seed that file.

Now some International-only routes. I use IS local DSL, so SAIX DNS queries should go through the SAIX connection even though the servers are local to ZA.

My /etc/network/routes-intl-dsl contains SAIX DNS servers and proxies:

196.25.255.3
196.25.1.9
196.25.1.11
196.43.1.14
196.43.1.11
196.43.34.190
196.43.38.190
196.43.42.190
196.43.45.190
196.43.46.190
196.43.50.190
196.43.53.190
196.43.9.21

Now we can tell /etc/network/interfaces about our connections so that they can get brought up automatically on bootup:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface eth0 inet static
        address 10.0.0.1
        netmask 255.255.255.0

auto local-dsl
iface local-dsl inet ppp
        provider local-dsl

auto intl-dsl
iface intl-dsl inet ppp
        provider intl-dsl

For DNS, I use dnsmasq, hardcoded to point to IS & SAIX upstreams. My machine's /etc/resolv.conf just points to this dnsmasq.

So something like /etc/resolv.conf:

nameserver 127.0.0.1

/etc/dnsmasq.conf:

no-resolv
# IS:
server=168.210.2.2
server=196.14.239.2
# SAIX:
server=196.43.34.190
server=196.43.46.190
server=196.25.1.11
domain=foobar.lan
dhcp-range=10.0.0.128,10.0.0.254,12h
dhcp-authoritative
no-negcache

If you haven't already, you'll need to turn on ip_forward. Add the following to /etc/sysctl.conf and then run sudo sysctl -p:

net.ipv4.ip_forward=1

Finally, you'll need masquerading set up in your firewall. Here is a trivial example firewall, put it in /etc/network/if-up.d/firewall and make it executable. You should probably change it to suit your needs or use something else, but this should work:

#!/bin/sh
if [ $IFACE != "eth0" ]; then
  exit;
fi

iptables -F INPUT
iptables -F FORWARD
iptables -t nat -F POSTROUTING
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i eth0 -s 10.0.0.0/24 -j ACCEPT
iptables -A INPUT -i ppp+ -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -j DROP
iptables -A FORWARD -i ppp+ -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i eth0 -o ppp+ -j ACCEPT
iptables -A FORWARD -j DROP
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o ppp+ -j MASQUERADE

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Split routing

Very useful for cutting and pasting :) thanks so much for this detailed, concise and well written article.

Error

I keep getting this when I try to seed the local routes file:

"/etc/cron.daily/za-routes: 9: [[: not found"

What is the reason for this? The /etc/network/routes-local-dsl file remains empty so the script is stalling?

Thanks

Bashism

Whoops, that looks liike a bashism. Change the double square brackets to single ones.

Wireless?

Hi,

I realise I'm making my life difficult but I would like to have this very set up except only on my computer; not on the family pc. That way split routing only takes place on my pc (for local torrenting) , but everywhere else I couldn't care and they can all access through the intl account like normal. Thus I assume masquerading isn't necessary, because there is no "internet connection sharing" (sorry to use the windows lingo) that needs to take place.

Further, it makes sense not to have the default connection set up in the router itself (a mega 105W for interests sake) and rather allow each computer to make it's own pppoe connection. Problem is then you have 2 or 3 computers (including my split routed pc) all dialling the telkom account independantly - wouldn't that be a problem?

I tried to set it up as per your instructions with the exception of the dnsmasq (because I don't use it - rather just use 10.0.0.2) and left out masquerading too. Also I replaced etho0 with wlan0 thinking this would move the connection through the wireless. In my network connections dialogue I've got "ifupdown (wlan0)" under the wired tab - telling this set up is specific for wired connections - how can I tell it to dial through the wireless connection (is that even possible?)

My wireless was not working at all, because I coudn't even connect to 10.0.0.2. When i backtraced and removed the /etc/network/interfaces changes it was all back to normal.

Thanks
Grant

Re: Wireless?

Hi.

Yes you wouldn’t need masquerading.

Personally, I prefer to have the router handle all the ADSL-related technicalities, because then connecting new machines to the network is simpler (and usually requires no configuration at all). More generally, it just leads to a neater network - that’s how IP networks are supposed to be. However, yes you can have multiple PPPoE connections (and I think telkom technicians love reconfiguring PCs to do PPPoE directly). I think the limit is 4, but I’m not sure. Test.

Yes you can connect via wireless. It’s an ethernet network like any other. However the instructions I gave were for a server more than a desktop, you’d probably not want to implement it quite like tihs on a desktop. So as to your questions about dialogs, I have no idea what you are referring to. To use these instructions on a wireless network, you’ll have to configure the wireless network with /etc/network/interfaces, the old-school debian way.

Wireless

Hi,

Theoretically wireless should work, BUT if I understand your setup correctly it probably wont. Most wireless routers I've worked with doesn't have a layer 2 switch between the wireless and ethernet networks but rather a layer 3 switch (router). This means all IP packets (layer 3) can freely move from the wireless to the ethernet and your modem, but PPPoE packets (layer 2) is blocked.

If your wireless device is running linux like openwrt or ddwrt you can work around this by installing a pppoe forwarder like pppoe-relay (I beleive ddwrt has this installed by default) which will forward the pppoe packets to the ethernet segment where your modem is.

echo 1 > /proc/sys/net/ipv4/ip_forward

Hi

Thanks for the tutorial, it helped a lot.

But on my Ubuntu 8.10 server I needed to enable ip forwarding:

"echo 1 > /proc/sys/net/ipv4/ip_forward"

Regards,

sysctl.conf

You should rather set net.ipv4.ip_forward=1 in sysctl.conf.

Brilliant

Thanks a stax - I had to make slight changes as I use roaring penguin pppoe - but it all worked out perfectly.

Pingback

[...] recently followed tumbleweed’s split routing post (his post explains the odd reasoning why any one would go through all this effort) after doing this [...]

Test

Hi thanks for this great tut, stupid question, how will I know if it worked or not?

If it worked

This does assume some medium-level Linux networking knowledge, in which case you can answer that yourself.

Look at the packet counters on both ppp interfaces, do they rise?

Traceroute to local and international sites. The next hop for local should be different to international.

Single computer setup

I followed the instructions - worked great.

I put usepeerdns in the local-adsl, and made an entry

resolv-file=/etc/ppp/resolv.conf

in dnsmasq.conf.

Thanks!!

I lub you :(

It's working, though I messed up somewhere along the way. Took me hours of fiddling and doing things that probably broke more stuff than it fixed, and then had to fix those things too, didn't help that it was around three hours after taking a sleeping tablet :(

Single computer setup here so went without dnsmasq and just put the relevant name servers in /etc/resolv.conf
Totally want to get a decent router, can't find any command line reference for a Planet ADE 4400, though maybe I just don't know what I'm doing :S

Many, many thanks.

Pingback

[...] Split Routing on Ubuntu Server Config Guide Ubuntu Split Router Config Guide Based on http://tumbleweed.org.za/2008/09/19/split-routing-debianubuntu [...]

Account

this is for two account is it? (load balancing??cmiiw)

i only have 1 dsl acc n i want to split the local n the intl traffic,

can i use this to do that?

sorry if it's a stupid question

thanx a bunch

spot on

That's exactly what this is for (splitting local and international).

Multiple PPPoE connections to single NNTP server

Thanks, This guide worked like a charm. Just some pebkac where I forgot to make a scripts executable....

Anyways, I am busy moving away from my DDWRT setup using Gatecrashers script to an atom based Ubuntu 10.4 . The only thing that I lost was the ability to allow multiple PPPoE connections to access the same news server on the same port. Let me explain it like this:

So I would enter the the settings below on the left in my NNTP software, and the iptables rules would send it over to the same server and port, just on seperate PPPoE connections so that dont go over the connection limit per PPPoE session. i.e.:

news_server:119 => this would get routed normally via the local only routes defined to news_server:119
news_server:219 => this would go over ppp2 and be SNAT'ed to news_server:119
news_server:319 => this would go over ppp3 and be SNAT'ed to news_server:119
news_server:419 => this would go over ppp4 and be SNAT'ed to news_server:119

I just cant seem to get the iptables rules right for this to make it working.

Do you have some advice on this?

re: Multiple PPPoE connections to single NNTP server

Sounds abusive, check those terms of use.

I think fwmark is going to be the solution to this.

Pingback

[...] Some years ago, before the age of cheap international access on local ISPs arrived here, dual-homing (or n-homing, depending on how pimp you were) on your residential connection was quite the fashion among .za tech-heads. But not the fancy sort with BGP and decent best-route selection, just a really grubby sort: two accounts, one local (as in .za routing table) and one international. You can read up about the full setup over here on Stefano’s site. [...]

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.