Supporting two default gateways on your Load balancer with PBR (Policy Based Routing)

Occasionally we get asked questions about supporting multiple default gateways. Customers may have our appliance in two networks, each with it’s own gateway to the internet. Alternatively it may just be a big complex network with multiple routing paths and stateful devices which are fussy about seeing the whole connection so things get dropped. They often want to host one VIP using one connection and a second VIP using the second connection. If you simply set two default gateways on a Linux server it will usually just round robin between them so packets can take the wrong routing path back to the client.

Imagine the following configuration on your Loadbalancer appliance:

Network 1 :
Eth0 = 172.16.0.100/16
Gateway = 172.16.0.1 (Current LB Default Gateway)
VIP = 172.16.0.254

Network 2 :
Eth1 = 192.168.0.100/24
Gateway = 192.168.0.1
VIP = 192.168.0.254

In the above example the gateway for each network will use a different ISP, users come in from the public internet and are NAT’d to the VIP 172.16.0.254. This works as expected because replies will go back via the default gateway therefore returning via the same path it arrived.

Now imagine another external user comes in via your second connection and is NAT’d to the VIP 192.168.0.254. In this example, replies will still try to go back via the default gateway which is currently set to 172.16.0.1 so replies return via the wrong ISP!

Don’t get me wrong it’s not always the case that your return path needs to be the same in the LAN but when dealing with 2 ISP’s and external public users then it does.

Enter Policy Based Routing…

Policy Based Routing (hereafter simply referred to as PBR) is a clever way to give us more control over which routing path connections will take. It allows us to specify custom routing tables and then add rules offering fine grained control over which routing table a connection will use.

Common commands:

To list the current rules execute the following command : ip rule ls
To list routes from a specific table : ip route ls table
To add a rule : ip rule add <to/from/fwmark> <ip-address/fwmark-id> table

Going back to the above example we could do something like the following :

echo "200 path2" >> /etc/iproute2/rt_tables
ip route add default via 192.168.0.1 table path2
ip rule add from 192.168.0.100 table path2
ip rule add from 192.168.0.254 table path2

The first line above creates an additional routing table called “path2”.
The second line adds our additional default gateway into path2.
The third and fourth lines make any traffic sourced from Eth1’s IP or the VIP use the new routing table.

UPDATE!!!

From v8.2.2 we now have a new and simpler PBR script available. It requires less configuration and automatically copies your existing local routes over making it much simpler to use. The old one still exists on the appliance should anyone run into issues while the newer script is still young but also for backward compatibility. However, with the addition of the new script it should make it much easier to utilise a separate default gateway for any VIP address.

Instructions:

First, you need to create a config file under “/etc/pbr.d/”, the config file should be called <something>.conf as the script ignores files without the “.conf” extension. The config file currently requires two variables as outlined below:

VIP=<Local-IP-Address>
GW=<Default-Gateway>

VIP_Name.conf
VIP=192.168.0.254
GW=192.168.0.1

You can use any IP address of the appliance for the VIP variable so if setting up a separate management interface which requires a separate default gateway simply create a config file as you would for a VIP.

Once complete and you’ve configured one or more config files you can enable the new “pbr” service, it’s fairly simple currently so we suggest stopping it before adding extra config files!

service pbr start

chkconfig pbr on

service pbr stop

That should be it! You’ve completed your first soiree with PBR and hopefully have a simple configuration up and running. Should your situation be more complex or you need assistance extending the rules then our support staff will be happy to help so please email support@loadbalancer.org.

Old Script(Deprecated)

In an attempt to make things easier we’ve included a script on the appliance to automate the process for two networks, two gateways and VIP’s in each. The script can be found on the appliance under : /etc/init.d/policy-based-routing

#!/bin/bash
# Version 1.0
# Maintained By: Aaron West
# chkconfig: 345 99 01
# description: Policy Based Routing rules to support multiple routing tables.
# processname: policy-based-routing

# Program Paths
IP=/sbin/ip
IPTABLES=/sbin/iptables

# Default Routing Path(current external interface)
NIC1=
NET1=
MASK1=
IP1=

# Additional Routing Path
EXNIC1=
EXIP1=
EXNET1=
EXMASK1=
EXGW1=
EXVIP1=

case "$1" in
start)
echo -n "Starting policy-based-routing: "

# Check path2 exists if not create it
grep -e "path2" /etc/iproute2/rt_tables
if [ "$?" -gt "0" ]; then
echo "200 path2" >> /etc/iproute2/rt_tables
fi

# Build additional Routing rules.
${IP} route add ${NET1}/${MASK1} dev ${NIC1} src ${IP1} table path2
${IP} route add ${EXNET1}/${EXMASK1} dev ${EXNIC1} src ${EXIP1} table path2
${IP} route add default via ${EXGW1} table path2

# Add rules.
${IP} rule add from ${EXIP1} table path2
${IP} rule add from ${EXVIP1} table path2
touch /var/lock/subsys/policy-based-routing
;;
stop)
echo -n "Shutting down policy-based-routing: "

# Remove additional Routing rules.
${IP} route del ${NET1}/${MASK1} dev ${NIC1} src ${IP1} table path2
${IP} route del ${EXNET1}/${EXMASK1} dev ${EXNIC1} src ${EXIP1} table path2
${IP} route del default via ${EXGW1} table path2

# Remove Rules.
${IP} rule del from ${EXIP1} table path2
${IP} rule del from ${EXVIP1} table path2
rm -f /var/lock/subsys/policy-based-routing
;;
restart)
$0 stop; $0 start
;;
*)
echo "Usage: policy-based-routing {start|stop|restart}"
exit 1
;;
esac

With the above script you can set the relevant variables for your network giving you a simple solution persistent across reboots :

# Default Routing Path(current external interface)
NIC1=eth0
NET1=172.16.0.0
MASK1=16
IP1=172.16.0.100

# Additional Routing Path
EXNIC1=eth1
EXIP1=192.168.0.100
EXNET1=192.168.0.0
EXMASK1=24
EXGW1=192.168.0.1
EXVIP1=192.168.0.254

Should you require you can also extend the script fairly easily to support more networks and routing paths.

To enable the script once configured please execute the following commands:

chkconfig --add policy-based-routing
service policy-based-routing start