How to stop TPROXY when used with HAProxy breaking clients in the real server subnet

Dealing with clients coming from the same subnet as the real servers when using Tproxy can be a pain. It creates an issue because once Haproxy is running transparently, it will allow the real server to see the client IP so the real server will reply directly back to the client bypassing the load balancer.

The obvious solution to this is to mess with the real servers routing tables as we often do with other load balancing methods such as L4 single subnet NAT mode as shown below for Linux Real Servers:

For L4 single subnet NAT mode we need to modify the local network route by changing to a higher metric.

First, remove the local subnet route :

route del -net 10.0.0.0 netmask 255.0.0.0 dev eth0

Then re-add with a higher metric :

route add -net 10.0.0.0 netmask 255.0.0.0 metric 2000 dev eth0

Next we need to make sure that local network access uses the load balancer as its next hop :

route add -net 10.0.0.0 netmask 255.0.0.0 gateway 10.0.0.21 metric 0 dev eth0

Once configured, any local traffic (same subnet) is then handled by this manual route and any external traffic is handled by the default gateway (which also points at the load balancer).

So what about if we don’t want to change the local subnet routes?

Those of you that have read our HAProxy with Tproxy Blog will know that adding a “source” configuration line to the VIP is the final piece to the puzzle when configuring transparent 2 arm HAProxy.

Haproxy also allows you to specify the “source” line for each real server or even by backend. This allows you to use ACL’s to decide if traffic is sent to a transparent server or not. This method is especially useful when you don’t want to start messing with routing tables other than changing the default gateway. Obviously if you don’t need client transparency for local subnet traffic, but just want the client IP for the logs you can still use X-Forwarded-For.

The first example is for a simple listen style VIP where you have a single server (server 3) set aside for non transparent traffic :

listen VIP_Name 192.168.2.87:80
mode http
option forwardfor
cookie SERVERID insert nocache indirect
acl local_subnet src 10.0.0.0/8
server server1 10.0.0.60:80 weight 1 cookie server1 check source 0.0.0.0 usesrc clientip
server server2 10.0.0.61:80 weight 1 cookie server2 check source 0.0.0.0 usesrc clientip
server server3 10.0.0.60:80 weight 0 cookie server3 check
server backup 127.0.0.1:80 backup
use-server server3 if local_subnet

So what are we doing here? We have an ACL “acl local_subnet src 10.0.0.0/8” to identify clients from the server side subnet and “use-server server3 if local_subnet” which selects our non transparent server. In this example I have set the non transparent server to a weight of 0 so it won’t be selected for normal client traffic.

Next is a more complicated example utilising a frontend/backend style config which allows for an entire non transparent backend.

frontend http
bind 192.168.2.87:80
mode http
acl local_subnet src 10.0.0.0/8
use_backend www-non-transparent if local_subnet
default_backend wwwbackend www
mode http
option forwardfor
cookie SERVERID insert nocache indirect
source 0.0.0.0 usesrc clientip
server server1 10.0.0.60:80 weight 1 cookie server1 check
server server2 10.0.0.61:80 weight 1 cookie server2 check
server backup 127.0.0.1:80 backup
backend www-non-transparentmode http
option forwardfor
cookie SERVERID insert nocache indirect
server server1 10.0.0.60:80 weight 1 cookie server1 check
server server2 10.0.0.61:80 weight 1 cookie server2 check
server backup 127.0.0.1:80 backup

As often is the case, there are many ways to achieve this, so mileage for this as a solution will vary. However it’s a lot less disruptive than changing routing tables and allows same subnet traffic to work so long as they can already route to the VIP in the second subnet.