Stunnel X-Forward-For (XFF) with HAProxy and the PROXY Protocol

When using proxies such as stunnel and HAProxy it's easy to loose track of the client source IP address. This occurs for example when HAProxy is used in it's default configuration to load balance a number of back-end web servers. By default, the source IP address of the packet reaching the web servers is the IP address of the load balancer and not the IP address of the client. One way around this is to enable X-Forward-For headers for HAProxy (the default for Loadbalancer.org appliances) and configure the web servers to track the IP address in this header. For more details on enabling this for IIS and Apache web servers, please see IIS and X-Forwarded-For Headers and Apache and X-Forwarded-For Headers.

For more complicated scenarios where SSL termination is also required on the load balancer and the original source IP address is still required, additional steps are needed.

Stunnel & HAProxy

By default, in the above example the IP address in the X-Forward-For header reaching the Web Servers is the load balancers own IP address. This is because stunnel is not transparent by default. To force stunnel to pass the original client IP address the protocol directive in stunnel must be added and set to proxy as shown below:

[STunnel]
cert = /etc/loadbalancer.org/certs/STunnel.pem
ciphers = ALL
accept = 192.168.10.10:443
connect = 192.168.10.10:80
options = NO_SSLv2
options = CIPHER_SERVER_PREFERENCE
protocol = proxy
TIMEOUTclose = 0

Note: For more details on the protocol option please refer to this page.

This enables stunnel to pass the original client IP address to HAProxy.

HAProxy must also be configured to accept and use this information by inserting the accept-proxy & option forwardfor directives:

listen L7-VIP
bind 192.168.10.10:80 transparent accept-proxy
mode http
balance leastconn
cookie SERVERID insert nocache indirect
server backup 127.0.0.1:9081 backup non-stick
option http-keep-alive
option forwardfor
option redispatch
option abortonclose
maxconn 40000
server WEB1 192.168.10.11:80 weight 100 check inter 6000 rise 2 fall 3 minconn 0 maxconn 0 on-marked-down shutdown-sessions
server WEB2 192.168.10.12:80 weight 100 check inter 6000 rise 2 fall 3 minconn 0 maxconn 0 on-marked-down shutdown-sessions
server WEB3 192.168.10.13:80 weight 100 check inter 6000 rise 2 fall 3 minconn 0 maxconn 0 on-marked-down shutdown-sessions

Note: For more details on the accept-proxy option please refer to this page, for more details on the proxy protocol please refer to this page.

Fortunately, stunnel & HAProxy can easily be configured in this way using the built-in Web User Interface:

a) Configuring stunnel

Using the WUI option: Cluster Configuration > SSL Terminatonclick [Modify] next to the relevant stunnel Virtual Service and enable the option Enable Proxy Protocol as shown below:

NB. When you select the target HAproxy VIP from the drop down list - it will automatically modify HAProxy to accept PROXY protocol AND insert the XFF header.

b) Check HAProxy Configuration (optional)

Using the WUI option: Cluster Configuration > Layer 7 - Virtual Services click [Modify] next to the relevant HAProxy Virtual Service and check that the correct options have been enabled Set X-Forward-for Header(enabled by default) and Proxy Protocol as shown below:

c) Reload Services

Finally, reload HAProxy and stunnel to apply the changes using the restart buttons that appear at the top of the screen as shown below:

NB. Reload is completely safe - no traffic will be dropped.

As mentioned at the start you'll also need to ensure that your web servers are correctly configured to log the X-Forward-For header details:
/blog/category/x-forwarded-for-header/

But as far as configuring the load balancer is concerned, that's it!

Did you know, HAProxy can even use the XFF for a stick table?

An awesome feature of HAProxy is that you can define a stick table (persistence) based on the client source IP contained in the XFF header (or from the PROXY protocol).

Simply define your stick table as follows:

    option forwardfor if-none
    stick on hdr(X-Forwarded-For,-1)
    stick on src
    stick-table type string len 64 size 10240k expire 2m     

How cool is that?

WARNING: Obviously you MUST TRUST your up-stream proxy!