NGINX and X-Forwarded-For Header (XFF)

The X-Forwarded-For Header is a simple yet powerful solution to a very common problem. I'm not sure why, but for some reason it also seems to cause a lot of confusion.

NGINX is often deployed as a cluster behind a layer 7 load balancer (Reverse Proxy). Being a proxy implementation, Layer 7 offers a whole host of options such as ACLs, clever persistence methods, the ability to add/remove/modify HTTP headers, and so on.

So what's the problem?

A reverse proxy is NOT source IP address transparent.

This is a pain when you need the client source IP address to be correct in the logs of the backend servers.

I can think of a couple of solutions to this problem:

1. Implement a fully transparent two-arm reverse proxy using TPROXY (yuk!).
2. Configure the load balancer to add an X-Forwarded-For Header with the source IP of the client.
3. Use Layer 4 instead (although I guess you've already ruled that out?)

Personally, I think that by far the easiest option when load balancing a website/web application is to use the X-Forwarded-For Header.

With Loadbalancer.org, when you create a Layer 7 HTTP mode VIP configuration, the X-Forwarded-For Header is enabled by default. All you need to do is slighty modify the logging directive in the web server configuration (to tell it to use the header).

What about other reverse proxies?

NGINX can also be used as the load balancer of course:

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

Or you can use HAProxy (our preference):

      option forwardfor

How do you configure NGINX to use the X-Forwarded-For Header?

With NGINX, there are two ways the service can be modified to use the X-Forwarded-For Header. Which method you might use depends whether the NGINX binary was compiled with the option --with-http_realip_module . You can check if the module was included by running the following command: nginx -V and reviewing the output.

Option 1 - Altering the log directive format

This option can be implemented whether or not the --with-http_realip_module was specified at compilation, and modifies the format for the access_log directive to include the X-Forwarded-For Header contents.

In the configuration file /etc/nginx/nginx.conf you will need to change the entries:

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

To show the X-Forwarded-For Header contents first in the log line entries:

    log_format  main  '$http_x_forwarded_for - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent"';

    access_log  /var/log/nginx/access.log  main;

After making this change, the NGINX service will need restarting in the usual manner.
systemctl restart nginx

Option 2 - Using the 'real_ip' module

Assuming that NGINX has been compiled with the --with-http_realip_module option, the httpd or server stanza in the /etc/nginx/nginx.conf file needs to be modified with the set_real_ip_from and real_ip_header directives:

    # Directives for setting real_ip/XFF IP address in log files
    set_real_ip_from    192.168.101.10; #IP address of master LB
    set_real_ip_from    192.168.101.11; #IP Address of slave LB
    real_ip_header      X-Forwarded-For;

The real ip module is used to change client source IP address (and optionally, the port too) to the value stored in the specified header. The reason I have set the set_real_ip_from directive twice is that I was running an HA clustered pair of Loadbalancer.org appliances and this covers the base IP addresses from each node.

After making this change, the NGINX service will need restarting in the usual manner.
systemctl restart nginx

After completing the changes detailed in either option, the access log on the real servers should now show the X-Forwarded-For Header contents.

If you experience any issues when configuring this or have any questions, contact our support engineers support@loadbalancer.org and we'll do everything we can to assist.