How to limit connections with HAProxy, iptables or nftables

Recently I was asked to develop a method to rate limit connections passing through our load balancer to just one!  

Why?

One of our customers had a critical piece of proprietary software in their architecture that was only licensed for one connection at a time. But annoyingly the licensed software had no built in enforcement mechanism. It would just log the overrun in connections and give our customer a hefty increase in costs!

So they wanted to be absolutely sure that they didn't break the licensing restriction. And could we help?

These were the requirements:

• Allow only 1 connection per client source IP
• Maintain source IP persistence
• Make it easy to manage

So, let's investigate my options...

How rate limiting normally works

A normal real world scenario for rate limiting would be when a piece of software that is being load balanced has a performance limitation, so you would want to limit the total number of connections going through it.

Now of course I know there are various blogs on the interwebz covering this functionality by using HAProxy or Linux's kernel filter (netfilter) and configuring it using an utility program such iptables or nftables.

However, my use case is a bit different

The specific application I was dealing with didn't have this feature fully operational and, of course, by default the load balancer was not aware of this limitation.

Therefore at the load balancer level I needed to limit the connections being queued or established once a free slot became available, regardless of how many nodes were running behind the load balancer. How I did this is shown below with a couple of options.

How to limit connections using iptables

The first and most obvious method I considered was to filter by Source MAC Address as we can find the client's MAC addresses by creating a small script and placing it's contents under /etc/rc.firewall , making it easier to maintain and persistent upon reboot.

# Define the MAC addresses below space separated
MAC=(0a:0b:0c:0d:0e:0f 08:00:27:2a:f3:1a fc:aa:14:9b:71:36)
# Change below to be the IP address of the Gateways VIP
VIPIP=192.168.77.200
VIPPORT=80
# Looping through the list of MAC addresses defined above
if [ -n "$MAC" ]
then
for MAC in ${MAC[@]}
do
iptables -A INPUT -p tcp --syn -m mac --mac-source $MAC -d $VIPIP --dport $VIPPORT -m conntrack --ctstate NEW -m connlimit --connlimit-above 1 -j REJECT
done
fi

Haters Some of you will say distributions are now deprecating iptables and are migrating to nftables. So here is how you would do it using nftables:

How to limit connections using nftables

## Create table and chain to filter the traffic
nft add table ip filter
nft add chain ip filter input { type filter hook input priority 0; }
# Create dynamic sets
# For source IP
nft add set ip filter ip_limit_set '{ type ipv4_addr; flags dynamic; }'
# For source MAC addresses
# nft add set ip filter mac_limit_set '{ type ether_addr; flags dynamic; }'

NOTE: A timeout can be set with our desired timeout value such as timeout 1m; , otherwise records will be kept indefinitely.

# the rules allowing 1 connection per source IP
# Replace 192.168.1.100 with your VIP IP
nft add rule ip filter input ct state new ip saddr @ip_limit_set ip daddr 192.168.1.100 drop
nft add rule ip filter input ct state new add @ip_limit_set { ip saddr } accept
# the rules allowing 1 connection per source MAC address
# Replace 192.168.1.100 with your VIP IP
nft add rule ip filter input ct state new ether saddr @mac_limit_set ip daddr 192.168.1.100 drop
nft add rule ip filter input ct state new ip daddr 192.168.1.100 add @ip_limit_set { ip saddr } accept
# List the ruleset we've just created
nft list ruleset
# Save the ruleset and make it persistent upon reboots
nft list ruleset > /etc/nftables.conf

How to rate limit connections using HAProxy

Unless something like Quality Of Service, Smart Queue Management or similar can be done upstream, filtering traffic using nftables or iptables is great when the system administrator allows the load balancer engineer to install or to configure such tools.

I'm not sure if its just me, but it seems Proxy ARP has become the preferred method of late to enable traffic to traverse remote subnets. So rate limiting per MAC address may not be the best way, as entire networks can be filtered out.
As opposed to the iptables example, the nftables example provided does both IP and MAC filtering.

Without further ado, if its not possible to use any utility tools for packet filtering then we could implement rate limit by IP using an ACL as per this HAProxy configuration:

global
    daemon
    stats socket /var/run/haproxy.stat mode 600 level admin
    pidfile /var/run/haproxy.pid
    maxconn 40000
    tune.bufsize 16384
    tune.maxrewrite 1024
    tune.ssl.default-dh-param 2048
defaults
    mode http
    no option http-use-htx
    balance roundrobin
    timeout connect 4000
    timeout client 42000
    timeout server 43000
# Default peers to sync the stick tables per VIPs between the load balancers
peers loadbalancer_replication
    peer lbmaster localhost:7778
    peer lbslave localhost:7778
# create another peer section and stick table specifically for limiting
peers loadbalancer_replication2
peer lbmaster 192.168.77.221:9999
peer lbslave 192.168.77.222:9999
table connlimit type ip size 1m expire 1h store conn_cur
listen RatelimitVIP
    bind 192.168.77.201:80 transparent
    default-server on-marked-up shutdown-backup-sessions
# Works in both TCP and HTTP mode
    mode tcp
    balance leastconn
# Default stick table for source IP persistence
    stick on src
    stick-table type ip size 10240k expire 1hm peers loadbalancer_replication
    server :backup 127.0.0.1:9081 backup non-stick
    timeout client 30m
    timeout server 30m
    acl :connection_via_termination always_false
    option tcpka
    option redispatch
    option abortonclose
    maxconn 40000
    server gateway1 192.168.77.135 id 2 weight 100 check port inter 4000 rise 3 fall 3 slowstart 8000 minconn 0 maxconn 12 on-marked-down shutdown-sessions
    server gateway2 192.168.77.136 id 3 weight 0 check port inter 4000 rise 3 fall 3 slowstart 8000 minconn 0 maxconn 0 on-marked-down shutdown-sessions
# ACL to read the entries from the additional stick table
    acl connection_abuse sc0_conn_cur gt 1
    tcp-request connection track-sc0 src table loadbalancer_replication2/connlimit
    tcp-request connection reject if connection_abuse

And, hey presto, problem solved! A working solution. Something I'm going to keep in my back pocket for next time.

Hang on, isn't that all a little bit complicated?

Well, yes.  But I've been describing how you can do this for free with powerful open source tools like the awesome HAProxy.

When it came to implementing this for the customer, it was a lot easier with our appliance...

How to rate limit using Loadbalancer's Enterprise ADC

Because our appliance is based on a number of open source tools, including HAProxy. We've made it pretty simple to implement custom rules for rate limiting. In fact just about anything you can do with iptables or HAProxy is much easier on our appliance.

All I needed to do was add the 3 lines below as a Free Type ACL on the corresponding Layer 7 Virtual Service:

# ACL to read the entries from the additional stick table
acl connection_abuse sc0_conn_cur gt 1
    tcp-request connection track-sc0 src table loadbalancer_replication2/connlimit
    tcp-request connection reject if connection_abuse

And then add the 4 lines below as a manual configuration:

# Peer node replication for the source IP connection limit table
peers loadbalancer_replication2
peer lbmaster 192.168.77.221:9999
peer lbslave 192.168.77.222:9999
table connlimit type ip size 1m expire 1h store conn_cur

I could have combined them by using another backend for the stick table, but I like to keep the Free Type rules in nice and easy manageable chunks.

Here's a video explaining just how effortless it is to implement ACL rules with the Loadbalancer Enterprise web interface:

Oh and I forgot to mention to view the records from the connlimit table we call:

watch "echo "show table loadbalancer_replication2/connlimit" | socat stdio /var/run/haproxy.stat"

Anyway, I thought I'd share here in case others might need a similar workaround.

I hope it's been helpful. If you have any questions/suggestions feel free to comment below!