How to overcome the peculiarities of load balancing Cisco ACI environments

We've had a number of inquiries recently from customers having issues deploying load balancers in their Cisco ACI environment. So we set to work finding a solution for the peculiarities of this network infrastructure.

During the course of our research, we came across this great workaround written by Michael Van Kleij at Cisco, which we're sharing here for your reference. Feel free to also share this with your customers in case they need help.

This is a universal quirk of Cisco ACI environments, so the solution is valid for all load balancer appliances.

Cisco ACI quirks

Cisco Application Centric Infrastructure (ACI) decouples the network control plane from the data forwarding plane.

The advantage of this architecture is that it can be used to optimize, simplify, and accelerate, network deployment, configuration, and management.

However, one disadvantage is that this Cisco ACI network solution turns off certain settings by default, which can therefore introduce some complexities when it comes to failing over from one load balancer appliance to another and successfully implementing Direct Routing (DR).

Problem 1: Cisco ACI and load balancer failover

How most networks work

In normal circumstances, when a failover happens, Virtual IP's are moved from the former Primary appliance to the Secondary load balancer appliance.

In a typical network, the MAC address-to-IP association is achieved by means of Address Resolution Protocol (ARP). Devices attached to a switch send a broadcast frame to the destination MAC (FF:FF:FF:FF:FF:FF) asking who has a particular address. As the request is broadcast, it's sent to all devices in the broadcast domain. The owning device then sends a response of its MAC address to the originator of the broadcast. This creates the binding of the MAC address-to-IP relationship in the ARP cache of the original requester.

The crucial thing to remember here is that ARP is sent by a device that wants to know the location of an IP address in a network.

When a load balancer failover happens, the secondary load balancer has to take over the networking. The principal mechanism for this, from the point of view of the network, is Gratuitous ARP (GARP). GARP is sent by a device that is telling the network the location of an IP address. This will cause the devices on the network to update their ARP tables with this new information.  This ensures that any new communication to the VIPs is now sent to the Secondary (and now active) load balancer, instead of the Primary (and now failed) load balancer.

How Cisco ACI works

However, in the case of the Cisco ACI, the network devices are much smarter, as the switching fabric learns the mapping of the network from every packet that flows through, not just the ARP packets. Being so much smarter than everything around you is both a blessing and a curse (or so I am told!).

For the most part, it means that everything works much better. However, sometimes the dumber things are actually right, although the smarter things that are in control don’t believe the dumb things as surely they are dumb and can’t be trusted to be correct (take from that what you will)....?!

This is essentially what is going on with ACI, it employs much more complicated and sophisticated mechanisms to learn and map the network than just ARP, to make everything as stable and efficient as possible. However, this does mean that sometimes ACI is confidently wrong in the location of a device and can take too long to age out an endpoint mapping, which means that traffic is incorrectly being sent to an unreachable destination until it is updated.

Having an unreachable Virtual IP on a load balancer is the antithesis of high availability!

This issue with the virtual IP’s occurs when the load balancer is connected to the Cisco ACI environment with Unicast Routing enabled.

Why?

Essentially, by default, Cisco ACI learns the source IP address of the packets via data-plane, which is called IP data-plane learning. This means the already learned local IP endpoints are retained. To make matters worse, Cisco ACIs learn based on every IP packet, not just ARP packets.

So...?

What happens is that after failover, the new Primary appliance sends out GARPs for VIP/SNIPs and the Cisco ACI leaf switch updates the MAC/IP information in its table, which is called the endpoint table.

But...?

When the former primary times-out or resets old connections, TCP ACK’s and RSTs, or UDP datagrams sent with the MAC belonging to the VIP address, will unintentionally be re-learned by the wrong data-plane endpoint.

Meaning...?

Cisco ACI then forwards packets destined for the VIP back to the old primary appliance because the end-point moves back to the old location, causing an outage.

To unlearn this behaviour, the ACI fabric must be told not to retain the VIP address.

Problem 2: Cisco ACI and Direct Server Return

How most networks work

Direct Server Return load balancing (DSR) (which Loadbalancer.org calls Direct Routing (DR)), is a super simple, transparent, and fast way of load balancing application servers at Layer 4.

It directs traffic through a Virtual IP (VIP) to multiple Real Servers within the same Layer 2 network domain as the load balancer. Because in traditional networks, the load balancer and Real Servers reside in the same Virtual Local Area Network (VLAN) and the Real Servers don't respond to ARP messages, Layer 4 DR load balancing is typically straightforward.

These servers respond directly to the client, bypassing the load balancer for return traffic, meaning Direct Server Return (DSR/DR) load balancing maximizes the throughput of return traffic and allows for near-endless scalability.

This is achieved by switching the destination MAC address for a request that is received to be that of one of the real server MAC addresses. The request still has the original destination IP address of the VIP but it is forwarded on to one of the real servers at layer 2. This is why the real servers need to be configured with the IP address of the VIP, so that these requests can be processed and are not dealt with as misrouted packets. Additionally, the real servers are configured to not respond to ARP requests for the VIP address because, otherwise, the network would start circumventing the load balancer and sending requests directly to the real servers.

This load balancing method contrasts with other Layer 4 load balancing techniques that utilize source NAT, or when the load balancer serves as a gateway.

How Cisco ACI works

However, in Cisco ACI environments, the process differs.

As explained above, unlike standard switches that don't learn IP address locations, the Cisco ACI fabric does learn these. And it learns from every IP packet, not just ARP packets.

This time, this Cisco ACI quirk poses a challenge for Direct Server Return (DSR) load balancing. What happens in ACI is that the fabric learns the eventual destination of one of the real servers and then sends all of the traffic directly to that one server. This not only removes the load balancer entirely from the process, it also sends ALL of the traffic to just one real server no matter how many real servers are in the pool.

To mitigate this, the fabric again needs to be instructed not to learn the VIP address.  

Here's how....

Solution: Overcome both of these Cisco ACI problems by enabling GARP

Anyone trying to implement Cisco ACI Direct Routing (DR) or failover can use the following configuration, which will solve both problems.

The steps that need to be taken are as follows:

• Step One: Configure the L4-L7 Virtual IP
• Step Two: Configure GARP-based EP move detection
• Step Three: Verify the solution

Collectively, this will ensure the load balancer's IP address can be added as a Cisco Layer 4 to Layer 7 Virtual IP, ensuring proper functionality within its ACI fabric.

Further information on this solution can be found in this excellent blog by Michael Van Kleij from Cisco.

Step One: Configure the L4-L7 Virtual IP

• Go to Tenant, Application Profiles, Your Application Profile, Your EPG, L4-L7 Virtual IPs
• Right click and choose 'Create L4-L7 Virtual IP'
• Enter the IP address of the VIP
• Click 'Submit'

Step Two: Configure GARP-based EP move detection

• Go to Tenant, Networking, Bridge Domains, Your Bridge Domain, L3 Configuration
• Check the box at GARP-based detection

• Click 'Submit'

Now, with a Virtual IP configured for the Endpoint Group (EPG), this IP address is solely learned through ARP messages and will relocate based on Gratuitous ARP (GARP) messages. This setup facilitates failover in case of a load balancer failure.

Step Three: Verify the solution

To ensure the fabric accurately learns the MAC address of the load balancer, you can verify using the following command:

aci-leaf-007# show system internal epm endpoint ip 10.10.10.10

This command will display the MAC address of the system, which should match the MAC address of the load balancer. Additionally, you will observe the line "EP Flags" at the bottom, which should indicate "VIP."

Here's an example of the kind of information you will receive:

MAC : 0050.5693.95a3 ::: Num IPs : 1
IP# 0 : 10.10.10.10 ::: IP# 0 flags : VIP
...
---output truncated---
...
EP Create Timestamp : 31/02/2024 12:34:56.6969
EP Update Timestamp : 31/02/2024 12:34:56.6969
EP Flags : VIP|psvi|
::::

Conclusion

And there you have it. How to deploy load balancers in a Cisco ACI environment in order to successfully implement failover and Direct Server Return load balancing/Direct Routing.

If you or your customers are experiencing issues, please don't hesitate to reach out to our lovely Support team.