The importance of adding a load balancer to your test environments

Back in a previous life, I worked on several different and exciting projects. Two things each of these projects had in common was that they were all very much run using ITIL as a basis for service management (plenty of paperwork and auditing), and also that they all had multiple test systems used for different purposes. Some were for development and script testing, whilst others were used for user acceptance testing (UAT) and operation performance testing (OPT), or deployment testing. Whatever the test system's final purpose was, they all offered certain functions that could be found in the final production environments.

From the non-exhaustive list of test systems above, certainly not all of the test systems required a fully stacked deployment that is representative of the production environment. However, there are certainly some environments that could benefit from having a load balancer added to the infrastructure.

Imagine my surprise when I joined Loadbalancer.org and found that a fair number of customers only run a production environment and, even worse, make changes to the environment on the fly (i.e. a lack of change control).

Why you need a test environment (or two)

What I’d like to do now is explain why you should have a test environment (or two!) deployed in your infrastructure.

1. Testing Code Changes - Hands up here who is guilty of making that “simple” code change in a production environment, only to find it has broken some other functionality. By implementing a test system (and by virtue a well-documented test strategy), you can ensure any code changes you have made work as expected and that you don’t unexpectedly break any other functionality.
2. Deployment and Rollback Testing - Before making a change to your production environment, you should really test the deployment and rollback of the change in a pre-production-based environment. Why you may ask? Simply put, this helps you not only understand the steps that are needed to be taken in order to deploy successfully into production but also (if the worst comes to it), helps you know how to rollback the changes quickly and efficiently to avoid production downtime.
3. Performance Testing - You are running a service that needs to be highly performant. Why then would you test and benchmark that performance on different infrastructure to that which your production system is running on? You shouldn’t. At a minimum, you should be testing on the same hardware as your production servers and be running through an infrastructure setup very similar to your production environment.
4. Patch Testing - We’ve all heard reports of companies that release patches (maybe feature enhancing or security-based), where on deployment it turns out that the patches break something (the recent print nightmare example comes to mind). If you can deploy the patches to a test environment first, it gives you time to work out whether the patch is safe to be deployed into production or whether it should be held back for a subsequent patch to be released (a patch for the patch if you will).
5. Support Testing - Let's just imagine that an end user encounters an issue with your service that requires you to do some deep analysis of the system to pinpoint the issue. You “could” do this all in a production environment, but do you really want to potentially impact other users whilst you are analyzing and testing possible fixes? The answer should probably be “no”. By using a test environment, you are free to poke and prod it whilst replicating the issue without fear of breaking the production system. Another benefit here is that the test system would be a reference configuration, so you have a way of comparing the two systems to ensure everything is the same.
6. Penetration Testing - This is something that I think a lot of us are guilty of and it is a big one...security! Do you really want to just bring a production network online without caring if it is in fact secure? With scheduled, regular penetration testing (whether you employ an external company or use tools such as Nessus to scan your environment), you can ensure that you are well aware of any security vulnerabilities that you can address before releasing the system into the wild. Adding to this, remember where I said “regular”? Penetration testing should be an ongoing thing - not just a one-off. So if you were to perform the testing in production and IF there was an issue, it “could” potentially bring down your production system. By using a test system, you remain safe in the knowledge that your production system will remain up, even if the tester manages to break your test system.

Why you need a load balancer in your test environment

I’ve listed above 6 reasons for needing a test system (and I know full well that there are more reasons than that). However, I’ve not specifically given a reason as to why you need a load balancer in the test systems.

For me, the reason here is quite simple: if you implement a test system that is in some way representative of the production system (i.e. you are replicating production traffic flows, or performing OPT), then you really should have a load balancer implemented there too. Depending on the test system, it may not need to be exactly the same model as the production system (e.g. if it is just for functionality then you probably don’t need another pair of 100G appliances in your test environment), but if it is for performance testing, then you probably want the same model (or at least review your testing strategy so that you can create a representative throughput at a lower rate). If, on the other hand, you are simply doing development work or small-scale testing then you may not need an additional load balancer in there (although there is always a way to fit one in).

Now, I fully understand that not everyone is able to run two (or more) separate environments (after all, each separate environment costs time, effort and money to maintain). BUT if you can, then why wouldn’t you?

Ultimately, a load balancer is part of your infrastructure and is subject to any changes that you want to make to your system (software updates, configuration changes, performance testing etc). So, why would you not add another load balancer into the mix to ensure that your test environment is identical to your production environment without the added worry of impacting production breaking your production?