How to tackle bugs and vulnerabilities – a solutions architect’s opinion

Be it Twitter, Google, Apple, Microsoft, Citrix, or Zoom, bugs and vulnerabilities are common in tech. Similarly, a critical vulnerability in versions 1.8 and above of HAProxy’s HTTP/2 HPACK decoder (CVE-2020-11100) was discovered and fixed earlier this year. Security researcher Felix Wilhelm found this vulnerability that could have been exploited to cause an out-of-bounds memory write potentially leading to corruption of data, a crash, or code execution.

As our products use HAProxy, we were notified in advance, although the majority of our customers and partners were not affected. Nevertheless, the early notification enabled us to test the patch with our appliance and hotfix even before the issue was made public. We spoke to Aaron West, the head of Solutions at Loadbalancer.org – and our resident technical guru – about our approach to dealing with such issues, and more.

Himakshi (H): Hi Aaron (A), in your experience, are these types of vulnerabilities unavoidable?

A: Yes, unavoidable but easily manageable. I think such vulnerabilities are an expected part of software development. I see this as an endless task – crushing bugs and taking down attack vectors – which is just part of the modern software development. There will always be the next big software vulnerability. All you can do is, deal with it quickly and efficiently, just like HAProxy’s lead developer, Willy Tarreau and his team always does. Unlike say Citrix, who took a month to fix their vulnerability that occurred in December last year.

Let’s refer to the HAProxy blog as a source and look at the timeline to understand how they managed the vulnerability pretty efficiently:

• 24 March 18:37 UTC: Felix Wilhelm contacts Willy Tarreau, HAProxy’s lead developer, privately with a detailed report and a reproducer
• 24 March 18:50 UTC: Tarreau confirms receipt and starts looking at the issue
• 25 March 8:31 UTC: Tarreau informs distros about an upcoming H2/HPACK vulnerability
• 27 March 16:38 UTC: Tarreau proposes a candidate fix to Wilhelm
• 27 March 17:10 UTC: Tarreau notifies distros about possible workarounds
• 29 March 08:32 UTC: Tarreau sends the candidate patches to distros and proposes a coordinated release date for April 2nd 13:00 UTC
• 30 March 11:38 UTC: all distros have already agreed on the CRD
• 30 March 16:46 UTC: Wilhelm confirms the fix
• 31 March 05:59 UTC: CVE ID assigned by the Debian team
• 2 April 13:00 UTC: new packages published, making the vulnerability public

H: How did we deal with this vulnerability? Given that most of our customers were unaffected, how do we ensure minimal impact when these issues occur?

A: You may call us lucky, but we normally do not fully embrace the latest versions or add the latest features before they are fully mature. We try to be extra careful and use the validated version rather than using technologies with bleeding edge features as they are very much subjected to vulnerabilities while they develop. But we’re well ahead of the old stable versions. Instead, we thoroughly validate, patch and test the version we consider to have the right balance of stability, performance and functionality for our customers. Actually, not only with HAProxy, we do that with a lot of other open source software.

The recent HAProxy vulnerability didn’t affect the majority of our customers because we don’t take advantage of the latest HTTP/2 feature set in HAProxy or HAProxy 1.9+ which uses a newer HTX HTTP parser by default. Of course, we don’t stop our customers from using HTTP/2 pass through in TCP mode. So they can do HTTP/2 on the backend, just without termination or Layer 7 processing on the load balancer. We can support manual configurations in rare cases where there are requirements to terminate HTTP/2 at the load balancer.

When we do provide manual configuration, we make it a point to keep note of it in our helpdesk and associated information systems so we can tackle these cases individually when they might be affected by a specific security concern that is not fully shared by the standard appliance feature set. That way, we are able to work specifically with those customers to help them remain secure even when their configuration goes out of track. In fact, that is how we managed the customers that were affected by HAProxy HTTP/2 HPACK decoder vulnerability.

H: How did we plan to patch and test this vulnerability in our upcoming release(s)?

A: As it was not critical to our customer base, we immediately produced a hotfix for anyone with a manual configuration or particular concern – but delayed the actual production fix for two weeks for the next full release of version 8.4.3. Had it been an urgent security patch for our customers, we would have gone ahead and released 8.4.3 immediately.

H: Talking about HAProxy, can you briefly talk about our association with the open-source community and how that has helped us as a business?

A: I think ours is a pretty standard open source story. If you go back to the roots of this company, it started with Malcolm (the co-founder and CEO) having an interest in load balancing, which triggered him to become a Subject Matter Expert and grow this business to what it is today. You may also want to check Archive.org, the original community site that was the birthplace of Loadbalancer.org, which contains a wealth of information gathered by Malcolm that eventually grew into a proper commercial offering.

I feel the best thing about open source is that we can all communicate, share ideas and information, learn from each other and grow as a community. As Malcolm puts it, while we have come a long way and grown into what we are as a company today, but we still maintain our links to the open source community – helping where we can by sponsoring development work, offering advice on how open source is used in big businesses and/ or just by aiding on the community help pages or mailing lists. Employees from different departments of our organization create and comment on blog posts on various open-source subjects. Occasionally, we sponsor features such as email alerts and adaptive feedback agent support in HAProxy. Or, we contribute to the community by open sourcing our own tools such as our SNMP MIB (management information base) for HAProxy and the free open-source windows feedback agent for Ultramonkey and HAProxy. Recently (before the COVID-19 hit the entire world), we’ve been coming out more and even visiting events like the HAProxy conference that was held towards the end of last year in Amsterdam.

H: Does working closely with HAProxy make our products more secure?

A: Yes! If we didn’t have such a good relationship with them, we would struggle to patch zero-day vulnerabilities so quickly. So there’s definitely the benefit of finding out early. If we were told when everyone else is told, we would then need to build the latest version with the patch and test it after the vulnerability was publicly known. It would have taken time, at least a day but likely more, depending on how much testing is required. The headstart we get is invaluable (in this case, we knew by 25th March!) as it allows us the time to turn around a fix before it becomes public.

H: Have we started using the latest version of HAProxy? If not, why?

A: Now, you have various versions of HAProxy which are called different branches and on top of that, you have the current LTS (long term support) and development versions. In version 8, we still use the very reliable 1.8 which is an LTS release.

Some people might say, “What about version 2”? Well, version 2+ is where all the cool features are being added and I’m sure, we will continue to move with the times but 1.8 is considerably older, better tested and generally has proven rock solid. So why expose ourselves to more bugs and vulnerabilities than needed?!

H: In general, how do we respond to and handle bug fixes in our products?

A: Bugs are typically discovered either in our vigorous testing or reported by customers. Our support team is often the first to know about the later and they always treat it on a high priority to get it reproduced and reported upstream to development.

Normally, a bug report is raised and evidence is provided for the run-of-the-mill stuff such as minor corrections to the Web UI or improvements. Bugs are fixed and the fixes become part of the next feature release. Should a more serious bug or vulnerability take place, we may need to escalate to releasing a faster security-focused release or maybe, provide a workaround as well/ instead.

Usually, all the technical team managers (Mark from development, Dave from support and myself) come together to decide on such things and Malcolm definitely weighs in too. Together, we consider the impact on the products, customers and if it affects any of our key partners. Here, I'm thinking about downtime that might affect customers or specific features that customers commonly use that might be impacted. We often inherit problems from the installed libraries on the system, such as OpenSSL. Now, OpenSSL is a vital component for a product like ours but also a frequent target. Don't get me wrong, I’m not taking a swipe at OpenSSL. It provides SSL capabilities to so many systems that it will always need to be updated often to stay one step ahead of the latest CVEs (Common Vulnerabilities and Exposures). We also have specific tooling to reduce self-caused security issues in our code that we may inadvertently create. Microfocus Fortify has proven to be a useful addition providing static code analysis to mitigate such problems before the product even gets to the user testing stage.

H: Many say open source is way more secure than proprietary or commercial software – do you agree?

A: I feel doing things in an open source fashion really does bolster the security aspect because you've got so many eyes around the world looking at the source code. Since it is peer reviewed, vulnerabilities are dealt with a lot quicker. Therefore, in theory, there shouldn't (while there can) be as many security vulnerabilities compared to commercial software. And if there are, they get squashed, spotted and pushed out into the open quicker. Proprietary people may argue that they are more secure because their code is hidden and no one can read them. But that’s exactly the reason why bugs and vulnerabilities cannot be easily spotted unless some hacker cracks the code.

I think in general, it's fair to say that open source does lead to a more secure and better peer reviewed product. That is not to say there is not some very well-written and secure proprietary software out there too. But I personally prefer the open source approach. I think that peer review is a very, very powerful tool.

H: When is it important to keep critical bugs a secret?

A: Until a workaround or fix can be published. There is no point exposing a bug or a vulnerability if no fix is available. However, you do need to consider if it’s already out there and if so, this may not be appropriate. It’s mostly a balancing act. Unfortunately, sometimes companies do get it wrong which is somewhat understandable.

H: How do customers know if a proprietary load balancing vendor is transparent about its bugs and vulnerabilities?

A: To me, the first warning sign would be any company saying that they don't get any (or even very few). Because almost all software products contain bugs and vulnerabilities; the important thing is how vendors handle them. What I would like to say is, always go for a company that openly talks about bugs and problems and how to fix them, and I think you’ll be in safe hands as a customer.

H: Once an issue is identified, how can customers  minimize the impact?

A: If you’re one of our customers, should you be affected by any bug or vulnerability, we would contact you proactively. But if you feel you may be affected and want to reach out to support for assistance, we are available 24/7 to help discuss any concerns you may have. For everyone else, the update facility in the product should be the best way to address this. Again, as I said, our excellent support team is always on-hand for any kind of assistance.

In general, as a customer, the best thing to do is to follow the vendor’s advice and make sure that you stay on your vendor’s security mailing list so that you're made aware whenever there is a problem. For example, because we’re on Willy’s special security advisory list, we are immediately informed if there are any bugs or vulnerabilities within HAProxy. In fact, we have the same sort of security mailing list set up for our customers. Customers who’ve opted in are informed whenever there’s a vulnerability and the actions they should take thereafter to tackle it, either by applying patches or installing security updates.

Overall, I’d like to reiterate that every software project will suffer bugs and vulnerabilities, and it would be foolish to expect otherwise. To me, it’s all about how you deal with them, and projects like HAProxy have an exemplary record!