The ModSecurity based Web Application Firewall (WAF) we introduced a few years ago, has proven fast and reliable. But it has teeth. One customer recently joked that 'Taming a dragon would be easier than configuring a WAF!'
We've developed some tools to make the whole process easier. Let's look at the best way to use the WAF with as little pain as possible...
UPDATE: Good news, v8.3.3 now has the tools described in this blog — as easy to use one-click options.
What is a WAF?
Your application should be secure already, but having a WAF doubles this up by:
- Blocking hackers from exploiting the OWASP top 10, the ten most critical security risks to web applications.
- Rapidly deploying new rules to block new threats (virtual patching).
- Offering extensive logging & entrapment capabilities.
For more information, listen below to an audio introduction to WAFs - from our very own Andrew Howe:
Why is configuring so painful?
All WAFs are implemented as a complicated set of REGEX rules. These are designed to be as efficient as possible (and therefore impossible for a normal human to read or understand). The rules should be secure enough to do the job - but not so secure that they generate too many false positives.
Understandably, the ModSecurity team's priorities are security first, performance second, and ease of use... maybe next time.
A new WAF won't work like magic. WAFs are useful tools, but they are also complex. The whole point is to block suspicious activity, so if the default rules don't like your application you could get false positives. With many applications you'll need to exclude WAF rules so that they don't lock your users out!
Some WAFs will offer training modes, auto-learning and similar features. Even using these helpful utilities, you'll still need to understand what's going on in order to secure your application safely.
Managing a WAF requires you to become knowledgeable about security and vulnerabilities. Before excluding rules in the WAF, understand what you're excluding and what the implications might be.
Training your dragon
OAT Phase
First you need to test or 'train' your WAF to see how it will behave in front of your application. It's important to do this during an OAT (Operational Acceptance Testing) phase, so that you can identify and resolve problems while you have clean traffic.
If you try to 'train' with external users, you may get real attack behavior recorded in the logs - and you could exclude a rule that inadvertently allows malicious traffic!
You can work around this by searching the logs for just the IPs of your actual trusted users. However, this still creates scope for error. It's much better to go through a real OAT phase followed by a UAT (User Acceptance Testing) phase before your application and WAF deployment are deemed ready for production.
This approach gets the app working right with the WAF, using 'clean' test traffic (a closed group of users) to build up your initial whitelist. You should work in 'Detection Only Mode' - the default when deploying a new WAF - which means that you don't enable "Rule Engine Traffic Blocking" in the WUI.
UAT Phase
Now you can move into the UAT phase with a wider user base. Since most common user issues were in theory fixed in the OAT phase, you can now enable "Rule Engine Traffic Blocking". Issues can be fixed on a case-by-case basis as users report real problems. Ignore log entries that don't have an accompanying user report as these are probably just the WAF blocking undesirable behaviour.
Once this phase is complete you are ready for production, with your application securely protected by the WAF.
Production
In most cases it's best to test your WAF with known clean traffic initially, so you don't get burned by whitelisting a false positive. However, this approach isn't always possible - and there are some technicians who prefer fixing problems on the fly.
When it comes to identifying which rules to remove, or whitelisting in general, first consider fixing the application! If you have a direct line to the developers inform them of the problem and ask if they can make the code more secure or at least compatible with the WAF. If this is not an option (which it often isn't), identify the behaviour you want to allow and decide how you want to whitelist the traffic.
Loving your logs
Logs leave a lot to be desired. Trying to read them can feel like looking at a wall of incoherent text. ModSecurity logs are long and detailed, full of rule ID's and ARGS which won't mean much to you at first.
There are some simple things you can do to circumvent this problem - for example, highlighting sections of the log to make it clearer where each entry starts and finishes when using the WUI. Simple but effective!
When you need to deep-dive into the logs it may be best to read them in your favourite text editor, so that you can format them for your preferences quickly. You can retrieve the full logs via SCP (WinSCP for Windows).
Another tip is to zero the logs before doing initial testing. This restricts the ammount of data in the log file. In v8.3.3 you have a simple one-click option to do this, but for previous versions you can use the command line:
You could also use an external Syslog server to store the logs remotely - useful if you want to archive all of the WAF logs for auditing.
Whitelisting and excluding rules
Let’s look at some log entries. Here's the first one to look out for:
[Sun Feb 18 21:29:47.151788 2018] [:error] [pid 32289:tid 140260134610688] [client 172.16.202.152:60354] [client 172.16.202.152] ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/opt/httpd-waf/modsecurity.d/activated_rules/modsecurity_crs_60_correlation.conf"] [line "37"] [id "981204"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 102, SQLi=29, XSS=): 981243-Detects classic SQL injection probings 2/2"] [hostname "support.loadbalancer.org"] [uri "/staff/index.php"] [unique_id "WonwQn8AAAEAAH4hPNIAAABD"], referer: https://support.loadbalancer.org/staff/index.php?/Knowledgebase/Article/Edit/35
This tells you that the inbound anomaly score has been matched, and the total scores it received. Don't exclude it!
WARNING!
Never remove or whitelist this rule. In fact rules 981200-981205 are all best ignored! The example above is used to correlate the results and report the outcome back to you. The example below adds up the scores and blocks traffic when needed. So never whitelist or remove rule "ID 981204" or "ID 981176".
[Sun Feb 18 21:29:47.151347 2018] [:error] [pid 32289:tid 140260134610688] [client 172.16.202.152:60354] [client 172.16.202.152] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(.*)" at TX:960024-OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION-ARGS:articlecontents. [file "/opt/httpd-waf/modsecurity.d/activated_rules/modsecurity_crs_49_inbound_blocking.conf"] [line "26"] [id "981176"] [msg "Inbound Anomaly Score Exceeded (Total Score: 102, SQLi=29, XSS=): Last Matched Message: 981243-Detects classic SQL injection probings 2/2"] [data "Last Matched Data: \\x22>\\x0d\\x0a<"] [hostname "support.loadbalancer.org"] [uri "/staff/index.php"] [unique_id "WonwQn8AAAEAAH4hPNIAAABD"], referer: https://support.loadbalancer.org/staff/index.php?/Knowledgebase/Article/Edit/35
This tells you that the inbound anomaly score has reached the threshold to block the user. Don't exclude it!
The MSG you're looking for
Inside the next impossible-to-read mess, you should see the actual attack that was matched:
[id "981243"] [msg "Detects classic SQL injection probings 2/2"]
Let's look at a real example. The MSG above was found in the following section. It shows the regex matched (Pattern match), the argument it was detected in (ARGS), rule ID, hostname, URI and referrer. These will be the key pieces of information you use to whitelist. You should also pay attention to the rules config filename and the messages provided, which can help you better understand the problem detected:
[Sun Feb 18 21:29:38.929282 2018] [:error] [pid 32289:tid 140260134610688] [client 172.16.202.152:60354] [client 172.16.202.152] ModSecurity: Warning. Pattern match "(?i:(?:[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98]\\\\s*?\\\\*.+(?:x?or|div|like|between|and|id)\\\\W*?[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98]\\\\d)|(?:\\\\^[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98])|(?:^[\\\\w\\\\s\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98-]+(?<=and\\\\s)(?<=or|xor ..." at ARGS:articlecontents_htmlcontents. [file "/opt/httpd-waf/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "245"] [id "981243"] [msg "Detects classic SQL injection probings 2/2"] [data "Matched Data: \\x22>\\x0d\\x0a<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Semper feugiat nibh sed pulvinar proin gravida. Aliquam ut porttitor leo a diam sollicitudin tempor id. Magnis dis parturient montes nascetur ridiculus mus mauris vitae ultricies. Viverra nam libero justo laoreet sit amet cursus sit amet. Convallis aenean et tortor at risus viverra adipiscing. Sit amet massa vitae tortor. Non tellus orci ac auctor augue..."] [se [hostname "support.loadbalancer.org"] [uri "/staff/index.php"] [unique_id "WonwQn8AAAEAAH4hPNIAAABD"], referer: https://support.loadbalancer.org/staff/index.php?/Knowledgebase/Article/Edit/35
Let's break it down
Pattern match:
Pattern match "(?i:(?:[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98]\\\\s*?\\\\*.+(?:x?or|div|like|between|and|id)\\\\W*?[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98]\\\\d)|(?:\\\\^[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98])|(?:^[\\\\w\\\\s\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98-]+(?<=and\\\\s)(?<=or|xor ..."
Argument:
ARGS:articlecontents_htmlcontents
Rule ID:
id "981243"
Config Filename:
"modsecurity_crs_41_sql_injection_attacks.conf".
Message:
msg "Detects classic SQL injection probings 2/2"
Hostname:
hostname "support.loadbalancer.org"
URI:
uri "/staff/index.php"
Referrer:
referer: https://support.loadbalancer.org/staff/index.php?/Knowledgebase/Article/Edit/35
Stop blocking my page you stupid WAF!
The WAF is doing its job. It thinks it's found a malicious SQL injection attack - but it's just your web app posting SQL commands to itself. In a sense that means it is vulnerable, but you don't need to take action. A hacker would have to log in before they could do this attack - so it's fairly secure.
Now you have the details you need...
Let's start whitelisting
Now you have the pattern match and argument that triggered the alert, the rule ID as well as various other useful pieces of information to work with. Using this info you can decide how best to whitelist this match.
You can simply remove the rule entirely from your WAF:
SecRuleRemoveById 981243
Or better, update the rule so it doesn’t apply to the specific argument:
SecRuleUpdateTargetById 981243 !ARGS:articlecontents_htmlcontents
Or best, remove the rule for a specific location of your site:
<LocationMatch "^/staff/index.php$">
SecRuleRemoveById 981243
</LocationMatch>
Rules can be as clever or complex as you like. The example below removes the rule matching the referer header for the specific argument. However be aware that making things complex is probably inviting trouble later!
SecRule REQUEST_HEADERS:Referer "@streq https://support.loadbalancer.org/staff/index.php?/Knowledgebase/Article/Edit/35" \
"phase:1,nolog,pass,id:10000,ctl:ruleRemoveById=981243;ARGS:articlecontents_htmlcontents"
How do we bring all this together?
You have loads of logs to examine, and there are a few ways to make life easier. You can use the command line to parse the logs directly:
grep ModSecurity /var/log/httpd/error_Test_WAF.log | grep -v -e 'id "981204"' -e 'id "981176"' |sed -e 's#^.*\[id "\([0-9]*\).*hostname "\([a-z0-9\-\_\.]*\)"\].*uri "#\1 \2 #' | cut -d\" -f1 | sort -n | uniq -c | sort -n
[root@lbmaster ~]# grep ModSecurity /var/log/httpd/error_Test_WAF.log | grep -v -e 'id "981204"' -e 'id "981176"' | sed -e 's#^.*\[id "\([0-9]*\).*hostname "\([a-z0-9\-\_\.]*\)"\].*uri "#\1 \2 #' | cut -d\" -f1 | sort -n | uniq -c | sort -n
2 950901 support.loadbalancer.org /staff/index.php
2 960024 support.loadbalancer.org /staff/index.php
2 981173 support.loadbalancer.org /staff/index.php
2 981231 support.loadbalancer.org /staff/index.php
2 981240 support.loadbalancer.org /staff/index.php
2 981244 support.loadbalancer.org /staff/index.php
2 981245 support.loadbalancer.org /staff/index.php
2 981246 support.loadbalancer.org /staff/index.php
2 981248 support.loadbalancer.org /staff/index.php
2 981257 support.loadbalancer.org /staff/index.php
This gives you the unique occurences of the rules triggered plus the hostname and URI requested - exactly what you need for some quick location exclusions:
<LocationMatch "^/staff/index.php$">
SecRuleRemoveById 950901
SecRuleRemoveById 960024
SecRuleRemoveById 981173
SecRuleRemoveById 981231
SecRuleRemoveById 981240
SecRuleRemoveById 981244
SecRuleRemoveById 981245
SecRuleRemoveById 981246
SecRuleRemoveById 981248
SecRuleRemoveById 981257
</LocationMatch>
You could also use an alternative improved version of this command to pick out the arguments:
grep ModSecurity /var/log/httpd/error_Test_WAF.log | grep -v -e 'id "981204"' -e 'id "981176"' | sed -e 's#^.*ARGS:\(.*\). \[file.*\[id "\([0-9]*\).*hostname "\([a-z0-9\-\_\.]*\)"\].*uri "#\1 \2 \3 #' | cut -d\" -f1 | sort -n | uniq -c | sort -n
[root@lbmaster ~]# grep ModSecurity /var/log/httpd/error_Test_WAF.log | grep -v -e 'id "981204"' -e 'id "981176"' | sed -e 's#^.*ARGS:\(.*\). \[file.*\[id "\([0-9]*\).*hostname "\([a-z0-9\-\_\.]*\)"\].*uri "#\1 \2 \3 #' | cut -d\" -f1 | sort -n | uniq -c | sort -n
1 articlecontents 950901 support.loadbalancer.org /staff/index.php
1 articlecontents 960024 support.loadbalancer.org /staff/index.php
1 articlecontents 973300 support.loadbalancer.org /staff/index.php
1 articlecontents 973333 support.loadbalancer.org /staff/index.php
1 articlecontents 981173 support.loadbalancer.org /staff/index.php
1 articlecontents 981231 support.loadbalancer.org /staff/index.php
1 articlecontents 981240 support.loadbalancer.org /staff/index.php
1 articlecontents 981243 support.loadbalancer.org /staff/index.php
1 articlecontents 981244 support.loadbalancer.org /staff/index.php
1 articlecontents 981245 support.loadbalancer.org /staff/index.php
1 articlecontents 981246 support.loadbalancer.org /staff/index.php
1 articlecontents 981248 support.loadbalancer.org /staff/index.php
1 articlecontents 981257 support.loadbalancer.org /staff/index.php
1 articlecontents_htmlcontents 950901 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 960024 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 973300 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 973333 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 981173 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 981231 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 981240 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 981243 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 981244 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 981245 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 981246 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 981248 support.loadbalancer.org staff/index.php
1 articlecontents_htmlcontents 981257 support.loadbalancer.org staff/index.php
Ready to fly your dragon?
With this information, you should be well-prepared to tackle a slightly more complex WAF deployment. It's a process of trial and error, which is why it's best done in UAT. That way if you break things, no-one gets hurt.
If you're lucky your application will sail through UAT with nothing blocked by the deafult WAF configuration, and you won't need to use any of the information above — but it's always good to be prepared!
Do you think I've missed some tricks? Feel free to share by leaving a comment below.