Loadbalancer.org with Let's Encrypt - quick and dirty!

Let’s Encrypt offers us a free way to get SSL certs with the aim to be less complex than our current solutions, hmm... Well that sounds pretty cool! I had known about Let’s Encrypt for a while now but never had the need to take the plunge until this weekend.

Hopefully before reading this we all know what Let’s Encrypt is but just to recap from Wikipedia:

Let's Encrypt is a certificate authority that launched on April 12, 2016[1][2] that provides free X.509 certificates for Transport Layer Security (TLS) encryption via an automated process designed to eliminate the current complex process of manual creation, validation,signing, installation, and renewal of certificates for secure websites.

So Let’s Encrypt offers us a free way to get SSL certs with the aim of being less complex than our current solutions, hmm... Well that sounds pretty cool! I had known about Let’s Encrypt for a while now but never had the need to take the plunge until this weekend. My SSL cert was due for renewal and although it was for a site which is nothing more than my personal sandbox it was useful for me to have an SSL cert so I went checking for offers… Problem is that I picked up a few extra domains this year and I was thinking that it would be really nice to get a SAN cert that can cover all of these so I have one certificate for multiple jobs… Anyway the best deal I could find was about $25 for what they called a multi-domain certificate while some providers where going up towards $300 for a cert! Suddenly it struck me that I had been meaning to have a play with Let’s Encrypt and that this was the ideal time!

Now for my home hosted sandbox I have all my servers running in my shed under KVM on an old Dell R420. I then have an externally hosted VPS running HAproxy with a VPN into my LAN, this is partially because I only have one public IP address and it’s semi dynamic, so I like having something in front should anything break. HAproxy is what handles my SSL offloading so I needed to run Let’s Encrypt from my VPS ideally. This was easily accomplished using Certbot which because it’s CentOS 7 was quick to install and fairly impressive. Ever the perfectionist I decided I needed to automate the renewal process and I wanted it to detect and renew in the event of a certificate being revoked (which it didn’t seem to do unless I missed it), anyway this led me to writing a little script that would check the expiry date and revocation status and then renew as appropriate copying the new certificate in place for HAproxy before reloading the process.

While doing this I couldn’t help but notice the similarities between my scenario and the few customers we’ve already had ask about Let’s Encrypt support for the load balancer. They too need a way to issue a Let’s Encrypt cert from their load balancer and use it with a service other than a web server (Stunnel/Pound). Again they likely want it to renew automatically (or at least alert them) and handle the renewing of a revoked certificate. This got me thinking I could probably re-use some of my work to solve the problem for Loadbalancer.org customers too.

Okay there would likely be some issues… First I realised that for Certbot to work on the load balancer I’d need to use certbot-auto from here:https://certbot.eff.org/#centos6-haproxy

So I installed it which led to an awful lot of packages being installed, I was running v8.2 at the time and concerned seeing so much get pulled in I decided to try updating the appliance to see if I had any problems… Dependency Hell!!! Basically the appliance was in a right state, YUM hit an error and as such online updates were broken. I was in a complete pickle… I looked into it a bit for a resolution but on realising it was upgrading Python which the Dev team wouldn’t like, I decided to walk away from it. I then tried another couple of clients before finally settling with Acme.sh which is simple but pretty cool:https://github.com/Neilpang/acme.sh

Finally I set about writing a script and some simple instructions to use it on our appliance, the fruits of this labour can be found below although it’s very much “use at your own risk” as this is the produce of a weekends experimentation rather than a fully tested/proven solution. However it appears to work and I’m willing to put some further effort in at a later date to add fixes and improvements.

Feedback needed, please comment below.

Initial Setup:

The following will download and install Acme.sh, as part of the installation it also sets up a cron entry which you will want to disable as it wont know how to prepare the certificate for use on the load balancer or put it in place.

curl https://get.acme.sh | sh

Now edit the crontab and delete the cron entry added by Acme.sh.

crontab -e

I found that although the Acme.sh installer was supposed to have aliased the command it didn’t instantly work in my session so ran it again for certainty.

alias acme.sh=~/.acme.sh/acme.sh

The below generates your configuration and certificate, you can specify multiple domains by adding more options to the command “-d ”.

acme.sh --issue --tls -d example1.com -d example2.com 

Then we need to prepare the cert for use on the load balancer, we like certificates containing the full chain plus the key so the following common achieves that:

cat ~/.acme.sh/example.com/fullchain.cer  ~/.acme.sh/example.com/.key  | tee ~/mycert.pem

Finally download the resulting certificate from the load balancer using SCP(WinSCP for Windows) so you can upload it via the WebUI as normal (take note of the name you give it for later).

Using the script:

Now that you've uploaded the certificate to the load balancer WebUI you can use it with your SSL termination, next you will want to setup the script to monitor for renewal. You can run this script to check for a near expiration date or a revoked certificate and renew accordingly. It's best run through Cron in report mode so instead of interrupting services it'll email you allowing you to plan in maintenance. I also suggest running it a couple of times on the CLI by hand at first to make sure you have it working correctly.

The script supports the following options:

-v = IP of service

-p = Port of service

-a = Path to acme.sh certificate files

-c = Path to loadbalancer certificate

-d = Domain

-s = Service stunnel/pound

-f = From Address

-t = To Address

-r = SMTP Relay

-x = SMTP Port

Minimum requirement is to provide -v, -p, -a, -c, -d, -s options to run in the default take action mode.

Running in the default take action mode

./letsencrypt.sh -v *IP-ADDRESS* -p *Port* -a *Path to acme.sh cert* -c *Path to LB cert* -d *Domain* -s *Service Stunnel/Pound* 

./letsencrypt.sh -v 10.0.0.3 -p 443 -a ~/.acme.sh/example.com -c /etc/loadbalancer.org/certs/example/example.pem -d example.com -s stunnel

Running with the optional email mode instead of take action

./letsencrypt.sh -v *IP-ADDRESS* -p *Port* -a *Path to acme.sh cert* -c *Path to LB cert* -d *Domain* -s *Service Stunnel/Pound -f *From Email Address* -t *To Email Address* -r *SMTP Relay* -x *SMTP Port* 
./letsencrypt.sh -v 10.0.0.3 -p 443 -a ~/.acme.sh/example.com -c /etc/loadbalancer.org/certs/example/example.pem -d example.com -s stunnel -f lb@example.com -t noc@example.com -r smtp.example.com -x 25

As your certificate is new and hopefully not revoked you can expect the following output:

[root@lbmaster ~]# ./letsencrypt.sh -v 10.0.0.3 -p 443 -a ~/.acme.sh/example.com -c /etc/loadbalancer.org/certs/example/example.pem -d example.com -s stunnel -f lb@example.com -t noc@example.com -r smtp.example.com -x 25
Process stunnel is running, continuing.
Certificate has 87 days remaining, not renewing.
Certificate is not revoked, exiting.

Cron Job:

Execute daily with cron, edit crontab as root with "crontab -e" and replace the acme.sh entry with:

0 0 * * * /root/letsencrypt.sh -v 10.0.0.3 -p 443 -a ~/.acme.sh/example.com -c /etc/loadbalancer.org/certs/example/example.pem -d example.com -s stunnel -f lb@example.com -t noc@example.com -r smtp.example.com -x 25 > /dev/null

The Script itself: Download Here

#!/bin/bash
# Written and maintained by : Aaron West
# Version 1.0
#
# Script to integrate Let's Encrypt, acme.sh shell script and Loadbalancer.org appliance 8.2 and above.
#

# Set minimum life left before renewing the cert.
RENEW_TME=7

# Start Script
RPT=0
PATH=$PATH:~/.acme.sh/

while getopts 'v:p:a:c:d:s:f:t:r:x:' flag; do
  case "${flag}" in
    v) VIP="${OPTARG}" ;;
    p) PRT="${OPTARG}" ;;
    a) PAC="${OPTARG}" ;;
    c) PLC="${OPTARG}" ;;
    d) DOM="${OPTARG}" ;;
    s) SER="${OPTARG}" ;;
    f) FRM="${OPTARG}" ;;
    t) TOA="${OPTARG}" ;;
    r) REL="${OPTARG}" ;;
    x) RPT="${OPTARG}" ;;
    *) error "Unexpected option ${flag}" ;;
  esac
done

#Check service is running
    if pidof $SER >/dev/null; then
        echo "Process $SER is running, continuing." 
    else
        echo "Process $SER is not running, exiting."
        exit
    fi

#Email Function
function SMTP {
        echo "$MESSAGE" | mailx -v -r $FRM -s "Your Let's Encrypt Certificate Requires Action : $ACTION" -S smtp=$REL:$RPT $TOA          
           }

EXPIRE_DATE=$(date -d "`echo | openssl s_client -connect $VIP:$PRT 2>/dev/null | openssl x509 -noout -dates | grep "notAfter=" |cut -c10-`" +%s)
NOW=$(date -d "now" +%s)
DAYS_EXP=$(echo \( $EXPIRE_DATE - $NOW \) / 86400 |bc)
OCSP_URL=$(openssl x509 -noout -ocsp_uri -in   $PAC/$DOM.cer)
REVOKED=$(openssl ocsp -issuer $PAC/ca.cer -cert $PAC/$DOM.cer -text -url $OCSP_URL -header "HOST" "$(echo "$OCSP_URL" | awk -F/ '{print $3}')" 2>/dev/null |grep -c "Revocation Time")
    if [ $DAYS_EXP -lt $RENEW_TME ]; then
        echo "Certificate expires in $DAYS_EXP days which is less than the renew time of $RENEW_TME days so renewing"
        if [ $RPT -gt 0 ]; then
            ACTION="Certificate needs renewing, please renew."
            MESSAGE="Your certificate for $DOM needs to be renewed, please renew from the loadbalancer.org appliance."
            SMTP
        else
            service $SER stop
            if acme.sh --renew -d $DOM; then
                cat $PAC/fullchain.cer  $PAC/$DOM.key  | tee $PLC
                service $SER start
            else
                echo "Renewal failed"
                service $SER start
            fi
        fi
    else
        echo "Certificate has $DAYS_EXP days remaining, not renewing."
    fi
    if [ $REVOKED != 0 ]; then
        echo "Certificate is revoked, please force regeneration."
        if [ $RPT -gt 0 ]; then
            ACTION="Certificate is revoked, please renew."
            MESSAGE="Your certificate for $DOM is revoked, please renew from the loadbalancer.org appliance."
            SMTP
        else
            service $SER stop
            if acme.sh --renew -d $DOM --force; then
                cat $PAC/fullchain.cer  $PAC/$DOM.key  | tee $PLC
                service $SER start
            else
                echo "Renewal failed"
                service $SER start
            fi
        fi
    else
        echo "Certificate is not revoked, exiting."
    fi
exit