How to set up email notifications on your Loadbalancer.org appliance

Email notifications are a great way of alerting you to any changes or issues within your load balancer or high availability cluster. The email notifications can be used alongside or independent of any external monitoring you may have.

You can set up email notifications for the following:

• Heartbeat
• Layer 4 services
• Layer 7 services

Sending email notifications

All notifications will be sent from the load balancer using SMTP. If no smart host has been defined then, by default, any email alerts will be sent to the mail server defined in the destination domain's DNS MX record, directly from the load balancer.

However, configuring a smart host (or mail relay server) will, instead, direct all outgoing mail to the specified relay. This will significantly help to improve the deliverability of the email. A smart host is an email server via which approved devices can send emails and have them forwarded to the email recipients' email servers. Wherever possible, we recommend you define a smart host for use with email notifications.

Each of these services has a variety of configuration options available, which are explored further below.

Defining a smart host (mail relay server)

To define the smart host, from the load balancer's WebUI select "Local Configuration" > "Physical - Advanced Configuration", and then in the "SMTP Relay" section you will be able to specify the "Smart Host".  

A smart host can be a valid IPv4 IP address or a FQDN (Fully Qualified Domain Name).

Configuring Heartbeat notifications

To configure Heartbeat notifications, from the load balancer's WebUI select "Cluster Configuration" > "Heartbeat Configuration", and then in the "Email Alerts" section you can configure the values as required.  

• Email Alert Destination Address = This is the "To" address of the email that the notifications should be sent to (e.g. alert-system@mydomain.com)
• Email Alert Source Address = This is the "From" address of the email, which shows where it has originated from (e.g. lb-alerts@mydomain.com)

Note: Generally, if you do not have a high availability pair, then there is no need to configure email notifications for Heartbeat as it will not be utilized.

Example Heartbeat notifications

Heartbeat maintains the connection between the Active and Passive load balancers, so if a failover or takeover occurs and email notifications are enabled, an email will be generated.

The exact notifications you will receive will vary depending on the state of Heartbeat at the time. However, below are a couple of example notifications that you may see, and a summary of what the notification shows.

In the example above, the resources (floating IPs) are being migrated away from the appliance named "lbmaster", likely as a result of a failover to the secondary/passive appliance.

Configuring Layer 4 notifications

To configure Layer 4 notifications, from the load balancer's WebUI, select "Cluster Configuration" > "Layer 4 - Advanced Configuration". On that page there will be 2 email options, that can be configured with the values as required:

• Email Alert Destination Address = This is the "To" address of the email that the notifications should be sent to (e.g. alert-system@mydomain.com)
• Email Alert Source Address = This is the "From" address of the email, which shows where it has originated from (e.g. lb-alerts@mydomain.com)

After making the changes you will be prompted to "Restart ldirectord". Note this is a restart and not a reload, so will cause a momentary loss of service to any Layer 4 services running on that appliance.

Example Layer 4 notifications

The exact notifications you will receive will vary depending on the state of your virtual services and real servers. However, below are a couple of example notifications that you may see, and a summary of what the notification shows.

An example of a Layer 4 notification can be seen below:

In this case, the notification shows that a real server (RIP Name) listening on IP 192.168.85.0 on port 0 (all ports) has been added into the virtual service "L4-SNAT2" and that the weight of the real server has been set to 100. It also shows the status of the Layer 4 daemon (ldirectord) as "Running".

Another example might look like this:

In this case, the notification shows that a real server listening on IP 192.168.85.0 on port 0 (all ports) has been purged (removed) from the virtual service "L4-SNAT2". It also shows the status of the Layer 4 daemon (ldirectord) as "Reloading".

Configuring Layer 7 notifications

To configure Layer 7 notifications from the load balancer's WebUI, select "Cluster Configuration" > "Layer 7 - Advanced Configuration", on that page there will be several email options that can be configured with the values as required.  

• eMail Alert From = This is the "From" address of the email, which shows where it has originated from (e.g. lb-alerts@mydomain.com)
• eMail Alert To = This is the "To" address of the email that the notifications should be sent to (e.g. alert-system@mydomain.com)
• eMail Server Adress = This is the FQDN or IPv41P of the mail server used to send the email (e.g. mail.mydomain.com)
• eMail Server Port = This is the port number used to connect to the mail server (the default is 25)

Note: Layer 7 notifications use HAProxy to send the notifications and so any smart host already configured in the "Physical - Advanced Configuration" section, would not be used here.

Example Layer 7 notifications

The exact notifications you will receive will vary depending on the state of your virtual services and real servers. However, below are a couple of example notifications that you may see, and a summary of what the notification shows.

An example of a Layer 7 notification can be seen below:

In this case, the notification shows that a real server (R7Demo1) is DOWN in the virtual service "L7Demo1". However, there is still 1 active server (real server) and 1 backup (fallback) server remaining.

Another example could be:

In this case, the notification shows that a real server (R7Demo1) on the virtual service "L7Demo1" has been marked as DOWN, due to Layer 4 connection problems (Connection Refused), and that there are no active servers (real servers) remaining. There is 1 backup (fallback) server remaining though, and so the service is running on the backup server.

Additional considerations

SPF records - If you are not using a designated mail relay, make sure the load balancer IP or external gateway IP is configured in any DNS SPF records for the domain. Additionally, make sure the "From" address contains a domain you own/control.

"To" address - If you need to send notifications to multiple people, try setting up a distribution list or similar in your mail server, and send the notifications to that email address.

Are you load balancing your email on these load balancers? If so, ensure you don't create a point of failure and rely on those notifications to alert you of an issue. Because in a worst-case scenario (if everything does go down) you may not get the notifications.

Testing the notifications once they are first setup is important. We suggest you test them in a scheduled maintenance window, by performing a variety of tasks to ensure you actually get the notifications! For example, set up test notifications for:

• An appliance failover and takeover
• Taking a real server out of the cluster and rebooting it
• Temporarily causing the server to fail health checks