Highly Available NFS based Kerberos KDC aka. Ganesha + GlusterFS + HAProxy.
Open Source Published on •8 mins Last updated
NFS, the Network File System created by Sun Microsystems, has been around for a while. And as with any tool, when used properly, it is very powerful. NFS still gets used heavily in 10's of thousands of systems. However, load balancing NFS is a real pain — especially when it comes to the locked mounts issue. In this blog I'll explain how to create a highly available NFS server for Kerberos KDCs.
Why is NFS so popular?
It's not meant for sharing large files or to provide backup. It's meant for a very simple human need to interact with digital systems efficiently on a day to day basis. It's meant to store small to medium user files centrally instead of resulting in file duplication across a vast infrastructure, potentially leaving sensitive user information spread all over many server disks across a data center.
And so it's no wonder that NFS has seen it's share of development via the Open Source community.
Enter NFS Ganesha.
NFS is not without it's quirks, particularly with locked mounts. So creating a highly available NFS server for our Kerberos KDC's will be our article here to counter that particular issue and provide failover.
We'll cut down to the chase and demonstrate how to configure a highly available NFS Ganesha server with these techs in mind. All this based on our trials with these techs:
- GlusterFS
- NFS Ganesha
- CentOS 7
- HAPROXY
- keepalived
- firewalld
- selinux
And now for the quick steps:
Build at least 2 CentOS 7 servers. Add the hosts to your DNS server for a clean setup. Alternately add them to /etc/hosts (ugly) on both hosts (nfs01 / nfs02).
192.168.0.80 nfs-c01 (nfs01, nfs02) VIP DNS Entry
192.168.0.131 nfs01
192.168.0.119 nfs02
On both nodes, compile and build nfsganesha 2.60+ from latest stable source. (At this time RPM packages did not work). We do this to take advantage of the latest feature sets available and minimize failure. Install the listed packages as well:
wget https://github.com/nfs-ganesha/nfs-ganesha/archive/V2.6-.0.tar.gz
PACKAGES NEEDED :
yum install glusterfs-api-devel.x86_64
yum install xfsprogs-devel.x86_64
yum install xfsprogs.x86_64
xfsdump-3.1.4-1.el7.x86_64
libguestfs-xfs-1.36.3-6.el7_4.3.x86_64
libntirpc-devel-1.5.4-1.el7.x86_64
libntirpc-1.5.4-1.el7.x86_64
COMMANDS
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha;
git checkout V2.6-stable
git submodule update --init --recursive
ccmake /root/ganesha/nfs-ganesha/src/
# Press the c, e, c, g keys to create and generate the config and make files.
make
make install
Add a separate disk to each of the VM's such as /dev/sdb.
Add the gluster filesystem to both nodes and link them together. Steps are done on both nodes:
mkfs.xfs /dev/sdb
mount /dev/sdb /bricks/0
yum install centos-release-gluster
systemctl enable glusterd.service
yum -y install glusterfs glusterfs-fuse glusterfs-server glusterfs-api glusterfs-cli
( node01 ONLY ) gluster volume create gv01 replica 2 nfs01:/bricks/0/gv01 nfs02:/bricks/0/gv01
gluster volume info
gluster volume status
Install and Configure HAPROXY on both nodes. The config:
PACKAGES:
yum install haproxy # ( 1.5.18-6.el7.x86_64 used in this case )
/etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
stats socket /var/run/haproxy.sock mode 0600 level admin
# stats socket /var/lib/haproxy/stats
maxconn 4000
user haproxy
group haproxy
daemon
debug
defaults
mode tcp
log global
option dontlognull
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend nfs-in
bind nfs-c01:2049
mode tcp
option tcplog
default_backend nfs-back
backend nfs-back
balance roundrobin
server nfs01.nix.mine.dom nfs01.nix.mine.dom:2049 check
server nfs02.nix.mine.dom nfs02.nix.mine.dom:2049 check
The following kernel bind parameters are required to be set before configuring keepalived below (More on this available via the RedHat Documentation):
nfs01 / nfs02 # echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf
# echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
# sysctl -p
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
#
Configure keepalived on both nodes:
PACKAGES:
yum install keepalived # ( Used 1.3.5-1.el7.x86_64 in this case )
NFS01:
vrrp_script chk_haproxy {
script "killall -0 haproxy" # check the haproxy process
interval 2 # every 2 seconds
weight 2 # add 2 points if OK
}
vrrp_instance VI_1 {
interface eth0 # interface to monitor
state MASTER # MASTER on haproxy1, BACKUP on haproxy2
virtual_router_id 51
priority 101 # 101 on haproxy1, 100 on haproxy2
virtual_ipaddress {
192.168.0.80 # virtual ip address
}
track_script {
chk_haproxy
}
}
NFS02:
vrrp_script chk_haproxy {
script "killall -0 haproxy" # check the haproxy process
interval 2 # every 2 seconds
weight 2 # add 2 points if OK
}
vrrp_instance VI_1 {
interface eth0 # interface to monitor
state BACKUP # MASTER on haproxy1, BACKUP on haproxy2
virtual_router_id 51
priority 102 # 101 on haproxy1, 100 on haproxy2
virtual_ipaddress {
192.168.0.80 # virtual ip address
}
track_script {
chk_haproxy
}
}
Configure firewalld on both nodes. DO NOT disable firewalld.
# cat public.bash
firewall-cmd --zone=public --permanent --add-port=2049/tcp
firewall-cmd --zone=public --permanent --add-port=111/tcp
firewall-cmd --zone=public --permanent --add-port=111/udp
firewall-cmd --zone=public --permanent --add-port=24007-24008/tcp
firewall-cmd --zone=public --permanent --add-port=49152/tcp
firewall-cmd --zone=public --permanent --add-port=38465-38469/tcp
firewall-cmd --zone=public --permanent --add-port=4501/tcp
firewall-cmd --zone=public --permanent --add-port=4501/udp
firewall-cmd --zone=public --permanent --add-port=20048/udp
firewall-cmd --zone=public --permanent --add-port=20048/tcp
firewall-cmd --reload
# cat dmz.bash
firewall-cmd --zone=dmz --permanent --add-port=2049/tcp
firewall-cmd --zone=dmz --permanent --add-port=111/tcp
firewall-cmd --zone=dmz --permanent --add-port=111/udp
firewall-cmd --zone=dmz --permanent --add-port=24007-24008/tcp
firewall-cmd --zone=dmz --permanent --add-port=49152/tcp
firewall-cmd --zone=dmz --permanent --add-port=38465-38469/tcp
firewall-cmd --zone=dmz --permanent --add-port=4501/tcp
firewall-cmd --zone=dmz --permanent --add-port=4501/udp
firewall-cmd --zone=dmz --permanent --add-port=20048/tcp
firewall-cmd --zone=dmz --permanent --add-port=20048/udp
firewall-cmd --reload
#
# On Both
firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -m pkttype --pkt-type multicast -j ACCEPT
firewall-cmd --reload
HANDY STUFF:
firewall-cmd --zone=dmz --list-all
firewall-cmd --zone=public --list-all
firewall-cmd --set-log-denied=all
firewall-cmd --permanent --add-service=haproxy
firewall-cmd --list-all
Configure selinux. Don't disable it. This actually makes your host safer and is actually easy to work with using just these commands.
Run any of the following command, or a combination of, on deny entries in /var/log/audit/audit.log that may appear as you stop, start or install above services:
METHOD 1:
grep AVC /var/log/audit/audit.log | tail -n1 | audit2allow -M systemd-allow
semodule -i systemd-allow.pp
METHOD 2:
audit2allow -a
audit2allow -a -M ganesha_<NUM>_port
semodule -i ganesha_<NUM>_port.pp
Configure NFS Ganesha on both nodes. The configs differ slightly:
NODE 1:
[root@nfs01 ~]# cat /etc/ganesha/ganesha.conf
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#
###################################################
NFS_Core_Param {
Bind_addr = 192.168.0.131;
NFS_Port = 2049;
MNT_Port = 20048;
NLM_Port = 38468;
Rquota_Port = 4501;
}
%include "/etc/ganesha/export.conf"
[root@nfs01 ~]# cat /etc/ganesha/export.conf
EXPORT{
Export_Id = 1 ; # Export ID unique to each export
Path = "/n"; # Path of the volume to be exported. Eg: "/test_volume"
FSAL {
name = GLUSTER;
hostname = "nfs01.nix.mine.dom"; # IP of one of the nodes in the trusted pool
volume = "gv01"; # Volume name. Eg: "test_volume"
}
Access_type = RW; # Access permissions
Squash = No_root_squash; # To enable/disable root squashing
Disable_ACL = FALSE; # To enable/disable ACL
Pseudo = "/n"; # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
Protocols = "3","4"; # NFS protocols supported
Transports = "UDP","TCP" ; # Transport protocols supported
SecType = "sys"; # Security flavors supported
}
[root@nfs01 ~]#
NODE 2:
[root@nfs02 ~]# cd /etc/ganesha/
[root@nfs02 ganesha]# cat ganesha.conf
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#
###################################################
NFS_Core_Param {
Bind_addr=192.168.0.119;
NFS_Port=2049;
MNT_Port=20048;
NLM_Port=38468;
Rquota_Port=4501;
}
%include "/etc/ganesha/export.conf"
[root@nfs02 ganesha]# cat export.conf
EXPORT{
Export_Id = 1 ; # Export ID unique to each export
Path = "/n"; # Path of the volume to be exported. Eg: "/test_volume"
FSAL {
name = GLUSTER;
hostname = "nfs02.nix.mine.dom"; # IP of one of the nodes in the trusted pool
volume = "gv01"; # Volume name. Eg: "test_volume"
}
Access_type = RW; # Access permissions
Squash = No_root_squash; # To enable/disable root squashing
Disable_ACL = FALSE; # To enable/disable ACL
Pseudo = "/n"; # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
Protocols = "3","4"; # NFS protocols supported
Transports = "UDP","TCP" ; # Transport protocols supported
SecType = "sys"; # Security flavors supported
}
[root@nfs02 ganesha]#
STARTUP:
/usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
Ensure mounts are done and everything is started up on both nodes.
[root@nfs01 ~]# cat /etc/fstab|grep -Ei "brick|gv01"
/dev/sdb /bricks/0 xfs defaults 0 0
nfs01:/gv01 /n glusterfs defaults 0 0
[root@nfs01 ~]#
[root@nfs01 ~]# mount|grep -Ei "brick|gv01"
/dev/sdb on /bricks/0 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
nfs01:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs01 ~]#
[root@nfs01 ~]# ps -ef|grep -Ei "haproxy|keepalived|ganesha"; netstat -pnlt|grep -Ei "haproxy|ganesha|keepalived"
root 1402 1 0 00:59 ? 00:00:00 /usr/sbin/keepalived -D
root 1403 1402 0 00:59 ? 00:00:00 /usr/sbin/keepalived -D
root 1404 1402 0 00:59 ? 00:00:02 /usr/sbin/keepalived -D
root 13087 1 0 01:02 ? 00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy 13088 13087 0 01:02 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
haproxy 13089 13088 0 01:02 ? 00:00:01 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root 13129 1 15 01:02 ? 00:13:11 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
root 19742 15633 0 02:30 pts/2 00:00:00 grep --color=auto -Ei haproxy|keepalived|ganesha
tcp 0 0 192.168.0.80:2049 0.0.0.0:* LISTEN 13089/haproxy
tcp6 0 0 192.168.0.131:20048 :::* LISTEN 13129/ganesha.nfsd
tcp6 0 0 :::564 :::* LISTEN 13129/ganesha.nfsd
tcp6 0 0 192.168.0.131:4501 :::* LISTEN 13129/ganesha.nfsd
tcp6 0 0 192.168.0.131:2049 :::* LISTEN 13129/ganesha.nfsd
tcp6 0 0 192.168.0.131:38468 :::* LISTEN 13129/ganesha.nfsd
[root@nfs01 ~]#
Since you compiled from source you don't have nice startup scripts. To get your nice startup scripts from an existing ganesha RPM do the following. Then use systemctl to stop and start nfs-ganesha as you would any other service.
yumdownloader nfs-ganesha.x86_64
rpm2cpio nfs-ganesha-2.5.5-1.el7.x86_64.rpm | cpio -idmv ./usr/lib/systemd/system/nfs-ganesha-lock.service
rpm2cpio nfs-ganesha-2.5.5-1.el7.x86_64.rpm | cpio -idmv ./usr/lib/systemd/system/nfs-ganesha.service
rpm2cpio nfs-ganesha-2.5.5-1.el7.x86_64.rpm | cpio -idmv ./usr/lib/systemd/system/nfs-ganesha-config.service
rpm2cpio nfs-ganesha-2.5.5-1.el7.x86_64.rpm | cpio -idmv ./usr/libexec/ganesha/nfs-ganesha-config.sh
Copy above to the same folders under / instead of ./ :
systemctl enable nfs-ganesha.service
systemctl status nfs-ganesha.service
If you're reading this up to this point, then you're still with me and you're done with the setup. Good. So let's go down to the testing bits. But first ensure all services are up and working on both nodes. Two handy one liners for this are:
[root@nfs01 n]# ps -ef|grep -Ei "haproxy|ganesha|keepalived"; netstat -pnlt|grep -Ei ganesha; netstat -pnlt|grep -Ei haproxy; netstat -pnlt|grep -Ei keepalived
[root@nfs02 n]# gluster volume status
First mount the volume using the cluster IP on any client of your choice:
[root@ipaclient01 /]# mount -t nfs4 nfs-c01:/n /n
From here, it's all the fun bits because we get to blow up things. Start and stop either node one at a time to see if your client ever looses a connection and if data is replicated across via Gluster to ensure no files are ever lost. Note: Shut down the nodes one at a time and let the previous one come up before you shut down the second node. All in all, you should see an entirely uninterrupted session off the client end.
You can see the full NFS Ganesha HA NFS Server setup or the condensed version available on my OpenSource blog.