How to create a HA VIP with Corosync on OpenStack
Ansible Automation

How to create a HA VIP with Corosync on OpenStack

Running applications HA (Highly Available) has become the norm. One way of doing this is by deploying your applications in Kubernetes. Exposing your application endpoint is typically done using a loadbalancer, in Kubernetes you can run an Ingress for this. However, this still makes the node running the Ingress pod the single point of failure.

A solution to this problem is running Ingress pods on multiple nodes and let these nodes share a VIP (Virtual IP). This is where Corosync comes in. The VIP will live on one of the nodes. When this node fails, Corosync makes sure one of the other nodes takes over the VIP.

In this tutorial we’re not going to install an entire Kubernetes cluster. Instead, as a proof of concept, we’re going to install three nodes with the Nginx webserver.

The VIP and the instances are going to live in the range 10.0.0.0/24. We’re reserving 10.0.0.2-10.0.0.200 for instances and 10.0.0.201-10.0.0.255 for VIPs. We’re using just one VIP in this tutorial, which will be 10.0.0.201. In order to give the VIP and externally reachable address, we’ll make a Neutron port for the VIP and attach a floating IP address to it.

Configure virtual network

We need to create a network with a subnet and the range 10.0.0.0/24. We’re going to reserve everything up from 10.0.0.220 for VIP addresses.

openstack network create vip-net
openstack subnet create vip-subnet --network vip-net --allocation-pool start=10.0.0.2,end=10.0.0.200 --subnet-range 10.0.0.0/24

We need to attach the subnet to your projects router. To get a list of the routers you have type:

openstack router list

Giving a list that looks like this:

+--------------------------------------+----------------------------------------+--------+-------+-------------+------+----------------------------------+
| ID                                   | Name                                   | Status | State | Distributed | HA   | Project                          |
+--------------------------------------+----------------------------------------+--------+-------+-------------+------+----------------------------------+
| edc48681-b094-4b8e-9b0e-d6452f86f5ce | router-internal-to-external-my-project | ACTIVE | UP    | None        | None | c7a894e64e9a495082f1170fbdde3aa3 |
+--------------------------------------+----------------------------------------+--------+-------+-------------+------+----------------------------------+

Use the router ID in the first column to attach the new subnet to the router, giving us access to the outside world.

openstack router add subnet edc48681-b094-4b8e-9b0e-d6452f86f5ce vip-subnet

Security groups

We need to set some firewall rules in order to access our instances and to make sure Corosync can form a cluster.

openstack security group create vip
openstack security group rule create vip --remote-ip 0.0.0.0/0 --protocol tcp --dst-port 22:22 --description "Allow incoming SSH traffic"
openstack security group rule create vip --remote-ip 0.0.0.0/0 --protocol tcp --dst-port 80:80 --description "Allow incoming HTTP traffic"
openstack security group rule create vip --remote-group vip --protocol udp --dst-port 5405:5405 --description "Allow internal Corosync traffic"

Launch instances

Now that the network is in place, we need to launch the instances we’re going to run Corosync on.

To get access to the instances after we’ve launched them we need to set the key-name from your SSH key. If you don’t have one yet, see Configure secure access for instances.

 openstack keypair list

This gives a result that should look like this:

+----------------------+-------------------------------------------------+
| Name                 | Fingerprint                                     |
+----------------------+-------------------------------------------------+
| my-key               | a1:71:e0:c4:20:7f:90:c3:5f:67:a9:8e:78:7c:64:71 |
+----------------------+-------------------------------------------------+

Use in the following command the name of your key instead of my-key.

openstack server create --image "Ubuntu 18.04 LTS - Bionic Beaver - 64-bit - Fuga Cloud Based Image" --flavor "c1.small" --min 3 --max 3 --network vip-net --security-group vip --key-name my-key vip-instance

Configure ports

We now need to do some configuration on the network ports of the instances. By default OpenStack will drop traffic not belonging to the designated IP of an instance, so we need to allow the instances to use the VIP on the subnet they’re on. We’ll start with vip-instance-1.

openstack port list --server vip-instance-1

Giving a result like this:

+--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+
| ID                                   | Name | MAC Address       | Fixed IP Addresses                                                      | Status |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+
| 5ef652b2-0eaa-4a8f-8b9d-53c6fe89e009 |      | fa:16:3e:2a:f3:02 | ip_address='10.0.0.4', subnet_id='be475910-de5a-456f-a073-d3e6f8f4c84e' | ACTIVE |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+

Now let’s add our vip as allowed IP:

openstack port set 5ef652b2-0eaa-4a8f-8b9d-53c6fe89e009 --allowed-address ip-address=10.0.0.201

While we’re add it, let’s also attach a floating ip address to the port, so we can reach our instance.

openstack floating ip create external --port 5ef652b2-0eaa-4a8f-8b9d-53c6fe89e009

Do the same for the ports of vip-instance-2 and vip-instance-3.

Now we need to create a port for the VIP, so we can bind a floating IP and security group to it:

openstack port create vip-1 --network vip-net --fixed-ip ip-address=10.0.0.201 --security-group vip

Copy the newly create port ID and use it to attach a floating IP to it:

openstack floating ip create external --port ef2cd8eb-18fa-4bfa-b1c1-71b067a700f4

Note the floating ip somewhere, this is the VIP floating IP and we’ll use it to access our test application later.

Configure Corosync

Now that we have the OpenStack part of things configured we can focus on getting Corosync up and running.

Log in to your instances using the floating IP’s you attached to them.

ssh ubuntu@<instance-floating-ip>

On all three nodes install Pacemaker, which has Corosync as dependency, and the Corosync CLI:

sudo apt update
sudo apt install pacemaker crmsh

Open the Corosync configuration in your favorite editor (which is obviously VIM) and set the bindnetaddr to the vip-subnet range (10.0.0.0).

sudo vim /etc/corosync/corosync.conf

Which looks like this:

...
   interface {
...
        bindnetaddr: 10.0.0.0
...
   }
...

Check for syntax errors:

corosync -f

If everything is OK, restart corosync.

sudo systemctl restart corosync

The rest of the commands can be run on just one instance of the cluster, I’m using vip-instance-1.

You can check to state of the Corosync cluster with:

sudo crm_mon -1

The cluster first needs to form, so give it some time. After about a minute you should see this:

Stack: corosync
Current DC: vip-instance-3 (version 1.1.18-2b07d5c5a9) - partition with quorum
Last updated: Wed Jun 20 15:33:37 2018
Last change: Wed Jun 20 15:32:19 2018 by hacluster via crmd on vip-instance-3

3 nodes configured
0 resources configured

Online: [ vip-instance-1 vip-instance-2 vip-instance-3 ]

No active resources

We’re going to disable STONITH (Shoot The Other Node In The Head), because configuring it is outside the scope of this tutorial.

sudo crm configure property stonith-enabled=false

Now check the cluster configuration.

sudo crm_verify -LV

This should give an empty output.

Add the VIP to Corosync

Now that our Corosync cluster is up and running, we need to add our VIP as a Corosync service.

sudo crm configure primitive VIP ocf:IPaddr2 params ip=10.0.0.201 nic=ens3 op monitor interval=10s

The VIP should now be added a as service. Check with sudo crm_mon -1.

Stack: corosync
Current DC: vip-instance-3 (version 1.1.18-2b07d5c5a9) - partition with quorum
Last updated: Wed Jun 20 15:44:04 2018
Last change: Wed Jun 20 15:43:28 2018 by root via cibadmin on vip-instance-1

3 nodes configured
1 resource configured

Online: [ vip-instance-1 vip-instance-2 vip-instance-3 ]

Active resources:

 VIP	(ocf::heartbeat:IPaddr2):	Started vip-instance-1

Install the webserver

Let’s install Nginx on our instances, so we can test our public VIP. Do the following on all three instances:

sudo apt update
sudo apt install nginx

Test it

Let’s put it to the test. Browse to the floating IP you attached the to VIP port. You should see the Nginx welcome page.

Nginx welcome page

In my case the VIP lives on vip-instance-1, so I’m bringing it down.

openstack server stop vip-instance-1

The Nginx webpage should still be available. And you should see the VIP has been moved to one of the other instances:

Stack: corosync
Current DC: vip-instance-3 (version 1.1.18-2b07d5c5a9) - partition with quorum
Last updated: Fri Jun 22 09:00:25 2018
Last change: Wed Jun 20 15:43:28 2018 by root via cibadmin on vip-instance-1

3 nodes configured
1 resource configured

Online: [ vip-instance-2 vip-instance-3 ]
OFFLINE: [ vip-instance-1 ]

Active resources:

VIP     (ocf::heartbeat:IPaddr2):       Started vip-instance-2

Was this article helpful?


Next article:

How to connect to the Fuga Cloud ObjectStore using .NET Core

The possibilities with the Fuga Object Store are endless. One of the scenarios is connecting your .NET Core application to the Fuga Object Store, this way you can access for example files on your Object Store! In this tutorial we will teach you how to make a simple .NET Core MVC file upload application that uses the Fuga Object Store. Requirements .NET Core 2.1 SDK API keys To connect to your Fuga Object Store access and secret keys are required.