DC28 VMware HA cluster in enterprise – deep dive and best practises

An interesting subject for a presentation. Lots of people agreed on that and attended this session. Around 250 people were in the audience. The presenter was an employee of VMware. The presentation itself was a bit hard to understand as english was not the native language of the presenter.

While High Availability is easy to configuere by clicking a box, things can go wrong without understanding of what is underneath the hood. This session had interesing info on HA.

The technique of HA was bought by VMware from Legato. The Legato product is called Legato Automated Availability Manager. The presentation shows an overview of the communications between the AM agent used for HA and the vCenter agent. Even when vCenter Server is not availabe, HA will work because the HA agent can still ‘talk’ with the vCenter agent running on the ESX host.

Important to understand is that 5 nodes in a cluster can have the primary node role. Only a host having the primary node role is able to start VM’s on a ESX clusternode if one host has failed. If all 5 primary nodes are unavailable no VM’s will start.

The first 5 nodes added to a cluster will automatically be assigned as primary node. To change to primary node role to another ESX host, the AAM> console can be used. AAM>listnode gives an overview, demotenode and promote node commands can be used to change roles.

If one of the nodes having a primary role in the cluster is removed from the cluster, a random node will be selected as a primary node.

Each node in the cluster send a ‘ping’ signal to the configuered default gateway to check if it is still connected. 15 seconds after a node failure VM’s will be restarted on other nodes. This timeout can be configuered in the advanced properties of the cluster.

A couple of configurations can prevent VM’s to restart after node failure:

1- if all 5 primary nodes are located on the same failed blade enclosure. To prevent this, make sure a cluster has not more than 4 nodes.

2-if one of two switches fails. So even with redundant NIC’s it is possible to not have VM’s restarted. This is because the spanning tree on the remaining switchs needs time (around 60 seconds) tore build the routetable. This time is longer than the 15 seconds after which a node is considered down. As a result, all VM’s will be down. 

To prevent this situation on Cisco switches, enable the PortFast or fast-start feature. Alternatively the isolation time can be increased to a value higher than 60 seconds.