Clustering,High Availability,Linux,Xen July 8, 2012 1:56 pm

Fencing Linux Clusters Nodes on XenSever/XCP Using XenAPI

As many of you already know, fencing is an important component of maintaining the health of your cluster. When cluster nodes experience issues, behave improperly or overall, just aren’t playing nice with the remainder of the nodes, it’s important to bring down that node as fast as possible, otherwise you risk service interruption or even worse, data corruption!

Before the prevalence of Virtualization in the Data Center, the most common way to fence a node was log into it’s IPMI or DRAC card and issue a shutdown/reboot command. If the node didn’t have a DRAC or IPMI, you could also log into the PDU it was connected to and power-off the outlet. Either of the two methods ensured that if required, cluster nodes could be quickly taken offline when necessarily.

Well on virtualized cluster nodes, there isn’t a dedicated IPMI or DRAC card. And you certainly wouldn’t want to log into the PDU and shutdown the entire physical host. So the only methods left are those that require the host to self-fence or those that require in-band access to the host to issue a reboot/shutdown command. Sadly, these methods unreliable at best. For example, if a node is unstable and has partially crashed, ssh access may be unavailable, or components of the OS may be unstable so it cannot properly self-fence. So then, what is the best way to fence an unstable, vm based cluster-node?

Well, if you are using XenServer or XCP, then I’d say the best way to do that would be through an agent or script that leverages XAPI. For example, if you could execute the xe vm-reboot or xe vm-reset-powerstate when a vm cluster node was unstable or hung, then that would be awesome. It would be even better if this script or agent was integrated with the cluster stack so that it worked inside rather than outside (e.g. so that the cluster itself was responsible for detecting when it was necessary to fence a node as opposed to an external script or agent), but does such a thing even exist??

Well kiddies, yes it does. And we have Matthew J Clark to thank for it. He wrote a pretty nice fencing agent for XenServer/XCP that does everything I said above. You can download it over at his Google Code Page or from the git repo.
I recommend you pull it down from the git repository at It works with CMAN (I think lol) and Pacemaker clusters, though I’ve only tested it with the later.

Pull it down, build it (it requires pexpect and python-suds) and install it. You should have /usr/sbin/fence_xenapi after you install it. Run stonith_admin -M -a fence_xenapi to check out it’s metadata:

<?xml version="1.0" ?>
<resource-agent name="fence_xenapi" shortdesc="XenAPI based fencing for the Citrix XenServer virtual machines." >
<longdesc>fence_cxs is an I/O Fencing agent used on Citrix XenServer hosts. It uses the XenAPI, supplied by Citrix, to establish an XML-RPC sesssion to a XenServer host. Once the session is established, further XML-RPC commands are issued in order to switch on, switch off, restart and query the status of virtual machines running on the host.</longdesc>
        <parameter name="action" unique="0" required="1">
                <getopt mixed="-o, --action=<action>" />
                <content type="string" default="reboot" />
                <shortdesc lang="en">Fencing Action</shortdesc>
        <parameter name="login" unique="0" required="0">
                <getopt mixed="-l, --username=<name>" />
                <content type="string"  />
                <shortdesc lang="en">Login Name</shortdesc>
        <parameter name="passwd" unique="0" required="0">
                <getopt mixed="-p, --password=<password>" />
                <content type="string"  />
                <shortdesc lang="en">Login password or passphrase</shortdesc>
        <parameter name="passwd_script" unique="0" required="0">
                <getopt mixed="-S, --password-script=<script>" />
                <content type="string"  />
                <shortdesc lang="en">Script to retrieve password</shortdesc>
        <parameter name="port" unique="0" required="1">
                <getopt mixed="-n, --plug=<id>" />
                <content type="string"  />
                <shortdesc lang="en">Physical plug number or name of virtual machine</shortdesc>
        <parameter name="session_url" unique="0" required="1">
                <getopt mixed="-s, --session-url" />
                <content type="string"  />
                <shortdesc lang="en">The URL of the XenServer host.</shortdesc>
        <parameter name="uuid" unique="0" required="0">
                <getopt mixed="-U, --uuid" />
                <content type="string"  />
                <shortdesc lang="en">The UUID of the virtual machine to fence.</shortdesc>
        <parameter name="verbose" unique="0" required="0">
                <getopt mixed="-v, --verbose" />
                <content type="boolean"  />
                <shortdesc lang="en">Verbose mode</shortdesc>
        <parameter name="debug" unique="0" required="0">
                <getopt mixed="-D, --debug-file=<debugfile>" />
                <content type="string"  />
                <shortdesc lang="en">Write debug information to given file</shortdesc>
        <parameter name="version" unique="0" required="0">
                <getopt mixed="-V, --version" />
                <content type="boolean"  />
                <shortdesc lang="en">Display version information and exit</shortdesc>
        <parameter name="help" unique="0" required="0">
                <getopt mixed="-h, --help" />
                <content type="boolean"  />
                <shortdesc lang="en">Display help and exit</shortdesc>
        <parameter name="separator" unique="0" required="0">
                <getopt mixed="-C, --separator=<char>" />
                <content type="string" default="," />
                <shortdesc lang="en">Separator for CSV created by operation list</shortdesc>
        <parameter name="power_timeout" unique="0" required="0">
                <getopt mixed="--power-timeout" />
                <content type="string" default="20" />
                <shortdesc lang="en">Test X seconds for status change after ON/OFF</shortdesc>
        <parameter name="shell_timeout" unique="0" required="0">
                <getopt mixed="--shell-timeout" />
                <content type="string" default="3" />
                <shortdesc lang="en">Wait X seconds for cmd prompt after issuing command</shortdesc>
        <parameter name="login_timeout" unique="0" required="0">
                <getopt mixed="--login-timeout" />
                <content type="string" default="5" />
                <shortdesc lang="en">Wait X seconds for cmd prompt after login</shortdesc>
        <parameter name="power_wait" unique="0" required="0">
                <getopt mixed="--power-wait" />
                <content type="string" default="0" />
                <shortdesc lang="en">Wait X seconds after issuing ON/OFF</shortdesc>
        <parameter name="delay" unique="0" required="0">
                <getopt mixed="--delay" />
                <content type="string" default="0" />
                <shortdesc lang="en">Wait X seconds before fencing is started</shortdesc>
        <parameter name="retry_on" unique="0" required="0">
                <getopt mixed="--retry-on" />
                <content type="string" default="1" />
                <shortdesc lang="en">Count of attempts to retry power on</shortdesc>
        <action name="on" />
        <action name="off" />
        <action name="reboot" />
        <action name="status" />
        <action name="list" />
        <action name="monitor" />
        <action name="metadata" />

Notice the use the pcmk_host and pcmk_host_check in the primitive. The agent isn’t able to map cluster nodes back to vm-labels on it’s own, so the pcmk_host stanza provides that mapping. Also, the location blocks ensure that a node cannot fence itself.
To test, you’d simply need to execute the stonith_admin -B <node-name>. That will reboot the specified node.
After you’ve tested and you are ready to go production, don’t forget to set stonith-enabled=true in your properties and no-quorum-policy="ignore" and you’ll be all set.


1 Comment

  • matthias hoffmann

    I am trying to get that stonith primitive run. could you please explain this pcmk_host and pcmk_host_check params? I have no idea which values are needed for them. Have two physical server with xenserver, wanna reboot a vm via stonith primitive.
    could you please provide an example ?

Leave a reply