First, test your version of Bash with this line:

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

If you get the world “vulnerable” in your output then you need to update Bash:

vulnerable
this is a test

If your output contains errors followed by “this is a test”, then your Bash version is not vulnerable:

bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'
this is a test

Check to see if your distribution has an updated/fixed version of Bash available in its repository.

I’m a heavy CentOS user, and I can verify that there is a fixed version available for both CentOS 5 and 6.

This is all you technically need:

yum update bash

It’s always recommended to fully update your system, but if you’re purposefully running “legacy” or deprecated versions of software then it can be advantageous (or necessary) to manually update Bash at minimum right now.

At work and at home I have pairs of redundant “core” routers in an active-passive (or master-backup as you like) configuration. They consist of commodity hardware, a few 4-port gigabit NICs, and CentOS. All of these machines had been running flawlessly for anywhere from two to six years (as they were put into service or upgraded).

That is until yesterday when my primary router at home had an SSD failure which completely stopped it in its tracks. The backup router took over, and in less than a second traffic was being routed. All of my point-to-point VPNs reconnected within about 20 seconds. In other words, it worked exactly as it should.

Until I turned off power to the broken router. Then everything stopped.

I had made a minor change to my router pair a few months ago, and didn’t think anything of it. Instead of running VRRP traffic through the switch, I had dedicated a NIC port on each machine and connected them directly using a crossover cable. I had only tested by bringing the primary router down gracefully, and did not pull the plug.

When the plug was pulled on the broken router, the now-master saw the link go down on the VRRP port and keepalived went into the FAULT state. It gave up its VIPs and basically stopped keeping anything alive.

That behavior can make sense in certain scenarios. For example, if just the NIC port used for VRRP went down on the master router, I wouldn’t want the backup also taking the VIPs (and certain routes, etc.) If I had VRRP going through one switch and production traffic going through another, I wouldn’t want a failure on the less important switch to again cause VIP conflicts.

In my case, I find it much (much, much, much) more likely that the link having gone down will mean that one of the machines has died completely. In my experience power supplies and HDDs (or SSDs) are far more likely to fail than a NIC or NIC port. It’s not to say that the latter is impossible, but rather that I have to plan for the most likely worst-case scenario.

All that being said, there is one setting for your keepalived.conf to obviate this issue: dont_track_primary

That’s it. It doesn’t have options or qualifiers. From the man page:

# Ignore VRRP interface faults (default unset)
dont_track_primary

From the keepalived changelog:

VRRP : Chris Caputo added "dont_track_primary"
vrrp_instance keyword which tells keepalived to ignore VRRP
interface faults. Can be useful on setup where two routers
are connected directly to each other on the interface used
for VRRP. Without this feature the link down caused
by one router crashing would also inspire the other router to lose
(or not gain) MASTER state, since it was also tracking link status.

Perfect, right?

Here’s my keepalive configuration that’s been sanitized and edited for brevity:

global_defs {
   notification_email {
     me@mydomain.corn
   }
   notification_email_from rtr-core02@int.meagain.net
   smtp_server 10.80.1.41
   smtp_connect_timeout 30
   router_id RTR-CORE-A
}
vrrp_instance VI_0 {
    state BACKUP
    interface p4p1
    smtp_alert
    virtual_router_id 50
    priority 50
    advert_int 1
    dont_track_primary
    notify_master /etc/keepalived/promotemaster
    notify_backup /etc/keepalived/promotebackup
    authentication {
        auth_type PASS
        auth_pass sanitizedpassword
    }
    virtual_ipaddress {
        192.168.1.1/24 brd 192.168.1.255 dev p3p1 label p3p1:100
        192.168.1.2/24 brd 192.168.1.255 dev p3p1 label p3p1:101
        10.1.1.1/24 brd 10.1.1.255 dev p3p2 label p3p2:100
        10.1.1.2/24 brd 10.1.1.255 dev p3p2 label p3p2:101
        # Many VIPs omitted here for brevity
    }
    virtual_routes {
        158.209.0.99/32 via 78.123.265.1 dev p1p1 table main
        0.0.0.0/0 via 91.59.24.131 dev p1p2 table 50
        193.266.0.0/16 via 91.59.24.131 dev p1p2 table main
        # Many routes omitted here for brevity.  IPs are sanitized/randomized
    }
}

I’m hoping that I put enough keywords in this article so that you found it easily. The whole point of this post is to counter the drought of discussion on this topic.

But as with all things computer related, that “no” needs to be followed by the caveat: “Well, it depends upon your needs.”

From what I’ve seen, Linux clustering was designed primarily for high-availability services, with only a secondary effort to share disk resources across nodes.

I have tried — and would never use in production — Linux clustering services for a VM host cluster. I know other people have done it and will continue to do it, but a properly configured (and managed) VM cluster does not need true clustering. (Again, “depending upon your needs”).

Linux clustering requires fencing. (It didn’t always, but now it does). Fencing is a great thing in a homogeneous cluster where every machine is a clone of every other, and the point of the cluster is that it can lose a machine or six and still provide the same service(s). The purpose of fencing is to “shoot a bad node in the head”. This can either mean power-cycling it with an iLO or PDU, or disconnecting it from shared resources such as a SAN at the switch level.

Fencing is tremendously undesirable in a VM host cluster. If the cluster decides that one of the nodes is bad, it will simply kill it. In a heterogeneous cluster. Killing potentially tens (or even hundreds) of your VM guests in one stroke.

Of course, in a VM cluster, fencing would still be required if you were using a shared file system. However, LVM2 is another matter.

Another downside about Linux clustering is that to bring a failed cluster back to a consistent state, the recommended solution is to reboot all of the machines in the cluster simultaneously. (I’ve found that recommendation made by developers in RHEL’s bug database, amongst other places). In a production VM cluster, that’s unacceptable.

Configuration and Management

From a configuration standpoint, there’s nothing special about running non-clustered LVM on a shared disk in a cluster. All you have to do is run vgcreate on one node using a shared LUN as a physical disk. Then run vgscan on the other nodes in the cluster and you’ll see your new volume group. No fuss, no muss.

From a management standpoint, you have to be careful. Very, very careful. Writing to the LVM metadata simultaneously from different nodes (such as doing two lvcreates) probably will result in metadata corruption, which could bork your entire cluster. It would be disastrous to employ cron jobs on more than one host, for example, that wrote to LVM’s metadata.

The best practice in this case would be to designate one node as the “metadata writer”. That simply means that you’d make all changes to LVM metadata from that machine. On all other cluster nodes, rename the LVM tools (usually /sbin/lvm and /sbin/lvmconf), and put your own script in their place. The script should output something like, “Please use the metadata writer node for changes to LVM”.

In most cases, command-line instances of LVM commands (e.g. vgcreate, lvchange, etc) are just symlinked to lvm. If you want to be thorough, change the symlinks for read-only operations like vgscan, vgdisplay, lvdisplay, etc. to point to the renamed lvm utility.

Another gotcha is that there’s nothing to stop you from running two different instances of the same VM using the same logical volume. Because there’s no distributed locking, the system will let you do it. Of course, this is great if you designate the virtual disk as read-only, because you can share things like repositories and application images across as many VMs as you’d like. But running two read/write instances will result in data corruption.

You do have recent backups, right?

Be Warned

I use this technique within a few CentOS / Xen clusters that I manage. The key being that I manage them. Bringing in an outside technician or new employee that doesn’t know how (and why) things are configured could very easily screw everything up, even with the best of intentions. (“Gee, I wonder why he renamed lvm to lvm.forreadonly. I’ll just use it anyway.”)

In fact, I’d wager that 99% of sysadmins would recommend that you don’t listen to me. However, I’ve been running things this way for over four years now without a hiccup (knock on chassis). Not having the overhead and headache of running a true cluster has been great.

Even I’d have to recommend against using this technique in a cluster larger than a handful of nodes.

The Benefit

Your mileage may vary, but my implementation is fairly straightforward: In my SAN array, I export each set of spindles as a LUN consisting of 100% of their capacity. The LUN is exposed to all the VM hosts, and I create a single volume group on the LUN-cum-physical disk. I then create logical volumes within that VG as needed for the VM guests. A LV is then used as a physical disk asset by a VM guest. (I know some people stick partitions on top of their LVs, but I don’t see the benefit to that).

I can do live migrations, and for times where that’s not necessary or desirable (e.g. a VM with a large memory footprint that is not mission critical), I use shared configuration files.

I store all of the conf files for my virtual machines on a shared LV that’s mounted as read-only on all VM hosts (dom0s) except for the “metadata writer” where it’s mounted as read/write. Therefore, when I want to migrate a guest VM from one host to another it’s just a matter of bringing the VM down, symlinking to the shared conf file on the new host, and removing the symlink from the old host.

In Conclusion

I wanted to share my experiences and techniques not as a HOWTO, but simply as another way of looking at sharing resources between servers. I’ve gotten many a raised eyebrow (and worse) from other sysadmins when I’ve described my setup. But it does work, and despite all the pitfalls and caveats involved I maintain that it’s still pretty easy to ruin or disrupt a “traditional” cluster. This just has less overhead, and fewer places to make mistakes (though mistakes can be really, really B-A-D).