Stray Photons: Adventures in virtualization

I suspected the day was going to suck when I got up Friday morning and noticed that our Internet connection didn't work. This was quickly confirmed when I wandered down to check on the servers and found the firewall lying in a gooey puddle of Pentium soup. Well, not really, but it was pretty close. You see, the power supply fan had failed during the night and the whole system was running a serious fever. So high, in fact, that two of the three NICs didn't initialize on boot, one didn't even show up anymore, and the BIOS setup program wouldn't load.

Well, I didn't have spare PC hardware lying around, and I've been looking for an excuse to try out the new VMWare ESXi hypervisor, so I decided to give it a spin. I rigged up temporary Internet access by connecting the router directly to another server, gave VMWare all my personal information (a rant for another day), downloaded and burned ESXi. Then I booted it up... and it crashed spectacularly with a "NOT_IMPLEMENTED" error that looks a lot like a Windows blue screen of death, only pinkish. Sigh. Now I suspect that my old AMD Athlon64 PC might not be on the hardware compatibility list, but that's a pretty piss-poor way of handling an exception. Time for plan B.

Enter Xen: something else I've been meaning to try out. I installed Linux on the host (on a spare disk), and spent several hours trying to figure out how to get an existing installation of NetBSD on a physical disk into a Xen VM. Unfortunately, I just couldn't manage it. Although I had the netbsd-XEN3_DOMU kernel image, I couldn't convince Xen to boot it. At first, it kept telling me that the kernel image didn't exist. I finally got enough of a working Xen domain config file written (by hand) that it would attempt to load the NetBSD kernel - only to discover that NetBSD kernels don't have PAE support, which Xen needs. (Well... which the Xen built into Fedora needs, anyway.) At this point, I could have rebuilt Xen without PAE support, but then I'd have trouble booting other OSes that do have PAE. Or I could have reinstalled the host with the x86_64 version of Fedora and installed NetBSD/x86_64 into the domU VM, and migrated everything from the old install to the new install... but somehow that just wasn't appealing.

I decided to go with tried-and-true plan C: VMWare Server. I already have it, it's fairly solid and stable, and it can handle physical disks reasonably well which makes for easier physical to virtual migration. I don't think it's as efficient as paravirtualization, but at least it works. (By now it had been about 8 hours of hacking on this so I didn't feel much like "exploring" anymore and just wanted something to work.) I created a small virtual disk for the guest OS, added the real disk as the second drive, and booted it up from a CD. Copying all the files over took a long time. I reinstalled the boot blocks and fired up the VM. Success! Well, almost. I had to adjust the /etc/fstab to mount the new virtual disks properly, and the network cards now have new names so I had to edit the network setup scripts and firewall rules. But success anwyay!

Then I had to migrate the other server into a VM as well, since I am now using it's physical hardware as the host. I created a small virtual disk for the root partition and copied the files from the physical disk. Then I added the two former physical disks to my new LVM volume group and extended the filesystem across them. Two new virtual disks and lots of slow, slow copying later, I had another running VM. After similar fstab and config adjustments, I was finally done. At 2:00 AM.

Anyway, despite all the hassles and work, I learned quite a bit about Xen and also about LVM. Unfortunately, I think this solution will have to be temporary. I'm not convinced that the VMWare guest is "real-time" enough to handle being a NAT gateway for VoIP traffic and other latency-sensitive applications. If I get more than 60ms to the World of Warcraft server, it's just not an acceptable solution :) I'm also not convinced that it will be secure enough as a VM. But, I'll give it a try for a while and we'll see how it goes.