Creating an NFS-root VM template for Xen / Oracle VM
Following up on my previous post about building a Xen / NFS-root kernel, this will take you through creating a VM template capable of using that kernel (or a P2V process for converting existing VM’s / linux servers to nfs-root). I think the intro from the last article still applies, so I’ll include it here:
Over the past year or two, we have transitioned all our servers and hosting to Xen – specifically, we use the Oracle VM management tools on top of xen, and most of our VM’s are Centos 5 x86_64. Since we use NFS NAS’ as storage across our infrastructure, it would be very convenient if we could use a NFS volume as the root drive for VM’s. With the NFS root, we gain things like easy use of filer snapshots, and on-the-fly volume resizing – if we’re ever running short on space in a given VM, its a single command (or a click on a web page) to expand the root drive.
The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.
There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.
rsync -av /path/to/kernel/modules/kernel-ver /mnt/tmp/lib/modules/#copy all files over to nfs mount
cd /
cp -ax /{bin,dev,etc,lib,lib64,opt,root,sbin,usr,var,folders} /mnt/tmp
mkdir /mnt/tmp/{home,proc,sys,tmp}
chmod 777 /mnt/tmp/tmp
/dev/nfs / rootfs defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs defaults 0 0
dhcp = 'off'
extra = 'nfsroot=10.0.0.1:/vol/nfsroot,noacl,nfsvers=3,tcp,rsize=32768,wsize=32768 selinux=0 acpi=off noapic'
gateway = '10.0.0.254'
hostname = 'nfsroot-vm.mycompany.com'
ip = '10.0.0.10'
kernel = '/OVS/P2V_STUFF/nfsboot-custom/vmlinuz-2.6.31.1nfsboot'
memory = '512'
name = 'nfsroot-vm'
netmask = '255.255.255.0'
on_crash = 'restart'
on_reboot = 'restart'
root = '/dev/nfs'
uuid = '9af0a816-0123-4567-ad50-bc32be92bff7'
vcpus = 2
vfb = ['type=vnc,vncunused=1,vnclisten=0.0.0.0,vncpasswd=password']
vif = ['bridge=vlan100,mac=00:16:3E:31:FF:37,type=netfront']
vif_other_config = []
