Build a custom Xen kernel capable of booting from a NFS Root Filesystem
Over the past year or two, we have transitioned all our servers and hosting to Xen – specifically, we use the Oracle VM management tools on top of xen, and most of our VM’s are Centos 5 x86_64. Since we use NFS NAS’ as storage across our infrastructure, it would be very convenient if we could use a NFS volume as the root drive for VM’s. With the NFS root, we gain things like easy use of filer snapshots, and on-the-fly volume resizing – if we’re ever running short on space in a given VM, its a single command (or a click on a web page) to expand the root drive.
The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.
There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.
First, you need to build a kernel that supports both xen and an NFS-root – you’ll need a linux machine with a complete build environment, i.e. gcc, make, etc. I have been through this process with 2.6.31, but I would guess that the latest stable kernel version available at http://kernel.org will work just fine.
- Download the latest kernel source
- unzip, cd into source directory
- Copy in attached .config as starting point ( copy as …./linux-(version)/.config )
- make menuconfig
Important config options are listed here:
http://www.virtuatopia.com/index.php/CentOS_5_-_Build_a_Custom_Xen_Kernel_with_NFS_Root_Filesystem_(CONFIG_ROOT_NFS)_Support
Most importantly, nfs client options and nfs_root need to be built into kernel (not as modules). Also need to make sure to build Xen modules, and select most of the iptables filters (state is an important one).
- make
- make modules_install
Once the make and install is complete, you will have a kernel capable of nfs-booting under xen. To collect all the pieces needed:
- copy over the file vmlinux in the base build directory (this one is ~85MB), this is the kernel (can’t use bzImage with xen nfs boot).
- Tar up /lib/modules/kernelVer to distribute to nfs client vm.
See the next article for creating a proper vm.cfg under xen / Oracle VM.
No responses yet
