The ramblings of Steve-0Posts RSS Comments RSS

steve on Mar 15th 2010 at 11:06 am tech

Creating an NFS-root VM template for Xen / Oracle VM

Following up on my previous post about building a Xen / NFS-root kernel, this will take you through creating a VM template capable of using that kernel (or a P2V process for converting existing VM’s / linux servers to nfs-root). I think the intro from the last article still applies, so I’ll include it here:

Over the past year or two, we have transitioned all our servers and hosting to Xen – specifically, we use the Oracle VM management tools on top of xen, and most of our VM’s are Centos 5 x86_64. Since we use NFS NAS’ as storage across our infrastructure, it would be very convenient if we could use a NFS volume as the root drive for VM’s. With the NFS root, we gain things like easy use of filer snapshots, and on-the-fly volume resizing – if we’re ever running short on space in a given VM, its a single command (or a click on a web page) to expand the root drive.

The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.

There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.

  • Install linux / kickstart / etc to get a good, minimal install of linux, and configure it as you would any linux server. I would suggest a minimal install for your requirements, turn off any unneeded services, lock down permissions, firewall, and any other deployment process you usually go through. Since this is going to be a template, you want this to be ready for any task with minimal configuration changes.
  • On that configured linux system, mount a NFS share that will serve as your future NFS root (or storage for your VM template) to /mnt/tmp
  • Shut down as many running services as possible, so there are no file conflicts/open databases/etc.
  • Copy over the kernel modules from your nfs-root kernel build:rsync -av /path/to/kernel/modules/kernel-ver /mnt/tmp/lib/modules/
  • #copy all files over to nfs mount
    cd /
    cp -ax /{bin,dev,etc,lib,lib64,opt,root,sbin,usr,var,folders} /mnt/tmp
    mkdir /mnt/tmp/{home,proc,sys,tmp}
    chmod 777 /mnt/tmp/tmp
  • Edit /mnt/tmp/etc/fstab to look something like this, main change here is to /:

    /dev/nfs                /                       rootfs  defaults        0 0
    tmpfs                   /dev/shm                tmpfs   defaults        0 0
    devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
    sysfs                   /sys                    sysfs   defaults        0 0
    proc                    /proc                   proc    defaults        0 0
    rpc_pipefs              /var/lib/nfs/rpc_pipefs rpc_pipefs defaults     0 0
  • Edit /mnt/tmp/etc/sysconfig/network-scripts/ifcfg-eth0 as needed to setup the network
  • Create a vm.cfg file for the Xen vm. If using this is an Oracle VM template, make sure to add a zero-byte disk image, and put this under the seed_pool directory. Edit options as needed for your infrastructure.
    dhcp = 'off'
    extra = 'nfsroot=10.0.0.1:/vol/nfsroot,noacl,nfsvers=3,tcp,rsize=32768,wsize=32768 selinux=0 acpi=off noapic'
    gateway = '10.0.0.254'
    hostname = 'nfsroot-vm.mycompany.com'
    ip = '10.0.0.10'
    kernel = '/OVS/P2V_STUFF/nfsboot-custom/vmlinuz-2.6.31.1nfsboot'
    memory = '512'
    name = 'nfsroot-vm'
    netmask = '255.255.255.0'
    on_crash = 'restart'
    on_reboot = 'restart'
    root = '/dev/nfs'
    uuid = '9af0a816-0123-4567-ad50-bc32be92bff7'
    vcpus = 2
    vfb = ['type=vnc,vncunused=1,vnclisten=0.0.0.0,vncpasswd=password']
    vif = ['bridge=vlan100,mac=00:16:3E:31:FF:37,type=netfront']
    vif_other_config = []
  • Start up the vm with Oracle VM manager, or start on the command line with xm create vm.cfg, and cross your fingers!

    • Tags: , , , , ,

    No responses yet

    Trackback URI | Comments RSS

    Leave a Reply

    You must be logged in to post a comment.