The ramblings of Steve-0Posts RSS Comments RSS

Archive for March, 2010

on Mar 25th 2010 tech

Setting up Oracle VM Server 2.2 with a NFS Repository

Oracle’s documentation on this is a not very well organized, and simply doesn’t work out of the box, at least with the 2.2 install CD that’s available for download. The main problem is that the version of ovs-agent that ships with Oracle VM 2.2 is broken, so following the instructions you’ll end up frustrated and wondering what you did wrong, i.e.:
[root@oraclevm8 /]# /opt/ovs-agent-2.3/utils/repos.py --init
Cluster not available.

To set up OracleVM 2.2 Server, with a NFS repository for VM’s:

  • Download the ISO, and run through the install process
  • Once it is up and you are logged in:
  • Download and install the updated ovs-agent rpm from oracle here: http://oss.oracle.com/oraclevm/server/2.2/RPMS/ovs-agent-2.3-31.noarch.rpm
    • wget http://oss.oracle.com/oraclevm/server/2.2/RPMS/ovs-agent-2.3-31.noarch.rpm
    • rpm -Uvh ovs-agent-2.3-31.noarch.rpm
  • Add your repository with the repos.py tool, make it the root repository, and initialize it:

    # /opt/ovs-agent-2.3/utils/repos.py --new mynfsserver.company.com:/volumes/datavol/ovs
    # /opt/ovs-agent-2.3/utils/repos.py -l
    # /opt/ovs-agent-2.3/utils/repos.py -r 7c885f7e-7dfe-4108-bae4-4dc329e2f017
    # /opt/ovs-agent-2.3/utils/repos.py -i

At this point, you are ready to go, and can continue setting things up (including your Oracle VM Manager VM).

No responses yet

on Mar 15th 2010 tech

Creating an NFS-root VM template for Xen / Oracle VM

Following up on my previous post about building a Xen / NFS-root kernel, this will take you through creating a VM template capable of using that kernel (or a P2V process for converting existing VM’s / linux servers to nfs-root). I think the intro from the last article still applies, so I’ll include it here:

Over the past year or two, we have transitioned all our servers and hosting to Xen – specifically, we use the Oracle VM management tools on top of xen, and most of our VM’s are Centos 5 x86_64. Since we use NFS NAS’ as storage across our infrastructure, it would be very convenient if we could use a NFS volume as the root drive for VM’s. With the NFS root, we gain things like easy use of filer snapshots, and on-the-fly volume resizing – if we’re ever running short on space in a given VM, its a single command (or a click on a web page) to expand the root drive.

The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.

There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.

  • Install linux / kickstart / etc to get a good, minimal install of linux, and configure it as you would any linux server. I would suggest a minimal install for your requirements, turn off any unneeded services, lock down permissions, firewall, and any other deployment process you usually go through. Since this is going to be a template, you want this to be ready for any task with minimal configuration changes.
  • On that configured linux system, mount a NFS share that will serve as your future NFS root (or storage for your VM template) to /mnt/tmp
  • Shut down as many running services as possible, so there are no file conflicts/open databases/etc.
  • Copy over the kernel modules from your nfs-root kernel build:rsync -av /path/to/kernel/modules/kernel-ver /mnt/tmp/lib/modules/
  • #copy all files over to nfs mount
    cd /
    cp -ax /{bin,dev,etc,lib,lib64,opt,root,sbin,usr,var,folders} /mnt/tmp
    mkdir /mnt/tmp/{home,proc,sys,tmp}
    chmod 777 /mnt/tmp/tmp
  • Edit /mnt/tmp/etc/fstab to look something like this, main change here is to /:

    /dev/nfs                /                       rootfs  defaults        0 0
    tmpfs                   /dev/shm                tmpfs   defaults        0 0
    devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
    sysfs                   /sys                    sysfs   defaults        0 0
    proc                    /proc                   proc    defaults        0 0
    rpc_pipefs              /var/lib/nfs/rpc_pipefs rpc_pipefs defaults     0 0
  • Edit /mnt/tmp/etc/sysconfig/network-scripts/ifcfg-eth0 as needed to setup the network
  • Create a vm.cfg file for the Xen vm. If using this is an Oracle VM template, make sure to add a zero-byte disk image, and put this under the seed_pool directory. Edit options as needed for your infrastructure.
    dhcp = 'off'
    extra = 'nfsroot=10.0.0.1:/vol/nfsroot,noacl,nfsvers=3,tcp,rsize=32768,wsize=32768 selinux=0 acpi=off noapic'
    gateway = '10.0.0.254'
    hostname = 'nfsroot-vm.mycompany.com'
    ip = '10.0.0.10'
    kernel = '/OVS/P2V_STUFF/nfsboot-custom/vmlinuz-2.6.31.1nfsboot'
    memory = '512'
    name = 'nfsroot-vm'
    netmask = '255.255.255.0'
    on_crash = 'restart'
    on_reboot = 'restart'
    root = '/dev/nfs'
    uuid = '9af0a816-0123-4567-ad50-bc32be92bff7'
    vcpus = 2
    vfb = ['type=vnc,vncunused=1,vnclisten=0.0.0.0,vncpasswd=password']
    vif = ['bridge=vlan100,mac=00:16:3E:31:FF:37,type=netfront']
    vif_other_config = []
  • Start up the vm with Oracle VM manager, or start on the command line with xm create vm.cfg, and cross your fingers!

    • No responses yet

    on Mar 15th 2010 tech

    Build a custom Xen kernel capable of booting from a NFS Root Filesystem

    Over the past year or two, we have transitioned all our servers and hosting to Xen – specifically, we use the Oracle VM management tools on top of xen, and most of our VM’s are Centos 5 x86_64. Since we use NFS NAS’ as storage across our infrastructure, it would be very convenient if we could use a NFS volume as the root drive for VM’s. With the NFS root, we gain things like easy use of filer snapshots, and on-the-fly volume resizing – if we’re ever running short on space in a given VM, its a single command (or a click on a web page) to expand the root drive.

    The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.

    There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.

    First, you need to build a kernel that supports both xen and an NFS-root – you’ll need a linux machine with a complete build environment, i.e. gcc, make, etc. I have been through this process with 2.6.31, but I would guess that the latest stable kernel version available at http://kernel.org will work just fine.
    - Download the latest kernel source
    - unzip, cd into source directory
    - Copy in attached .config as starting point ( copy as …./linux-(version)/.config )
    - make menuconfig

    Important config options are listed here:

    http://www.virtuatopia.com/index.php/CentOS_5_-_Build_a_Custom_Xen_Kernel_with_NFS_Root_Filesystem_(CONFIG_ROOT_NFS)_Support

    Most importantly, nfs client options and nfs_root need to be built into kernel (not as modules).  Also need to make sure to build Xen modules, and select most of the iptables filters (state is an important one). 

    - make
    - make modules_install

    Once the make and install is complete, you will have a kernel capable of nfs-booting under xen. To collect all the pieces needed:
    - copy over the file vmlinux in the base build directory (this one is ~85MB), this is the kernel (can’t use bzImage with xen nfs boot).
    - Tar up /lib/modules/kernelVer to distribute to nfs client vm.

    See the next article for creating a proper vm.cfg under xen / Oracle VM.

    No responses yet