Ruizs Place

Following up on my previous post about building a Xen / NFS-root kernel, this will take you through creating a VM template capable of using that kernel (or a P2V process for converting existing VM’s / linux servers to nfs-root). I think the intro from the last article still applies, so I’ll include it here:

Over the past year or two, we have transitioned all our servers and hosting to Xen – specifically, we use the Oracle VM management tools on top of xen, and most of our VM’s are Centos 5 x86_64. Since we use NFS NAS’ as storage across our infrastructure, it would be very convenient if we could use a NFS volume as the root drive for VM’s. With the NFS root, we gain things like easy use of filer snapshots, and on-the-fly volume resizing – if we’re ever running short on space in a given VM, its a single command (or a click on a web page) to expand the root drive.

The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.

There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.

Install linux / kickstart / etc to get a good, minimal install of linux, and configure it as you would any linux server. I would suggest a minimal install for your requirements, turn off any unneeded services, lock down permissions, firewall, and any other deployment process you usually go through. Since this is going to be a template, you want this to be ready for any task with minimal configuration changes.

On that configured linux system, mount a NFS share that will serve as your future NFS root (or storage for your VM template) to /mnt/tmp

Shut down as many running services as possible, so there are no file conflicts/open databases/etc.

Copy over the kernel modules from your nfs-root kernel build:rsync -av /path/to/kernel/modules/kernel-ver /mnt/tmp/lib/modules/

#copy all files over to nfs mount
cd /
cp -ax /{bin,dev,etc,lib,lib64,opt,root,sbin,usr,var,folders} /mnt/tmp
mkdir /mnt/tmp/{home,proc,sys,tmp}
chmod 777 /mnt/tmp/tmp

Edit /mnt/tmp/etc/fstab to look something like this, main change here is to /:

/dev/nfs                /                       rootfs  defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
rpc_pipefs              /var/lib/nfs/rpc_pipefs rpc_pipefs defaults     0 0


Edit /mnt/tmp/etc/sysconfig/network-scripts/ifcfg-eth0 as needed to setup the network

Create a vm.cfg file for the Xen vm. If using this is an Oracle VM template, make sure to add a zero-byte disk image, and put this under the seed_pool directory. Edit options as needed for your infrastructure.
dhcp = 'off'
extra = 'nfsroot=10.0.0.1:/vol/nfsroot,noacl,nfsvers=3,tcp,rsize=32768,wsize=32768 selinux=0 acpi=off noapic'
gateway = '10.0.0.254'
hostname = 'nfsroot-vm.mycompany.com'
ip = '10.0.0.10'
kernel = '/OVS/P2V_STUFF/nfsboot-custom/vmlinuz-2.6.31.1nfsboot'
memory = '512'
name = 'nfsroot-vm'
netmask = '255.255.255.0'
on_crash = 'restart'
on_reboot = 'restart'
root = '/dev/nfs'
uuid = '9af0a816-0123-4567-ad50-bc32be92bff7'
vcpus = 2
vfb = ['type=vnc,vncunused=1,vnclisten=0.0.0.0,vncpasswd=password']
vif = ['bridge=vlan100,mac=00:16:3E:31:FF:37,type=netfront']
vif_other_config = []


Start up the vm with Oracle VM manager, or start on the command line with xm create vm.cfg, and cross your fingers!

Over the past year or two, we have transitioned all our servers and hosting to Xen – specifically, we use the Oracle VM management tools on top of xen, and most of our VM’s are Centos 5 x86_64. Since we use NFS NAS’ as storage across our infrastructure, it would be very convenient if we could use a NFS volume as the root drive for VM’s. With the NFS root, we gain things like easy use of filer snapshots, and on-the-fly volume resizing – if we’re ever running short on space in a given VM, its a single command (or a click on a web page) to expand the root drive.

The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.

There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.

First, you need to build a kernel that supports both xen and an NFS-root – you’ll need a linux machine with a complete build environment, i.e. gcc, make, etc. I have been through this process with 2.6.31, but I would guess that the latest stable kernel version available at http://kernel.org will work just fine.
- Download the latest kernel source
- unzip, cd into source directory
- Copy in attached .config as starting point ( copy as …./linux-(version)/.config )
- make menuconfig

Important config options are listed here:

http://www.virtuatopia.com/index.php/CentOS_5_-_Build_a_Custom_Xen_Kernel_with_NFS_Root_Filesystem_(CONFIG_ROOT_NFS)_Support

Most importantly, nfs client options and nfs_root need to be built into kernel (not as modules).  Also need to make sure to build Xen modules, and select most of the iptables filters (state is an important one). 

- make
- make modules_install

Once the make and install is complete, you will have a kernel capable of nfs-booting under xen. To collect all the pieces needed:
- copy over the file vmlinux in the base build directory (this one is ~85MB), this is the kernel (can’t use bzImage with xen nfs boot).
- Tar up /lib/modules/kernelVer to distribute to nfs client vm.

See the next article for creating a proper vm.cfg under xen / Oracle VM.

If you need to get in and edit some files on your xen domU instance, i.e. it isn’t booting up properly, etc, here’s how to mount it on dom0. In this case, we want to mount the second partion on the virtual disk (our root partition):

Print out the partition layout:
fdisk -l /path/to/img/file.img

Disk System.img: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
System.img1 1 4 32098+ 83 Linux
System.img2 * 5 619 4939987+ 83 Linux
System.img3 620 750 1052257+ 82 Linux swap / Solaris

create directory to mount image under: mkdir /mnt/tmp

mount the image using the offset from the fdisk output, the start block (in this case 5)
mount mount -o loop,offset=$((512*5) /path/to/img/file.img /mnt/tmp

Edit files as needed, and unmount when done.

When I first brought home my Gaggia Baby espresso machine, it had been used for a number of years, and not cleaned very well. I tried a few shots from it, but no matter what I did, they always came through too fast. When I adjusted my grinder to the point that the burrs were touching, the espresso would come out slowly at first (and look good), and get gradually faster until the shot was done after about 15 seconds. If you are having these issues too, its time to take a close look at the brew pressure of your machine.

The OPV, or over-pressure valve (also known as pressure relief valve), is fitted to most mid-level home machines with vibratory pumps. Vibe pumps aren’t really precision adjustable devices, so they are basically either on or off, and inside an espresso machine, whatever pressure they’re operating at is the pressure you’re getting through your espresso. In most espresso machine marketing literature, they advertise the power of pumps, 16 bar in this one, 18 bar in that one, but this really doesn’t matter – espresso needs about 9 bar, give or take, to be brewed properly, and this is where the OPV comes in. As the pressure raises, it will gradually open up to maintain a set pressure, usually about 9 bar in an espresso machine. Excess water is routed back to the reservoir, and the group head, and thus your espresso puck) see’s the proper pressure. Gaggia Baby and Classic models are fitted with an adjustable OPV, other models don’t have one, but if you can find one its a great upgrade for a Carezza or Espresso.

As you can read in some other posts here, the Gaggia machines are very easy to take apart and repair/clean, so while going through that process, I spent extra time on cleaning / restoring the OPV valve. It turns out that mine was completely sealed shut from past years’ scale deposits, and it wasn’t opening at all. I took the valve apart completely, and soaked it in a durgol bath and in a citric acid bath about 5 or 6 times – there was a lot of crud on there, and it took a while to get it off. Afterwards, things looked pretty good, and it was very easy to put back together.

When my machine re-assembly was complete, I hooked up my newly built pressure gauge, and proceeded to dial in the correct pressure. This is a bit complicated because the valve is not easily accessible, and adjustment requires disassembly of the valve. So its turn on machine, read pressure, turn off, unscrew valve, adjust (while not burning hands on hot boiler), reassemble, and repeat. It took 4 or 5 iterations to get it the way I wanted, I was moving about 1/2 turn of the adjustment nut each time. I think these come from the factory set very high, so I’m guessing you could improve your results just by loosening the nut a turn or two from factory tight. Espresso is now much easer to make (and much better tasting), and with the pressure gauge, I know any problems are my fault, and not the machine’s.

Details of the pressure gauge: I had a friend weld a piece of stainless tubing to a blank filter basket (could probably get this done at a welding shop for $10 and a six pack as well), and attached to a tee, a needle valve, and a liquid-filled pressure gauge. I can adjust the flow to approximate espresso flow rates, and dial in the pressure from there.

I recently purchased an espresso machine to fuel my addiction – a Gaggia Baby circa mid 1990’s. You might wonder why I bought such an old machine, but it was a smoking deal – I got the machine and the grinder for a super-low price, and then sold the grinder on ebay to basically pay for both. So, it was worth it to put some time and a little money into making it like new again.

Gaggia home machines are pretty ubiquitous, as the internals have been the same for 20 years or so, with only cosmetic changes outside – so parts and instructions aren’t too difficult to find. My particular model is a “Baby”, which includes the 3-way valve and an adjustable OPV (over-pressure or pressure relief) valve, otherwise all models are basically the same. Being about 15 years old, it was showing some wear and tear, and was also showing that it hadn’t been cleaned in a LONG time. Before I noticed how much gunk was built up on it, I tried to pull a few shots – they all came out way too quickly, even with the grinder burrs basically touching.

So, I did what I usually do, and took the whole thing apart . Gaggia machines are very easy to tear down, all it takes is a couple screwdrivers and allen wrenches. When you do take these machines apart, just be sure to label all the wires and take some photos to reference when re-assembling – a little masking tape and a sharpie can go a long way.

Once everything was apart, I started cleaning. The group (part the holds the portafilter) needed a ton of cleaning and scraping, there was years of coffee gunk in there. All parts got a few soaks in descaling solution and citric acid baths, there was quite a bit of scale on everything. It took about 5 soaks to get everything cleaned up, and a bit of scrubbing inside the boiler as well. I purchased a complete Gaggia rebuild kit on ebay from “supercoffeeman”, which includes all seals and gaskets needed to make the machine like new. For an older machine, its safe just to replace everything, for a newer machine all you may need to replace is the group gasket. In any case, if the rubber is brittle or cracked, its time to replace- mine certainly was. In subsequent posts, I’ll go over the details of the mods and adjustments I went through to get it dialed in pefectly…