Ruizs Place

Following up on my previous post about building a Xen / NFS-root kernel, this will take you through creating a VM template capable of using that kernel (or a P2V process for converting existing VM’s / linux servers to nfs-root). I think the intro from the last article still applies, so I’ll include it here:

Over the past year or two, we have transitioned all our servers and hosting to Xen – specifically, we use the Oracle VM management tools on top of xen, and most of our VM’s are Centos 5 x86_64. Since we use NFS NAS’ as storage across our infrastructure, it would be very convenient if we could use a NFS volume as the root drive for VM’s. With the NFS root, we gain things like easy use of filer snapshots, and on-the-fly volume resizing – if we’re ever running short on space in a given VM, its a single command (or a click on a web page) to expand the root drive.

The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.

There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.

Over the past year or two, we have transitioned all our servers and hosting to Xen – specifically, we use the Oracle VM management tools on top of xen, and most of our VM’s are Centos 5 x86_64. Since we use NFS NAS’ as storage across our infrastructure, it would be very convenient if we could use a NFS volume as the root drive for VM’s. With the NFS root, we gain things like easy use of filer snapshots, and on-the-fly volume resizing – if we’re ever running short on space in a given VM, its a single command (or a click on a web page) to expand the root drive.

The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.

There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.

First, you need to build a kernel that supports both xen and an NFS-root – you’ll need a linux machine with a complete build environment, i.e. gcc, make, etc. I have been through this process with 2.6.31, but I would guess that the latest stable kernel version available at http://kernel.org will work just fine.
- Download the latest kernel source
- unzip, cd into source directory
- Copy in attached .config as starting point ( copy as …./linux-(version)/.config )
- make menuconfig

Important config options are listed here:

http://www.virtuatopia.com/index.php/CentOS_5_-_Build_a_Custom_Xen_Kernel_with_NFS_Root_Filesystem_(CONFIG_ROOT_NFS)_Support

Most importantly, nfs client options and nfs_root need to be built into kernel (not as modules).  Also need to make sure to build Xen modules, and select most of the iptables filters (state is an important one). 

- make
- make modules_install

Once the make and install is complete, you will have a kernel capable of nfs-booting under xen. To collect all the pieces needed:
- copy over the file vmlinux in the base build directory (this one is ~85MB), this is the kernel (can’t use bzImage with xen nfs boot).
- Tar up /lib/modules/kernelVer to distribute to nfs client vm.

See the next article for creating a proper vm.cfg under xen / Oracle VM.

If you need to get in and edit some files on your xen domU instance, i.e. it isn’t booting up properly, etc, here’s how to mount it on dom0. In this case, we want to mount the second partion on the virtual disk (our root partition):

On and off over the past few years, I’ve looked for a good way to synchronize 2 or more iTunes libraries, and until recently, never found a good solution. The main problem is synchronizing library metadata – syncing files isn’t that difficult, but things like ratings, play counts, etc. is.

My previous solution was simple – I considered one computer my “master”, and simply sync’ed all the files from it to my other computers. Since I do most of my itunes listening (and rating) at work, I used my iMac as the master and my macbook and home desktop were the slaves. This has its downfalls though, as anything I added or rated on my other systems never made its way back to the master system, and I had to make those changes manually.
I decided to search again a few days ago, and found two applications that do exactly what I want:

myTuneSync from SocketHead studios
Syncopation from Sonzea

I checked them both out, and they both work as advertised (both offer free time-limited trials). You install the application, point it at the other itunes library to keep in sync with, and set them up for automatic updates. Syncopation is the clear winner for me, with its cleaner interface, faster performance, and ability to synchronize file deletions as well. Syncopation works only on macs, so if you have a mix of mac and Windows (or only Windows), myTuneSync is for you.

Interestingly, I found these programs two days before the new iTunes 9 was released, and looking at the keynote, I wondered if I just wasted money on something that was now a standard feature of iTunes. (Unfortunately) It turns out that the syncing included with iTunes is limited to purchases from the iTunes store, and doesn’t include anything added by you. It also seems that the metadata does not get transferred over, only the purchased song files.

I develop a product at work that (among many other things) allows users to easily create and manage databases to interact with other packages on their appliance. One feature that was requested and green-lighted is the ability to create a read-only user for existing databases- a user that can connect to a given database and access all tables and views in that DB, and nothing else.

Since PostgreSQL is our database of choice, I started researching the process, thinking it would be a couple mysql-esqe GRANT statements and that would be it. Turns out that it is a huge PITA in postgres – even my solution I’m documenting here has its shortcomings, but as far as I know is the best / only way to accomplish this task. I ran across quite a few sites that helped with pieces of this, but none that actually tied the process together for production usage. I am by no means a Postgres expert, but do have a good bit of experience mucking around with back-end settings and figuring how to script common tasks.

There are two main things to watch out for when trying to create a read-only user in PostgreSQL, especially if you come from a DB like MySQL:

For this example, we’ll use database name “mydb”, database user/owner “mydbuser”, and we’ll create a read-only user named “mydbuser_ro”. This assumes that you did not define a schema for your database and are using the default “public” schema.

Once complete, you can verify the settings with a quick sql query. You should see something like this, with the user=UC (for Usage/Create), and user_ro=U (for Usage):
mydb=> select * from pg_namespace where nspname='public';
nspname | nspowner | nspacl
---------+----------+------------------------------------------------------------------
public | 10 |
{postgres=UC/postgres,mydbuser=UC/postgres,mydbuser_ro=U/postgres}


I previously posted about a rather convoluted method to use mobileme and idisk to sync adium logs across computers, but after using that for a while, it turned into an exercise in frustration, as it was not reliable, and idisk is horribly slow.
I just came across a new solution to the same problem that is very simple, costs nothing, and performs great. Basically, let dropbox do the syncing for you. Instructions as follows:
- Sign up, download and install dropbox here
- Shutdown adium (or other chat client)
- Open a Terminal session
- Move your adium logs folder to dropbox:
mv ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs ~/Dropbox/Private/AdiumLogs
-create a symbolic link from your adium folder to your dropbox private folder:
ln -s ~/Dropbox/Private/AdiumLogs ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs

On any other computer you’d like to keep in sync with:
- sync over your existing logs with rsync:
rsync -avl ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs/ ~/Dropbox/Private/AdiumLogs
- Move your old logs folder out of the way:
mv ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs.old
-create a symbolic link from your adium folder to your dropbox private folder:
ln -s ~/Dropbox/Private/AdiumLogs ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs

It really is that simple, and works great – thanks Dropbox for a great bit of software.

At work, we are moving from a huge datacenter space (about 12′ x 12′, 12 cabinets) to less than 1 cabinet. This will be a great change in all respects, mostly financially, for all the saved space, but also environmentally, for all the saved power. We probably use about the same amount of power to run our whole infrastructure as we used to to run two or three servers.

We are using an IBM Bladecenter, with fully loaded blades – 5 for now, expandable to 14. Each blade has 8 CPU cores, and 32GB of memory. It would be a waste to run a server OS on the blade itself, it would definitely not be properly utilized and we’d be wasting compute power, electrical power, and space. So, we’re moving ahead with deploying everything on Oracle VM Server – a Xen based virtualization platform, that does some cool stuff like load balancing, failover, live migration, etc.

In our old infrastructure, we have a few Windows servers, whose functionality we need to keep around, but we’re not too excited to move over these old, Pentium III waste-of-space servers. Last week my main task was to figure out how to get these old servers moved into Xen virtual machines.

There’s not much information out there on making this conversion – there are some commercial tools that will do everything for you, but they are quite expensive. The included P2V functionality in Oracle VM is documented only for linux, and for systems with large hard drives with lots of free space, is pretty wasteful.

The process I came up with requires no downtime on the server to be cloned, only minimal changes beforehand (installing the VMware converter software package), and allows you to resize hard disks as needed. Once you get the hard disk cloned, you change the drivers, installed software, etc. on the VM only, so the original machine still works and is your backup/fail safe. The process isn’t all that time consuming either – it mostly depends on your network and disk speed to clone the image, and convert it.

The basic process is as follows:
- Clone the physical server using VMware converter – you need to install the VMware converter software on a management host, and the agent on the physical server to clone. You need to provide proper login credentials, and a samba/cifs share that both machines can access. Full instructions are available here:
http://www.howtoforge.com/vmware_converter_windows_linux

- Once you have the vmware image (.vmdk and .vmx files) open them in vmware server, fusion, or whatever version of VMware you have install / converted to.
- Install vmware tools to make your life easier.
- Just in case, install the Windows recovery console: http://support.microsoft.com/kb/307654
- Run MergeIDE: MergeIDE — This is the secret sauce that copies in and installs the proper basic drivers in the Windows registry.
- Shutdown the VM.
- Copy vmdk files over to Oracle VM server (or Xen flavor of choice)
- Create a new windows VM in Oracle VM manager, power off after creating
- Delete the created disk file (System.img)
- Convert the vmware images with qemu-img: qemu-img convert vmwarefile.vmx System.img
- Start vm in Oracle VM, let it detect devices and install drivers
- Use and enjoy being unbound from crappy old hardware!

If you’re like me, you just brought home guitar hero 4, and want to play the songs listed on the box. Unfortunately, it comes with only half of the (or so) available to play. I found this elsewhere, although its not too obvious that this is the code you’re looking for on other sites.

To unlock all songs:
Blue, Blue, Red, Green, Green, Blue, Blue, Yellow.

This was tested on my Wii, and should work on all other consoles as well. You get here by going to Options->Cheats->Enter new code. Once you enter it, you have to find it in the list, and turn it “on”. I.e., if you just enter the code, the cheat is by default off.

Other cheat codes can be found here:
http://gosublogger.com/2008/10/guitar-hero-world-tour-cheats-for-xbox-360-playstation2-playstation3/

Like email, my IM chat logs have become a critical reference and database for me. With both OS X’ spotlight and Adium’s chat transcripts, searching for a conversation in these logs is quite easy. However, not so much so when I’m on my laptop at home, and the chat took place on my work computer. I am looking into services like simkl.com, which act as a proxy and store your chat logs an their servers – but, these don’t have the search interface I’m used to, and I’m not sure I’m ready for yet another company to have access to my personal communications.

So, here’s the procedure I am using to sync my logs using my iDisk included with my MobileMe subscription. It is pretty rudimentary, but works nicely for what I want it to do.

Here are the relevant references I used when putting this process together, as I didn’t feel like spending much time on this:

http://www.macosxhints.com/article.php?story=2008071710372562

http://www.macosxhints.com/article.php?story=20020207214002198&query=dd

http://forums.macosxhints.com/showthread.php?t=2185

http://www.bombich.com/mactips/rsync.html

http://lingon.sourceforge.net/

First, I built the latest rsync based on the link above – not that its really needed, but I figured it wasn’t a bad idea to have the latest, greatest, and most efficient.

Second, you need the actual script, here’s mine, with some names removed:

#!/bin/bash

exit 0


Now, I created a new job in Lingon to run the sync script. I named it com.myname.idisk.rsync, selected the script I had created (saved in ~/bin), and set it to run every 2 hours. After logging out and back in, everything was up and running. Status can be checked in the log file – after a long initial run, things run very quickly.

You’ll want to repeat this process on any other computer you have for it to work properly. After spotlight reads in all the new files, you should have searchable chat logs on all your macs.

Notes: You’ll need to create the appropriate directories here (idiskmount), and the funky webdav password file. This process is detailed in the macosxhints forum post, it involves typing a few non-std characters in your editor of choice.

Download/Install UseMetaKeys FireFox Extension

In celebration of the release of Firefox 3 yesterday (and since I had forgotten to install my extension previously), I am posting this quick, dirty, and oh so helpful firefox extension.
Firefox has long been my preferred browser, although for the last year or so Safari had been gaining ground. Firefox 2 was just to bloated and slow, especially on OS X, and Safari was much faster. I always had a hard time choosing between the additional functionality Firefox provides, and the simplicity and speed of Safari. With Firefox 3, that’s pretty much over, and it is back in its place as my primary browser.
However, one thing that has always bugged me about firefox on Mac OS X was that you can’t open a URL in a new tab using Command+Enter, it only works with Option(alt)+Enter. I couldn’t find a fix a year or two ago, and with some quick searching today, I still did not find a good way to reassign the key sequence. So, I looked up a couple howto’s, found a sample/donor project, and whipped up a simple extension with a single purpose – remap Command-Enter to open a url in a new tab (i.e. when typing in the address or search bars). As expected, no support is provided, no warranty intended, etc., but if you’re using Firefox on OS X, I highly suggest installing this.

For some reason, I called it “UseMetaKeys”, and now, a year later, I am too lazy to change the name.

At work, we have a kickstart setup we have been using for a couple years now, with probably 150 servers out in the field based on this install. Our distro of choice is CentOS, a RedHat clone, and we are at version 4.4. This is out of date now, but it still works great for our needs, as security fixes are regularly back-ported. It would also be a major pain to upgrade our existing installations, and/or support multiple OS versions.

On to the issue at hand: we recently received some new server models that we’ll be supporting, both which have hardware not supported in CentOS 4.4. One machine has a RealTek RTL-8110 ethernet chip, and the other as a 3Ware 9650SE Raid controller. As I later discovered, this presented two unique problems with the kickstart – without the proper storage controller driver, one server didn’t find any disk to install on, and without the proper network driver, the other server couldn’t even connect to our kickstart server at all.

So, as you might guess, there are two different solutions here. The more elegant is for the storage controller, we can create a driver disk with the proper drivers, and make it available on the network during the kickstart. The network driver is more difficult – we need to insert it into the initrd image we provide for PXE boot, and then somehow copy it over after installation (this is an updated driver, r8169.ko, that exists in CentOS 4.4 but doesn’t support our newer card).

Adding a RAID/Storage Card Driver to the Kickstart:
For the driver disk, things are especially easy, as 3Ware provides a driver-disk compatible download, although not yet in the correct format to share over the network.

The driver provided by 3ware (http://www.3ware.com/KB/article.aspx?id=14546 ) includes the following:

-rwxr-xr-x 1 stever stever 66B Oct 10 2007 modinfo*
-rwxr-xr-x 1 stever stever 249B Oct 10 2007 modules.alias*
-rw-r--r-- 1 stever stever 377K Oct 10 2007 modules.cgz
-rwxr-xr-x 1 stever stever 28B Oct 10 2007 modules.dep*
-rwxr-xr-x 1 stever stever 463B Oct 10 2007 modules.pcimap*
-rwxr-xr-x 1 stever stever 192B Oct 10 2007 pci.ids*
-rwxr-xr-x 1 stever stever 339B Oct 10 2007 pcitable*
-rwxr-xr-x 1 stever stever 37B Oct 10 2007 rhdd*

This is all you need on a driver disk, so all you need to do is create a disk image, and copy these files over:

#Create a blank, 20MB image
dd if=/dev/zero of=/root/driverdisk.img bs=1M count=2
#Format the image with ext2
mkfs -t ext2 -q /root/driverdisk.img
#mount it and copy the files over
mount -o loop /root/driverdisk.img /mnt/tmp
cp /root/3ware/* /mnt/tmp/
umount /mnt/tmp

Now, copy the image over to somewhere accesible on kickstart, and update your ks.cfg with the following:
driverdisk --source=nfs:servername:/vol/kickstart/CentOS-4.4-x86/drivers/driverdisk.img

On network kickstart, anaconda should grab the driver, load it, and proceed normally. This should work for any non-network-card driver you need.

Adding a Network Card Driver to the Kickstart:
This is considerably more arduous, but not too difficult with the magic commands. Much of the information here comes from my friend Steve, www.kehlet.cx.

There is no nicely package/built driver provided by RealTek, just some source code with instructions for compiling.

I downloaded the driver here:

http://wiki.centos.org/HardwareList/RealTekr1000

After untar’ing unzip’ing, I ran make with the default settings, and manually changed the kernel version to build a smp driver as well (assuming you’re building on a single-cpu system):

[root@lb4 ~]# cd r8169-6.006.00/
[root@lb4 ~]# make
[root@lb4 ~]# mv src/r8169.ko r8169.ko.2.6.9-42.EL
[root@lb4 ~]# make clean
(edit src/Makefile, change the line "KVER := $(shell uname -r)" to "KVER := 2.6.9-42.ELsmp"
[root@lb4 ~]# make
[root@lb4 ~]# mv src/r8169.ko r8169.ko.2.6.9-42.ELsmp


Now you should have two .ko module files compatible with the different kernels – we need to get these inserted into the initrd image. An initrd is basically a disk image that holds various drivers and programs needed to pre-boot your system. It is usually a gzipped disk image file, so its nothing too special. Basically, you need to unzip & mount the initrd image, gunzip/cpio the modules.cgz file in the initrd, make the required changes, and package everything back up.

Here’s those steps in gory detail:

mkdir /mnt/tmp
mkdir /mnt/initrd
mkdir /var/tmp/work
mkdir /var/tmp/work/bootnet
mkdir /var/tmp/work/drvnet
gunzip < /root/tftpboot/initrd.img > /var/tmp/work/bootnet/initrd.img.ungzipped
cd /var/tmp/work/bootnet/
mount -o loop initrd.img.ungzipped /mnt/tmp2
cd /mnt/tmp2/modules
gunzip < modules.cgz | (cd /var/tmp/work/bootnet && cpio -idv)
cd /var/tmp/work/bootnet/2.6.9-42.EL/i686
cp /root/r8169-6.006.00/r8169.ko.2.6.9-42.EL r8169.ko
cd /var/tmp/work/bootnet/2.6.9-42.ELsmp/i686
cp /root/r8169-6.006.00/r8169.ko.2.6.9-42.ELsmp r8169.ko
cd /var/tmp/work/bootnet/
find 2.6.9-42.EL | cpio -ov -H crc | gzip > /mnt/tmp2/modules/modules.cgz
#edit /mnt/initrd/modules/pcitable
#add this:
0x10ec 0x8167 "r8169" "Realtek|RTL-8110 Gigabit Ethernet"
umount /mnt/initrd
gzip < initrd.img.ungzipped > initrd.r8169.img


I had to boot up DSL, run lspci & lspci -n, to get the ID to put in here – third column has 10ec:8167, which is what we need

https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2007-September/027142.html

So now you can replace your initrd.img with the one you just created. The kickstart should work fine now, but upon reboot, the system will not be able to find the right driver. After the kickstart, you need to copy over the .ko files to the appropriate directories – we added a line in our post-install script to do this for us, it simply copies the .ko file to the appropriate directory (/lib/modules/`uname -r`/kernel/drivers/net/)

Hopefully this is useful to someone, I couldn’t find a good, comprehensive guide on how to do this, I had to pull data from a bunch of different sources.

I upgraded my home MacBook and work iMac to Leopard right when it came out. When I had Tiger, I installed an extention to Mail.app called “IMAP-IDLE”, which added support for the (logically named) IMAP Idle mode. This basically means that it will open an extra connection to your mail server, and when the mail server receives a new message, it notifies your client almost instantly. At work we have a Microsoft Exchange server, and for personal email I use gmail, and both were lightning fast – when I tested at work, I would see the new email in my inbox within a second of sending.

However, this behavior stopped when I upgraded to Leopard – it includes a built-in option to use IMAP IDLE, but it never seemed to work right – I would receive email only when I hit the “Get Mail” button, or my client checked on its specified interval.

I thought I had figured out how to make it work properly, it seems there are two settings that should impact this, but neither work well. Under Mail.app Preferences, under the Accounts Section, Advanced tab, if you select “Use IDLE command if the server supports it“, and you un-check “Include when automatically checking for new mail“, things work ok for a while.  After making the change, restart Mail.app.  However, if something happens, i.e. you close the lid on your laptop, lose the connection to the server, etc., it no longer works.

Why can’t apple get this working as well as the plugin was before?  No good solution at this point…