Searching for this issue on the net, I found that this is usually caused by a corrupt row in the database. The solution is to find the corrupt row, and delete it (and/or update it with proper, non-corrupt values).

Unfortunately, there is no easy way to find the row that is giving you grief (or even the table if you don’t know what is causing this error). You basically have to narrow down your search by looking at parts of the database. If you have some sort of date or timestamp column in your table, you can narrow it down a lot more easily, if you know approximately when the error started:


mydb=# select id from mytable where date_created > timestamp '2009-02-27 00:00:00' and date_created < timestamp '2009-03-01 23:59:00';
id
--------------------------------------
3ab226cf-a972-463d-b5a1-148fe39672b5
10daca73-b2b3-470c-a258-5c92d21cfbb6
(2 rows)


Luckily it's only two rows, and the first one is the culprit:

mydb=# select * from mytable where id='3ab226cf-a972-463d-b5a1-148fe39672b5';
ERROR: missing chunk number 0 for toast value 87246446


Now, on to delete it:
mydb=# delete from mytable where id='3ab226cf-a972-463d-b5a1-148fe39672b5';

If you don’t have a date/timestamp column in your table, you could use LIMIT and OFFSET to narrow it down. I.e. a series of statements like (increasing the offset each time):

select * from mytable order by id limit 1000 offset 5000;


The migration itself was pretty easy, but I wanted to gain some of bash’s features, specifically the emac-like keystrokes for moving forward/back a word in the command line, and also to the beginning and end of the line. The keystrokes, among others, are:

ctrl-a Move cursor to beginning of line
ctrl-e Move cursor to end of line
meta-b Move cursor back one word
meta-f Move cursor forward one word
ctrl-w Cut the last word
ctrl-u Cut everything before the cursor
ctrl-k Cut everything after the cursor
ctrl-y Paste the last thing to be cut
ctrl-_ Undo


I’m a big fan of iTerm – I use it as my primary terminal and have been for the last few years, as it matured in terms of performance and has a lot of great features. The most compelling feature it has over the built in OSX Terminal.app, for me, is that you can specify which characters to include when selecting words. A common operation in a terminal is to select a path, i.e. /home/myuser/testfile. In Terminal.app, if you double click on home, it will just select home – so you have to go back, double click on the first “/” character, and mouse right to get the whole word. With iTerm, you can set a preference with characters to include when connecting, i.e. “/”, so when I double click on home in iTerm, it selects the full path. Specifically, in iTerm -> Preferences -> Mouse, i have “Characters considered part of word:” set to /-_.

One thing I couldn’t figure out was how to get the “meta” key in iterm set to the alt/option key on the mac keyboard. Turns out it’s buried in a non-intuitive place, but it can be done. In iTerm, go to Bookmarks->Manage Profiles, and under Keyboard Profiles select Global. On the right side, you need to set Option Key as “+Esc” for this to work – yes, its strange that it has to be Esc instead of Meta, but hey, at least it all works.

To set up OracleVM 2.2 Server, with a NFS repository for VM’s:

• Download the ISO, and run through the install process
• Once it is up and you are logged in:
• Download and install the updated ovs-agent rpm from oracle here: http://oss.oracle.com/oraclevm/server/2.2/RPMS/ovs-agent-2.3-31.noarch.rpm
  • wget http://oss.oracle.com/oraclevm/server/2.2/RPMS/ovs-agent-2.3-31.noarch.rpm
  • rpm -Uvh ovs-agent-2.3-31.noarch.rpm
• Add your repository with the repos.py tool, make it the root repository, and initialize it:
  
# /opt/ovs-agent-2.3/utils/repos.py --new mynfsserver.company.com:/volumes/datavol/ovs
# /opt/ovs-agent-2.3/utils/repos.py -l
# /opt/ovs-agent-2.3/utils/repos.py -r 7c885f7e-7dfe-4108-bae4-4dc329e2f017
# /opt/ovs-agent-2.3/utils/repos.py -i


At this point, you are ready to go, and can continue setting things up (including your Oracle VM Manager VM).

Over the past year or two, we have transitioned all our servers and hosting to Xen – specifically, we use the Oracle VM management tools on top of xen, and most of our VM’s are Centos 5 x86_64. Since we use NFS NAS’ as storage across our infrastructure, it would be very convenient if we could use a NFS volume as the root drive for VM’s. With the NFS root, we gain things like easy use of filer snapshots, and on-the-fly volume resizing – if we’re ever running short on space in a given VM, its a single command (or a click on a web page) to expand the root drive.

The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.

There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.

The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.

There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.

First, you need to build a kernel that supports both xen and an NFS-root – you’ll need a linux machine with a complete build environment, i.e. gcc, make, etc. I have been through this process with 2.6.31, but I would guess that the latest stable kernel version available at http://kernel.org will work just fine.
- Download the latest kernel source
- unzip, cd into source directory
- Copy in attached .config as starting point ( copy as …./linux-(version)/.config )
- make menuconfig

Important config options are listed here:

http://www.virtuatopia.com/index.php/CentOS_5_-_Build_a_Custom_Xen_Kernel_with_NFS_Root_Filesystem_(CONFIG_ROOT_NFS)_Support

Most importantly, nfs client options and nfs_root need to be built into kernel (not as modules).  Also need to make sure to build Xen modules, and select most of the iptables filters (state is an important one). 

- make
- make modules_install

Once the make and install is complete, you will have a kernel capable of nfs-booting under xen. To collect all the pieces needed:
- copy over the file vmlinux in the base build directory (this one is ~85MB), this is the kernel (can’t use bzImage with xen nfs boot).
- Tar up /lib/modules/kernelVer to distribute to nfs client vm.

See the next article for creating a proper vm.cfg under xen / Oracle VM.

My previous solution was simple – I considered one computer my “master”, and simply sync’ed all the files from it to my other computers. Since I do most of my itunes listening (and rating) at work, I used my iMac as the master and my macbook and home desktop were the slaves. This has its downfalls though, as anything I added or rated on my other systems never made its way back to the master system, and I had to make those changes manually.
I decided to search again a few days ago, and found two applications that do exactly what I want:

myTuneSync from SocketHead studios
Syncopation from Sonzea

I checked them both out, and they both work as advertised (both offer free time-limited trials). You install the application, point it at the other itunes library to keep in sync with, and set them up for automatic updates. Syncopation is the clear winner for me, with its cleaner interface, faster performance, and ability to synchronize file deletions as well. Syncopation works only on macs, so if you have a mix of mac and Windows (or only Windows), myTuneSync is for you.

Interestingly, I found these programs two days before the new iTunes 9 was released, and looking at the keynote, I wondered if I just wasted money on something that was now a standard feature of iTunes. (Unfortunately) It turns out that the syncing included with iTunes is limited to purchases from the iTunes store, and doesn’t include anything added by you. It also seems that the metadata does not get transferred over, only the purchased song files.

Since PostgreSQL is our database of choice, I started researching the process, thinking it would be a couple mysql-esqe GRANT statements and that would be it. Turns out that it is a huge PITA in postgres – even my solution I’m documenting here has its shortcomings, but as far as I know is the best / only way to accomplish this task. I ran across quite a few sites that helped with pieces of this, but none that actually tied the process together for production usage. I am by no means a Postgres expert, but do have a good bit of experience mucking around with back-end settings and figuring how to script common tasks.

There are two main things to watch out for when trying to create a read-only user in PostgreSQL, especially if you come from a DB like MySQL:

For this example, we’ll use database name “mydb”, database user/owner “mydbuser”, and we’ll create a read-only user named “mydbuser_ro”. This assumes that you did not define a schema for your database and are using the default “public” schema.

Once complete, you can verify the settings with a quick sql query. You should see something like this, with the user=UC (for Usage/Create), and user_ro=U (for Usage):
mydb=> select * from pg_namespace where nspname='public';
nspname | nspowner | nspacl
---------+----------+------------------------------------------------------------------
public | 10 |
{postgres=UC/postgres,mydbuser=UC/postgres,mydbuser_ro=U/postgres}


On any other computer you’d like to keep in sync with:
- sync over your existing logs with rsync:
rsync -avl ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs/ ~/Dropbox/Private/AdiumLogs
- Move your old logs folder out of the way:
mv ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs.old
-create a symbolic link from your adium folder to your dropbox private folder:
ln -s ~/Dropbox/Private/AdiumLogs ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs

It really is that simple, and works great – thanks Dropbox for a great bit of software.

We are using an IBM Bladecenter, with fully loaded blades – 5 for now, expandable to 14. Each blade has 8 CPU cores, and 32GB of memory. It would be a waste to run a server OS on the blade itself, it would definitely not be properly utilized and we’d be wasting compute power, electrical power, and space. So, we’re moving ahead with deploying everything on Oracle VM Server – a Xen based virtualization platform, that does some cool stuff like load balancing, failover, live migration, etc.

In our old infrastructure, we have a few Windows servers, whose functionality we need to keep around, but we’re not too excited to move over these old, Pentium III waste-of-space servers. Last week my main task was to figure out how to get these old servers moved into Xen virtual machines.

There’s not much information out there on making this conversion – there are some commercial tools that will do everything for you, but they are quite expensive. The included P2V functionality in Oracle VM is documented only for linux, and for systems with large hard drives with lots of free space, is pretty wasteful.

The process I came up with requires no downtime on the server to be cloned, only minimal changes beforehand (installing the VMware converter software package), and allows you to resize hard disks as needed. Once you get the hard disk cloned, you change the drivers, installed software, etc. on the VM only, so the original machine still works and is your backup/fail safe. The process isn’t all that time consuming either – it mostly depends on your network and disk speed to clone the image, and convert it.

The basic process is as follows:
- Clone the physical server using VMware converter – you need to install the VMware converter software on a management host, and the agent on the physical server to clone. You need to provide proper login credentials, and a samba/cifs share that both machines can access. Full instructions are available here:
http://www.howtoforge.com/vmware_converter_windows_linux

- Once you have the vmware image (.vmdk and .vmx files) open them in vmware server, fusion, or whatever version of VMware you have install / converted to.
- Install vmware tools to make your life easier.
- Just in case, install the Windows recovery console: http://support.microsoft.com/kb/307654
- Run MergeIDE: MergeIDE — This is the secret sauce that copies in and installs the proper basic drivers in the Windows registry.
- Shutdown the VM.
- Copy vmdk files over to Oracle VM server (or Xen flavor of choice)
- Create a new windows VM in Oracle VM manager, power off after creating
- Delete the created disk file (System.img)
- Convert the vmware images with qemu-img: qemu-img convert vmwarefile.vmx System.img
- Start vm in Oracle VM, let it detect devices and install drivers
- Use and enjoy being unbound from crappy old hardware!