MacOS Xen: Snow Leopard as guest on a Xen domU

Some days ago I started trying to get MacOS X to run as a virtual machine on Xen. After all if the OSx86 guys are getting it to run on normal PC hardware, why not on virtual hardware?

There’s not much info available and most of them is incomplete. The more accurate sites I could find were:

Some notes:

  • I’m using Debian Squeeze with the bundled Xen 4.0 for the hypervisor and the dom0. The server si a cheap (~300€) PC with a DualCore E5700 @3Ghz and 4Gb of RAM. Other versions of Xen may need different options or have a slightly different config file syntax.
  • I assume you already know how to configure a Xen domU, access it through Xen’s built-in VNC server, etc.
  • Instead of using the original Snow Leopard DVD and then patching like here, I’m taking the easy road using iATKOS S3 v2 which is already upgraded to 10.6.3 and bundles several the OSx86 patches.

My /etc/xen/hackosx.cfg file is a mix of those on the previously mentioned pages:

kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
memory = 512
device_model='/usr/lib64/xen/bin/qemu-dm'

name = "hackosx"
#vcpus=1
pae=1
acpi=1
apic=1
#HPET=1
#timer_mode=0
#vpt_align=0
#vcpus_avail=1
#localtime=1
NE2000=0

vif = [ 'type=ioemu,ip=192.168.10.10,bridge=xenbr0' ]
disk = [
        'phy:/dev/mapper/xen-hackosx_hd,ioemu:hda,w',
        'file:/root/iATKOS_S3v2.iso,hdc:cdrom,r',
  ]

# first boot from disk, then from cd if that fails
boot="cd"
sdl=0
vnc=1
vnclisten="0.0.0.0"
vncdisplay=1
vncconsole=0
vncunused=1
vncpasswd='pass'

stdvga=1
serial='pty'
monitor=1

Fix the path to the HD image file/dev and the iATKOS ISO if you need.

So, let’s begin:

  • On the dom0 run “xen create /etc/xen/hackosx.cfg” and access it via VNC.
  • Run Disk Utility and partition the virtual HD. Return to the installer.
  • On the screen where you select the destination HD, click the customize button on the bottom left. Besides the default options (on the screenshot) I also selected the following (NOT on the screenshot and NEEDED for OS X to run on Xen):
    • Patches->Modified Kernels->qoopz 10.3.0
    • Drivers->Main Hardware->SATA/IDE->Intel SATA/IDE
    • Drivers->Main Hardware->PS/2->Apple PS/2
    • Drivers->Network->Wired->Realtek->RTL8139

    iATKOS default options

  • Continue with the installation.

The installer proceeds normally until the end. In my case it gets stuck in “Remaining time: 8 minutes approx.” so after a while seeing the progress bar not… well… progressing, I went to the dom0 and restarted the virtual machine. Note: I had to re-install several times until I got the right options and the installer hanged for me at that point always.

Despite the unfinished installation, access again the VM after restarting it with VNC. This time you’ll see the usual OS X 1st time wizard configuring the language, time zone, user account, etc. Yay!

Now, on to the upgrade to 10.6.6:

If everything worked as expected, now you’re running MacOS X Snow Leopard updated to 10.6.6 on a Xen domU. \o/

Some additional details:

  • Network performance with the default OS X RTL-8139 driver is TERRIBLE (at least right after installing iATKOS, before upgrading to 10.6.6). Follow these instructions to replace it.
  • If you go to apple menu on the upper left corner and then “About this Mac”, the Finder restarts and you don’t get the About window. :-/ Nevertheless the system is upgraded: ”uname -r” says 10.6.0 (same as in my MacBook Pro) and the App Store is installed.
  • Run Software Update. There’s a Safari update, a Java one and an iTunes one. ;-) (yes, Software Update works and these upgrades don’t break any of the MultiBeast patching).
  • Install VineServer (it’s free). Is a much better VNC server than the one integrated in Xen and more compatible/standard than the one bundled in OS X.
  • Disable power saving in System Preferences -> Energy Saver. When the virtual OS goes into power saving I don’t know how to wake it, you can VNC to it but it’s completely unresponsive. Maybe there’s a fix, iATKOS and/or MultiBeast have a fix for power saving (not for Xen of course, but maybe it helps) and Xen’s config seems to have some options for power management.

Enjoy. :-)

March 23th update: Apple released the MacOS X 10.6.7 upgrade yesterday. I’ve just tried to install it through Software Update and it works. No need to run MultiBeast even.

Xorg resolution on a Xen guest (domU)

  • english
  • spanish

When installing a Linux system as a Xen guest on a domU and activating the VNC access, hopefully everything runs fine except that you can’t change the resolution. No matter what you do (through the GUI, editing the xorg.conf file) it seems all it can do is 800×600.

After googling for a solution I stumbled upon this post which commented on a new extra=”xenfb.videomem=5,1024 “ option for dealing with guest video RAM (and thus, supported resolutions). But it didn’t worked out. Digging deeper on this approach I realized this “extra” Xen configuration was used for passing extra parameters to the guest’s kernel, so this “xenfb” should be a module on the guest OS. Then I looked for *xen* modules on my Ubuntu guest and found “xen-fbfront”:

$ modinfo xen-fbfront
filename:       /lib/modules/2.6.35-24-generic/kernel/drivers/video/xen-fbfront.ko
alias:          xen:vfb
license:        GPL
description:    Xen virtual framebuffer device frontend
srcversion:     0EB34742ACF8D2559403034
depends:        fb_sys_fops,syscopyarea,sysfillrect,sysimgblt
vermagic:       2.6.35-24-generic SMP mod_unload modversions
parm:           video:Video memory size in MB, width, height in pixels (default 2,800,600) (array of int)

So it looks like the module and the option got renamed along the way. Tried it, and it worked!

So, all in all, what you need to add is this line to your domU config file (/etc/xenDOMAIN.conf) on the dom0, adjusting memory and resolution parameters at will:

extra="xen_fbfront.video=4,1024,768 "

kpartx: Mounting the partitions in a disk image file

  • english
  • spanish

Sometimes it is useful to mount the partitions in a disk image file, e.g. from a backup (or dd) or a virtual machine, without having to use that backup or VM software. These files are usually a 1:1 copy of what should be on an actual disk, so the internal partition table format is the same the operating system would use on a disk and also the format of the partitions themselves are ext2/3/FAT/whatever. So the system should be able to use them, the only problem is that neither this image file nor its partitions are mapped to proper block device files in /dev.

Enter kpartx, a small utility that analyzes a disk image file, detects the partitions in it and creates device files for each partition, so that they can be mounted as if they were real partitions on a physical disk. Usage is very simple: just pass the file to it and kpartx shows the partition table. Add the -a parameter to that and kpartx creates the device files; -d and it deletes them. mount.

Kudos to this page where I learnt about kpartx, and which by the way explains a way to manually map the partitions using the loopback interface. 

tinydns por dominios

  • english
  • spanish

It’s no secret that I really like DJB‘s software. I think that qmail and djbdns are rock-solid pieces of software, something you can really rely on. But some people just don’t get them and bash them for whatever reasons.

Case in point, djbdns. Many a time I’ve heard the argument that having all tinydns’ domains in a single data file is a PITA. Well, that proves those people don’t get the point in DJB configuration files: you can edit them by hand, but their real power is that they’re easily scriptable! You don’t want having all the domains in a single file? Ok, split them and join them before compiling!

You can do this with the following makefile: split the domains, one per file on the “domains” directory, and work with those files, don’t ever touch the original “data” file again. And if you have a backup server put it’s IP address or FQDN on an env/BACKUP file. Then when you run “make” it will join all the files, compile them and sync the results with the backup server. Easy as pie, don’t you think?

all: data.cdb data: domains/*[^bB][^aA][^kK~] cat $+ > data data.cdb: data /usr/bin/tinydns-data [ -f ../env/BACKUP ] && rsync -azv * root@`cat ../env/BACKUP`:`pwd` clean: -rm -f *~ domains/*~

Depurando problemas con nginx

  • english
  • spanish

Last week I’ve been debugging a problem I had with this site’s nginx server: from time to time it hanged and I had to restart the process. Some time ago I wrote a little script that checked if it was running OK and restarted it otherwise, but anyway that wasn’t a real solution.

So I spent some days really looking into it and asking for support and reporting my findings to the nginx mailing list. One useful tip I got there was enabling the “debug” mode on the error log, which shows full traces of the processes (including their PID) as they’re processing the request, the rewrites, upstreams, etc.

error_log /var/log/nginx/$host-error.log debug;

With this extended log and the PID of the process malfunctioning, it’s quite easy finding out what that process was doing right before hanging. In order to find out the PID of the hanged processes, I extended my check-reboot script to log some generic system metrics right before restarting nginx: netstat -nap (which shows the PID), ps, vmstat, etc.

#!/bin/sh

TIMEOUT=20
CHECK=http://localhost/wp-admin/
LOG=/var/log/checkWeb/checkWeb-$(date +%Y%m%d).log
LOGR=/var/log/checkWeb/restart-$(date +%Y%m%d).log
TMP=/tmp/checkWeb-$RANDOM

if ! wget -t 1 -o /dev/null -O /dev/nul -T $TIMEOUT $CHECK
then
echo "ERROR, restarting nginx"
echo "** RESTARTING **" >> $TMP
date >> $TMP
echo "- CLOSE_WAIT:" >> $TMP
netstat -nap | grep -c CLOSE_WAIT >> $TMP
echo "- vmstat" >> $TMP
vmstat 1 5 >> $TMP
echo "- free" >> $TMP
free >> $TMP
echo "- ps" >> $TMP
ps aux >> $TMP
echo "- netstat" >> $TMP
netstat -nap >> $TMP
echo "" >> $TMP
echo "" >> $TMP

#       pkill -9 -f php-cgi
pkill -9 -f nginx
sleep 1s
/etc/init.d/nginx start

cat $TMP
cat $TMP >> $LOG
date >> $LOGR
fi

rm -rf $TMP

This way, each time localhost/wp-admin was unresponsive (I was debugging a WP site), besides restarting nginx I was getting a lot of system info. With time I got to realize that nginx processes were not actually hanging, but some of their sockets got on the CLOSE_WAIT state forever until the process was restarted. Looking for the PID of those processes according to netstat on the error log, the last request they were processing before getting to the CLOSE_WAIT state was always the same: on my blog I have some examples of how running servers with daemontools; daemontools uses named pipes (FIFOs), which can become kind of black holes if there’s no process feeding them; when nginx hit one of these FIFOs, it hanged.

Funny thing is that I never had this problem with either Apache nor lighttpd. But anyway the problem is not nginx but those FIFOs which shouldn’t really be there. I removed them and have had no hanged processes in five days, while before this nginx was restarting 3-4 times a day.

find -empty,borrar directorios vacíos

  • english
  • spanish

The find command has an -empty option which detects empty directories of files (0bytes). This is useful if you need to locate or remove empty directories or files from a directory structure:

find /path/ -type d -empty -exec rmdir {} \;
find /path/ -type f -empty

Pasar URL a minúsculas con nginx

  • english
  • spanish

With nginx and its embedded Perl module it is possible to automatically rewrite/redirect every incoming request to a lowercase URL with something like this:

http {
	...
	perl_modules  perl/lib;

 	perl_set $uri_lowercase 'sub {
   		my $r = shift;
   		my $uri = $r->uri;
   		$uri = lc($uri);
   		return $uri;
 	}';
	...
}

server {
	...

	location / {
               if ( $uri != $uri_lowercase ) {
                       rewrite . http://$host$uri_lowercase;
               }
# or just:
#		rewrite . $uri_lowercase;
	}
	...
}

The first option (with the if and http://$host) sends an HTTP redirect to the lowercase URL if the requested URL is not completely in lowercase. I like this approach better as it kind of normalizes all URL to a canonical form. The second option (without if nor http://) just accepts any URL and internally rewrites them to lowercase: if you rename all your directories and files to lowercase, this will imitate Windows’ case-insensitive behaviour.

Kudos to the following post on the nginx users list were I draw the “inspiration” :D from to get this done. The embedded perl doc and examples are indeed scarce. :-(

http://forum.nginx.org/read.php?2,39425,39944

Anyway I see two problems in this approach:

  • you need the embedded perl module which according to the docs is experimental and can lead to memory leaks.
  • the actual redirection is done with the rewrite, which you can put on the location you need. But the URL lowercase calculation, being on the “http” section of the config, is done for each and every request arriving to your server. Say you have a virtual server with 10 domains and you only need this on one particular location of one of them. The lowercase URL is going to be calculated for every request of every domain.

I guess that by defining a perl function the URL calculation could be restricted to a particular location, have to look into it further. Another option is writing a C module.

And why would anybody want to rewrite every URL to lowercase? We’re migrating a legacy Tomcat-based application from Windows to Linux, and it seems the original developers were not “case-aware” at all, they mixed upper and lowercase in the filenames without regarding how the actual files were named. On Windows this is not a problem but when moving the app to Linux it is. So we have renamed every file and directory to lowercase and used the previous configuration on the nginx servers that load-balance our Tomcat farm. Some corner-cases remain (includes in JSP files, which don’t route back to the nginx server and are dealt internally by Tomcat) and have been dealt with one by one, but the bulk of wrongly addressed images, videos, links to HTML files, etc. now works.

Ajustar la hora en servidores Amazon EC2

  • english
  • spanish

Amazon EC2 virtual servers’ internal clock is synchronized with that of their host server. But some host servers seems to be off-date: I’m working on a four-server cluster (will grow to eight when we go into production) and while three of them were perfectly synchronized, the fourth one lagged behind a couple of minutes.

By default you can’t adjust the date with either date nor ntp, the commands execute but do nothing. In order to decouple the VM’s clock from the host’s one and being able to set the time on your system you need to run the following command:

echo 1 > /proc/sys/xen/independent_wallclock

Bear in mind that, as pointed out here, after decoupling your clock from the host’s you can’t go back, you’ll be on your own forever.

Mantener las sesiones SSH abiertas

  • english
  • spanish

Sometimes SSH sessions die. At work I usually have several ssh sessions with a number of servers, and depending on the configuration of some intermediate router/firewall some TCP sessions get closed after a little while, the SSH sessions die and you have to log back in again.

To prevent this you can instruct sshd to implement some kind of keepalive mechanism preventing the TCP sessions from dying, by including these options on the sshd_config file:

KeepAlive yes
ClientAliveInterval 60