Red Hat Enterprise Linux 5.3 is out so I recreated my custom Kickstart CDs, which surprisingly failed when I tried to install into a VirtualBox virtual machine. After a bit of digging I found the problem thanks to the most excellent way Red Hat built debugging into the installer - Python is great in these situations.
The installer failed in a 'inVmware()' call soon after booting because lspci is not on the minstg2.img file on the CD. It tries to run lspci to test whether it's running in a VMware virtual machine.
I managed to fix the problem using my Debian workstation by adding three files to the minstg2.img on the Red Hat CD. These files were libpci.so.3.0.0 and its associated link, and the lspci binary - there was no need to rebuild the ld.so.cache. That was it. After rebuilding the CD the install continued fine. OSS is great :-)
One caveat was that the RHEL 5.3 CD kernel uses version 3.0 of squashfs to read the minstg2.img file. Debian Lenny uses version 3.1 so the CD did not work when I remade it the first time. Using the RHEL 5.3 mksquashfs binary to build the minstg2.img file fixed this.