Last month I was working on a FreeBSD 10.1 KVM image for SmartOS, SmartDataCenter and the Joyent Public Cloud. The first version of the image was released a few weeks ago and I'd like to share how I went about building the image. More specifically, I'd like to provide an overview of how I built a custom FreeBSD 10.1 ISO that I use as part of my build process.

A Little Background

If you're not familiar with how KVM images are created for SmartOS/SmartDataCenter the general workflow goes like this:

1. Create a blank KVM VM
2. Boot it with an installation ISO (in this case a FreeBSD ISO)
3. Install the OS via the ISO
4. Install the guest tools for the OS and customize the installation
5. Shut the VM down (poweroff)
6. Take a ZFS snapshot of the VM
7. Save the ZFS snapshot as a file using the zfs send command
8. Finally, create an image manifest for the image

For more detail about all of the steps above, you can read more in the SDC7 doc under How to Create a KVM Image.

For this post, I'm going to focus on steps 3 to 5 which are all handled via a custom ISO I built and helps to speed up the whole process tremendously.

Building a Custom FreeBSD 10.1 ISO

Before I begin, you can find all my scripts and files I used to build a custom FreeBSD ISO on GitHub here: mi-freebsd-10 . Pretty much everything I'm going to talk about can be found in the build_freebsd_iso script, so if you want take a look at that first, got for it.

Building a custom FreeBSD ISO is fairly easy but there are a few catches (more on that later). Once you have the source ISO (e.g., FreeBSD-10.1-RELEASE-amd64-disc1.iso) you mount it, make the modifications you need, then create a new custom ISO. If you're doing this under FreeBSD, it would look something like this:

# Mount the ISO using /mnt/freebsd-iso as the mount point
mount -t cd9660 /dev/$(mdconfig -f FreeBSD-10.1-RELEASE-amd64-disc1.iso`)/mnt/freebsd-iso

# Copy the ISO contents to /mnt/custom-freebsd-iso
rsync -aq  /mnt/freebsd-iso/ /mnt/custom-freebsd-iso

# After making modifications to /mnt/custom-freebsd-iso,
# create a new ISO
CUSTOM_ISO_TITLE=$(isoinfo -d -i ${ISO_DIR}/${ISO} | grep "Volume id" | awk '{print $3}')
mkisofs -J -R -no-emul-boot \
-V "$CUSTOM_ISO_TITLE" \
-p "Joyent" -b boot/cdboot \
-o freebsd-10-custom.iso /mnt/custom-freebsd-iso

One thing you'll notice about the above code is that I'm getting the ISO label using isoinfo. Starting with the FreeBSD 10.1 release, each ISO label (or "Volume ID") is unique and is based on the release information (version, architecture, that kind if thing). If you don't use the correct label in your custom ISO, the FreeBSD installation will fail spectacularly! I learned this the hard way...

As for actually customizing the ISO itself, there are a couple of tricks to that. First, modifications you make to the ISO won't actually be included in your FreeBSD installation. So for example, if you make changes to /mnt/custom-freebsd-iso/etc/rc.conf it won't be included in the installation when you start the install process. Typically you'd want to do things like add networking info to the rc.conf file so the installer has network access (ifconfig_vtnet0="DHCP") or shorten the boot delay by adding autoboot_delay="5" to /mnt/custom-freebsd-iso/boot/loader.conf (the default is 10 seconds). So what if you want to have an installation that has modifications to the rc.conf file? And how do you add new files to the installation? Well, it turns out that FreeBSD allows you to script the installation process!

Automating the Installation

FreeBSD 9.0 introduced a new installer called bsdinstall. When you boot a FreeBSD 9.0 or newer ISO the installer is automatically started once the boot timer runs out, which kicks off the bsdinstall process.

You can automate the installation by including a file called installerconfig in the /etc/ folder of the ISO. When the FreeBSD installation starts it looks for that file. The file allows you to set some configuration defaults in a "preamble" section using environment variables. It's similar to a Kickstart file you'd use in Fedora or CentOS. If you take a look at the installerconfig file in the mi-freebsd-10 repos you'll see that I have it partition vtbd0 using the typical defaults:

PARTITIONS=vtbd0

You'll also notice I'm telling the installer to include the following distributions:

DISTRIBUTIONS="kernel.txz base.txz joyent.txz"

If you want a minimal FreeBSD install, you'd typically just have kernel.txz base.txz here. So what's the joyent.txz distribution file and where did it come from? Glad you asked! That's how I install the various guest tools I need for our KVM image to run under SmartOS and SmartDataCenter. I'll get back to that in a bit, but there's more stuff in the installerconfig script I want to cover.

After the preamble section theres the "Setup Script" which starts with a shebang:

 #!/bin/sh

Here is where you do all your customization occurs. This section is run after the FreeBSD installation completes. Looking at the installerconfig file on GitHub you can see this is where I do things like modify the rc.conf file for the installation, create a custom /etc/motd file and install packages via pkg install -y. But what about adding new files?

Adding a Custom Distribution File

The installerconfig file lets you do a lot of useful things. One thing that you can't do is copy files into the installation. This is because at this point the installation is all happening within a chroot. By default that's the /mnt directory (which you can change via the BSDINSTALL_CHROOT environment variable, so the installation is run in that directory with no visibility to anything outside of it. That means you can't copy files over using the scripted part of installerconfig. This is where the joyent.txz distribution comes in.

To install files to the installation, you just save them all as a distribution file, the add that file to DISTRIBUTIONS= in the preamble of installerconfig. The bsdinstall process will take you distribution file and unpack it to where it needs to go.

Here's the basic steps for creating the distribution file called custom. Let's say you have a custom rc.conf file that you want to include. Create a directory called custom_files.

mkdir custom_files

The rc.conf file likes in /etc, so you need to create an /custom_files/etc folder to match:

mkdir custom_files/etc

Now, add your file to custom_files/etc and create the distribution using the tar command:

cp rc.conf custom_files/etc/
cd custom_files
tar -cvJpf ../custom.txz ./etc

And that's it, you're custom rc.conf file is now in a distribution file called "custom.txz". If you include in in the preamble part of your installerconfig file (e.g., DISTRIBUTIONS="kernel.txz base.txz custom.txz"), bsdinstall will unpack and copy everything in your custom distribution over to their respective directories.

One final thing to note about the installerconfig: at the end of the file I'm issuing the poweroff command. Normally the bsdinstall process will run what's after the shebang and then automatically reboot. Since I'm creating a SmartOS/SmartDataCenter image, I don't want that to happen because I need to do a snapshot. Issuing a full shutdown at the end prevents the automatic reboot from happening.

So that's pretty much it. The above information should hopefully be useful to anyone who wants to create a custom FreeBSD ISO with an automated/unattended installation.

And special thanks to Daniel Malon who provide a lot of help and introduced me to the bsdinstall man page :)