How to install FreeBSD 8.2-STABLE with ZFS root Filesystem

–> I realize there are a number of articles and stories around about how to install FreeBSD with a ZFS root (and subsequently the complete filesystem is ZFS, of course). Since not all articles are about FreeBSD 8.2 but older versions, some installs from those pages do not work. On this page I have gathered, combined, and rewrote all steps to insure a safe and working ZFS FreeBSD 8.2 installation.

First of all…

What is ZFS ?

ZFS is a combined file system and logical volume manager designed by Sun Microsystems. The features of ZFS include data integrity verification against data corruption modes (like bit rot), support for high storage capacities, integration of the concepts of filesystem and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs. ZFS is implemented as open-source software, licensed under the Common Development and Distribution License (CDDL). The ZFS name is a trademark of Oracle. read more

If you have read the above (and maybe the complete article on wikipedia), you will understand that ZFS is, under most circumstances, a perfect replacement for the native FreeBSD filesystem, the Fast File System or FFS which is built on the original Unix™ File System, also known as UFS.

Tutorial – How to install FreeBSD 8.2 with ZFS root Filesystem

Ok, so, now that we know this, how to install this great filesystem with FreeBSD 8.2-STABLE. In this tutorial I will assume you have an empty system, eg. nothing on disk, no residing operating system or dual boot. First we need to have the FreeBSD DVD ISO and burn it to disk. For this you should visit the FreeBSD mirror site and choose your nearest mirror. There go to releases/amd64/ISO-IMAGES/8.2 and choose the file FreeBSD-8.2-RELEASE-amd64-dvd1.iso.xz.

Since I live in the Netherlands, my closest mirror is in the Netherlands. Subsequently I downloaded this file:
ftp://ftp.nl.freebsd.org/pub/FreeBSD/releases/amd64/ISO-IMAGES/8.2/FreeBSD-8.2-RELEASE-amd64-dvd1.iso.xz

NB.: It is possible to choose another architecture, like i386 instead of amd64. I assume however that nowadays most people would use amd64 (I do!). You will need amd64 when you have more than 3GB of RAM (and nowadays most people do), and believe me, if you have a large filesystem like me, you will need a lot of RAM with ZFS.

As you can see this file is compressed with xz. xz is an archiver like gzip and is available on most unix and linux systems. On freebsd it is natively installed on the latest releases. You will have to extract the archive:

xz -d FreeBSD-8.2-RELEASE-amd64-dvd1.iso.xz

Now that you have the image file FreeBSD-8.2-RELEASE-amd64-dvd1.iso, you can burn it to disk with your favorite disk burning software.

Ok, let’s take your new disk, put it in the drive and up the system you want to install.

You have to boot off the disk, choose your country and region, then load until you are in the sysinstall menu. There, choose “Fixit” and then choose CDROM/DVD. It will then present you with the fixit prompt.

Let’s first load the kernel modules you need in order to create and use the ZFS filesystem:

kldload /mnt2/boot/kernel/opensolaris.ko
kldload /mnt2/boot/kernel/zfs.ko

Ok, Now, let’s assume we have 1 harddisk in the system. ad0. Let’s further assume that ad0 is 699GB (`gpart show ad0` will show you how large it actually is).

So let’s first start with installing the system on ad0.

Install the bootloader:

gpart create -s gpt ad0
gpart add -t freebsd-boot -s 64k ad0
gpart add -t freebsd-zfs -s 683G ad0
gpart add -t freebsd-swap ad0
gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 ad0

What did we do?

1) We created a GPT (Guid Partition Table). With GPT you don’t have the old limit of 4 partition tables anymore and you can easily span partitions over 2TB or larger. The old MBR systems had problems with that.

2) Then we created a 64K bootsector at the beginning of the disc.

3) Then we created the zfs filesystem with a size of 683 GB. It’s not really the filesystem yet, but it’s a slice. That’s a way to tell FreeBSD we want to have a ZFS filesystem there.

4) After that we created the last slice with the swap space. I always use double the size of my RAM. Since in this case I have 8 GB RAM, I calculated that 699 – 683 = 16GB swap space. This is why I created the zfs slice with 683GB so that I had 16GB left for swap space.

5) And then finally we copied the GPT ZFS Bootcode off the DVD on the bootsector.

If you want to have a look what your disk looks like now, just do:

unix@phenomium:~$ gpart show ad0
=> 34 1465149101 ada0 GPT (699G)
34 128 1 freebsd-boot (64k)
162 1432354816 2 freebsd-zfs (683G)
1432354978 32794157 3 freebsd-swap (16G)

Ok, on with the fun…and let’s create the ZFS filesystem on slice 2…

mkdir /boot/zfs
zpool create zroot ad0p2

zfs create -o mountpoint=legacy zroot/ROOT
zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/freebsd-8.2

zfs create -o compression=on -o exec=on -o setuid=off zroot/ROOT/freebsd-8.2/tmp
chmod 1777 /zroot/ROOT/freebsd-8.2/tmp

zfs create zroot/ROOT/freebsd-8.2/usr
zfs create zroot/ROOT/freebsd-8.2/usr/home
cd /zroot/ROOT/freebsd-8.2/ ; ln -s usr/home home

zfs create -o compression=lzjb -o setuid=off zroot/ROOT/freebsd-8.2/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/ROOT/freebsd-8.2/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/ROOT/freebsd-8.2/usr/ports/packages
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/ROOT/freebsd-8.2/usr/src

zfs create zroot/ROOT/freebsd-8.2/var
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/ROOT/freebsd-8.2/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/ROOT/freebsd-8.2/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/ROOT/freebsd-8.2/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/ROOT/freebsd-8.2/var/mail
zfs create -o exec=off -o setuid=off zroot/ROOT/freebsd-8.2/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/ROOT/freebsd-8.2/var/tmp
chmod 1777 /zroot/ROOT/freebsd-8.2/var/tmp

mount -t zfs zroot/ROOT/freebsd-8.2 /mnt

Here we have created a zpool with the name ‘zroot’ on slice 2 of disk ad0, ad0p2.
Then we created a volume ROOT and a volume freebsd-8.2 which we gave a standard mountpoint /. That will be our root in the future.
After that we created several different volumes with some interesting zfs options, like compression and exec rights and so on. Please read the zfs manual for more information about the zfs options; please do it…it will come very handy in the future, I tell ya!
then we mounted the zroot/ROOT/freebsd-8.2 volume in /mnt, because we have to copy some files to there now:

export DESTDIR=/mnt
cd /dist/8.2-RELEASE/
for dir in base catpages dict doc games info lib32 manpages ports; \
do (cd $dir ; ./install.sh) ; done
cd src ; ./install.sh all
cd ../kernels ; ./install.sh generic
cd /mnt/boot ; cp -Rlp GENERIC/* kernel/
zfs set readonly=on zroot/ROOT/freebsd-8.2/var/empty

You’ll have to answer yes/ok two or three times. And then the base system is installed.

Let’s get on with the final settings to make sure your system will boot correctly.

echo ‘zfs_enable=”YES”‘ > /mnt/etc/rc.conf
echo ‘hostname=”zfs.mydomain.local”‘ >> /mnt/etc/rc.conf
echo ‘ifconfig_em0=”DHCP”‘ >> /mnt/etc/rc.conf
echo ‘zfs_load=”YES”‘ > /mnt/boot/loader.conf
echo ‘vfs.root.mountfrom="zfs:zroot/ROOT/freebsd-8.2‘ >> /mnt/boot/loader.conf
echo ‘LOADER_ZFS_SUPPORT=YES’ > /mnt/etc/src.conf
zpool set bootfs=zroot/ROOT/freebsd-8.2 zroot
cp /boot/zfs/zpool.cache /mnt/boot/zfs/
touch /mnt/etc/fstab


of course here you will have to edit your hostname and maybe your ifconfig too. And there you have it. A complete installation with a ZFS root filesystem on FreeBSD 8.2 …RELEASE!

Yes, we’re not done yet We wanted 8.2-STABLE, right ?

You really want 8.2-STABLE because the ZFS version of STABLE is much higher than the default RELEASE version. The higher version will give you more options and lots of bugfixes were done since 8.2-RELEASE.

First let’s reboot the system. type exit at the prompt, and leave the sysinstall menu. The system should restart automatically and you should make sure you have taken out the DVD from the drive. The system should now boot up from harddisk and you should have a login prompt after the boot sequence. If something goes wrong, you probably have done something wrong in executing the tutorial. If you’re sure you did it right, leave me a message with what went wrong and I will investigate and correct the tutorial if necessary. Don’t hesitate to ask any questions if you don’t understand something. I’d be happy to help, provided I have the time.

Log in with root. I don’t think you have set a password yet, so it won’t ask for any. If the system at this time asks for a password, you must have been naughty and filled it in already in an earlier stage. please provide your password and log in.

first we will run the installer:

$ sysinstall

you’ll get the menu. The first things you should do now are:

1) go to the post-install configuration and set the systems managers password
2) in the post-install configuration set up a new usergroup and user
3) set which timezone you are in

If you have done that, in the post-install configuration, go to the ‘install pre-packaged software for FreeBSD’.
Choose “Install from an FTP server”
Choose “Main Site”
On the question whether to assume the network is already configured, choose Yes.

It will take some time loading all indices and package data, but eventually you will have a nice menu where you can scroll down and choose the option “Networking Utilities”. There you scoll down to the package “cvsup-without-gui-16.1h_4″ and select it. Then with TAB to OK and ENTER, then select Install and ENTER. It will then install the package at your bidding.

After the installation of cvsup, create the file ‘stable-supfile’ in the root directory. This file should have the following contents:

*default host=cvsup.se.FreeBSD.org
*default base=/var/db
*default prefix=/usr
*default release=cvs tag=RELENG_8
*default delete use-rel-suffix
*default compress
src-all

(you can go to the mirror sites page at freebsd.org to check if there is a server near you and change the default host. You should not change the rest!)

When this file is created, run the command:

cvsup /root/stable-supfile

This will contact the default host above and update your source tree with the latest updates of the 8.2-STABLE branch. This process can take some time, so get a cup of coffee while you’re at it.

ok, so the updates have finished. Now you have to rebuild the complete system:

cd /usr/src
make buildworld
make buildkernel KERNCONF=GENERIC
make installkernel KERNCONF=GENERIC
shutdown -r now

Your system will now reboot.

Log in again as root. On with the show:

cd /usr/src
mergemaster -p
make installworld
make delete-old
mergemaster
shutdown -r now

If everything went well you now have a running 8.2-STABLE system with ZFS root Filesystem. There are a few final thingies to do:

Log in as root.

cd /usr/src
make delete-old-libs

Now do:

zpool upgrade -v

if you see a message stating that your pool is out of date, you will have to upgrade the zpool.

zpool upgrade zroot

Now do:

zfs upgrade -v

If you see a message stating that your filesystem is out of date, you will have to upgrade the filesystem.

zfs upgrade zroot

Now your zfs filesystems and pools are upgraded to the latest version.

Congratulations ! You have completed the installation !