I have a 3 Disk RAIDz1 Pool, encrypted with AES128 in GEOM_ELI, that I have been using in FreeNAS since version 8. There have been many zpool upgrades, and over all I am very happy with ZFS.
Lately however I have been growing frustrated with FreeNAS. Largely many bugs that haven't been fixed over the years. But overall its the INSISTING on me using a Flash drive for their os, even though most of it is read only. It's still a Single point of failure and has always extended boot times by several minutes. Bottom line, I just want to use Vanilla FreeBSD with this pool. I am looking for more flexibility and a I wish to educate myself with this awesome Operating System.
Doing some more extended research I have found many tutorials on installing FreeBSD naively to a ZFS volume and mounting it as / It wasn't till I did more research and found an article on mounting a zfs encrypted volume as root. Later I found that FreeBSD 10 does this during installation, which is awesome to say the least.
I made a VM With VMWare workstation, with three 2TB Drives, passed through as Physical Disks, and followed every step to a T and everything worked out very well. Now that I had a better grasp on the commands I was doing and why I was doing them, I wanted to do this to an already existing pool, that has a lot of data already on it.
By Default, FreeNAS Creates a 2GB SWAP partition at the front of every data disk. I removed the swap space and made it 1.5GB partition on each drive with 512MB remaining for Swap. I followed through every step, changing things as needed. (I have 3 disks, tutorial speaks of 4, My pool name is foxhole, the tutorial is zroot.) I was successful in decrypting my volume with geom_eli and mounted it successfully.
I did not skip any steps provided. I even copied every command I was given and altered them in a text file so they would suit my case.
Here is my problem now. After finally restarting to test everything, The kernel begins starting, then I am spat at a mountroot terminal. It seems that geom_eli didn't make an attempt to decrypt my root volume. I have a suspicion why. Correct me if I am wrong. At the start of the tutorial, I am given commands to create new geoms for the encrypted volume:
geli init -b -B /boot/zfs/bootdir/da0p4.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da0p4
geli init -b -B /boot/zfs/bootdir/da1p4.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da1p4
geli init -b -B /boot/zfs/bootdir/da2p4.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da2p4
Since my volume already exists, I cant perform those commands that would have created "/boot/zfs/bootdir/daXp4.eli" files.
I am really just guessing at this being the cause. I noticed this when i attempted to perform:
mv bootdir/*.eli bootdir/boot/
Gave me "No Match."
I assumed those would have been created when the pool was decrypted.
I apologize for this post. I am trying to give as much info as I can without giving too much. I have been working on this for the last 18 hours. I would really love someone with a clear head to take a peek at this.
If I missed any useful information, let me know.
Turns out I was correct. The daXp4.eli files are necessary as it's the metadata of each disk. A reference point if you will.
By performing:
geli backup /dev/daXp4 /boot/daXp4.eli
It create the meta files required for geom to attempt a decryption at boot time.
I hope this helps someone else interested in this stuff. I now have a NAS with 23 Disks. 3 ZFS Volumes, all encrypted with geom_eli