Search code examples
additiondevicepoolmirrorzfs

repairing misconfigured mirrored zfs pool

My machine boots from a mirrored zfs pool of two USB devices. The pool used to look like this:

sudo zpool status
pool: freenas-boot
state: ONLINE
scan: resilvered 891M in 15h19m with 0 errors on Wed Mar 29 03:29:55 2017
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE     0     0     0
      mirror-0  ONLINE       0     0     0
        da0p2   ONLINE       0     0     3
        da1p2   ONLINE       0     0     0

errors: No known data errors

I tried to replace the media with the checksum error, but through a series of incorrectly used commands I ended up "adding" the new drive instead of "attaching" it. The pool is now a concatenation of two devices and no longer mirrored:

  NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE     0     0     0
      da0p2     ONLINE       0     0     0
      da1p2     ONLINE       0     0     0

I learned that I cannot remove the device from the pool any more and the obvious

 zpool attach -f pool device new_device

command does not work.

I planned to mirror all the data on (large enough) third and fourth drives and then get rid of the concatenated first and second. However, it appears that mirroring only works on a device basis. I would need four devices for a mirrored configuration, which is unfavourable as only two USB ports are inside the machine's casing.

What would be a good way to move towards a bootable mirrored pool of two devices? Please keep in mind that the machine in question is a server and its failure to boot would make several users unhappy.

Solution
    1. Recursively zfs snapshot your root pool
    2. zfs send/recv this snapshot to another stick or volume of equal size, configured as a separate root pool
    3. Make this separate root pool with your copied data bootable
    4. Reboot the system and boot from your new root pool
    5. Add both sticks as mirrors to the new pool, thereby destroying the old one
    6. If resilvering has finished, remove the initial stick or volume and destroy it

    You now have a correctly mirrored pool with both initial sticks.