Thornton 2 dot Com
1K5CO

GELI Encrypted USB Backup of ZFS Filesystems in FreeBSD 13

Posted 2021-12-22, edited 2021-12-23.

First off, this is mostly the result of two pages I found elsewhere on the 'Net.

  1. FreeBSD ZFS snapshots with zfstools
  2. Back Up ZFS to a Removable Drive using zxfer

I reference manual pages throughout. If you're new-ish to Unix, you'll see references like "intro(1)": a reference to a manual page (or manpage) and the section number it's in. To read intro(1), for example, type man 1 intro at a shell prompt. Type man man to read how to use your system's manual. Do a lot of reading and thinking as you follow along. Before you try out any commands for yourself, read their manpages and make sure you understand what my commands did and why. Have an xterm or other terminal window sitting next to this browser window.

Steps

Reasoning and System Installation to Back Up

My main home computer started out running FreeBSD 11, and I upgraded it to 12 before it came time to replace the hardware. I initially partitioned the HDD using the guided ZFS with encryption in the FreeBSD installer. However, I didn't know how to do proper encrypted backups, so I wound up making and (poorly) managing tarballs of files I didn't want to lose.

The reason I chose full disk encryption is because I use my main computer for sensitive activities, and if it gets stolen, I don't want to worry about the thief stealing my identity and ruining my life in addition. Without my backup drives also being encrypted, I ran that add-on risk from having a backup lost or stolen.

When it came time to replace my computer, I chose an all-in-one with the opposite size HDD and a laptop with an NVMe instead of spinning platters. Because of this, I had to do fresh installs of FreeBSD 13 and muck about with those tarballs. I didn't lose any data, but I did lose an awful lot of time that I didn't need to. Anyway, with the new computers came new USB drives to use for backup and a new determination to do it better if not do it right.

I named my computers "swiftpaw" and "braveplumes," and you'll see their names in the shell prompts prefacing every command.

Assumptions and Preparation

I want to have ZFS snapshots made automatically and covering reasonable stretches of time so that clobbered or deleted files can be recovered when the clobbering is discovered. I want to send those snapshots to my backup drive so that I can recover files, filesystems, or the whole pool of data if I ever need to. At the same time, I want those snapshots managed automatically and old snapshots culled to avoid filling up either main or backup disks. ZFS snapshots are also easier to back up because they're basically a moment of the filesystem frozen in time, and I won't need to worry about files changing or disappearing mid-backup while the snapshot persists.

The most important assumption is that this is the start of a new backup regimen, no snapshots worth keeping exist in the ZFS pool being backed up, and that any backup regimen already in place will be abandoned in favor of this one. Since I was using huge and poorly managed tarballs before, I'm pretty happy to abandon it for snapshot shuttling.

The backup regimen I've adopted needs the following at minimum:

I'm going to use one USB storage device to back up both of my computers. That bumps up the requirement to a storage device at least as big as both computers' ZFS pools combined.

A good backup maintenance policy involves, when possible, backups stored on multiple physical storage media and rotation of those media through on-site and off-site storage locations. That bumps up the number of USB drives for backup from one to two or more. I'll get more as my day job allows, but this guide assumes just one drive, and adding more drives is as simple as repeating these steps for each.

Choosing ZFS Filesystems to Snapshot Automatically

Look at the ZFS filesystems in your pool (see zpool-list(8) and zfs-list(8)), decide which ones hold data you can't easily replace, and decide roughly how frequently their files change. When I installed FreeBSD 13, it gave me these filesystems in my pool:

(It also gave me zroot, zroot/ROOT, zroot/usr, and zroot/var, but these are basically containers for children filesystems, not used directly for data storage.)

Of these, the ones I either can't replace or can't easily replace are:

Filesystem Reason
zroot/ROOT/default Configs and root's home directory.
zroot/usr/home My home directory lives here.
zroot/var/log System logs to figure out what went wrong.
zroot/var/mail Mail spools for me and root.

That means the ones that don't need to be snapshotted are the ones left:

Filesystem Reason Why Not
zroot/tmp Temporary files, cleaned at reboot.
zroot/usr/ports I can just portsnap(8) a new tree.
zroot/usr/src Can get again by following the Handbook.
zroot/var/audit Audit logging not enabled by default.
zroot/var/crash Don't need kernel core dumps snapshotted.
zroot/var/tmp Don't need temporary files snapshotted.

Of course, your needs and levels of importance may be different, so look at your pool and make your own decisions.

Enabling Automatic ZFS Filesystem Snapshotting

Zfstools looks at the ZFS user property com.sun:auto-snapshot to determine whether to snapshot a filesystem or not. On a new system, the property should be unset, neither true nor false, neither local nor inherited. Use zfs get com.sun:autosnapshot to see its status for all ZFS filesystems, and see zfs-get(8) for details.

Set the ZFS property com.sun:auto-snapshot to false on ZFS filesystems zfstools shouldn't snapshot for you. In my case, the command looked like:

root@braveplumes:~ # zfs set com.sun:autosnapshot=false \
        zroot/tmp \
        zroot/usr/ports \
        zroot/usr/src \
        zroot/var/audit \
        zroot/var/
root@braveplumes:~ #

See zfs-set(8), zfs-inherit(8), and the "User Properties" section of zfsprops(8) for details.

Once you have filesystems set as excluded, turn on snapshotting for the rest of the pool. I used:

root@braveplumes:~ # zfs set com.sun:auto-snapshot=true zroot
root@braveplumes:~ #

Now it looked like:

root@braveplumes:~ # zfs get -r com.sun:auto-snapshot zroot
NAME                PROPERTY               VALUE  SOURCE
zroot               com.sun:auto-snapshot  true   local
zroot/ROOT          com.sun:auto-snapshot  true   inherited from zroot
zroot/ROOT/default  com.sun:auto-snapshot  true   inherited from zroot
zroot/tmp           com.sun:auto-snapshot  false  local
zroot/usr           com.sun:auto-snapshot  true   inherited from zroot
zroot/usr/home      com.sun:auto-snapshot  true   inherited from zroot
zroot/usr/ports     com.sun:auto-snapshot  false  local
zroot/usr/src       com.sun:auto-snapshot  false  local
zroot/var           com.sun:auto-snapshot  true   inherited from zroot
zroot/var/audit     com.sun:auto-snapshot  false  local
zroot/var/crash     com.sun:auto-snapshot  false  local
zroot/var/log       com.sun:auto-snapshot  true   inherited from zroot
zroot/var/mail      com.sun:auto-snapshot  true   inherited from zroot
zroot/var/tmp       com.sun:auto-snapshot  false  local
root@braveplumes:~ #

Now read /usr/local/share/doc/zfstools/README.md if you haven't already. Zfstools expects to run as a cron job, and the documentation provides example crontab(5) lines. The "INTERVAL" names it offers as examples are frequent, hourly, daily, weekly, and monthly, but these are free-form names solely for your benefit. You can choose any name you want for each interval, any number of intervals you want, and any times those intervals will run.

I like all the defaults, but I decided to change the names. Instead of frequent, hourly, daily, weekly, and monthly, I chose the 4-character names 015m, 1hly, 2dly, 3wky, and 4mth respectively because I'm weird like that. So the crontab I installed looks like this:

SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin

15,30,45 * * * * /usr/local/sbin/zfs-auto-snapshot 015m  4
0        * * * * /usr/local/sbin/zfs-auto-snapshot 1hly 24
7        0 * * * /usr/local/sbin/zfs-auto-snapshot 2dly  7
14       0 * * 7 /usr/local/sbin/zfs-auto-snapshot 3wky  4
28       0 1 * * /usr/local/sbin/zfs-auto-snapshot 4mth 12

Edit: After some thinking, I realized my laptop won't be turned on during the daily, weekly, or monthly runs, so I modified its crontab to have just three intervals (every 15 minutes, every hour, and at each reboot) instead. I named the new interval "boot":

SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin

# zfstools automatic snapshotting
15,30,45 * * * * /usr/local/sbin/zfs-auto-snapshot 015m  4
0        * * * * /usr/local/sbin/zfs-auto-snapshot 1hly 24
@reboot          /usr/local/sbin/zfs-auto-snapshot boot 14

Don't forget to set the $PATH right. Zfstools is a collection of ruby scripts, and weird things happen if env(1) can't find ruby. If cron starts logging or mailing you weird errors like this, check your $PATH.

/usr/local/lib/ruby/site_ruby/2.7/zfstools/dataset.rb:29:in `popen': No such file or directory - zfs (Errno::ENOENT)
        from /usr/local/lib/ruby/site_ruby/2.7/zfstools/dataset.rb:29:in `list'
        from /usr/local/lib/ruby/site_ruby/2.7/zfstools.rb:132:in `find_eligible_datasets'
        from /usr/local/sbin/zfs-auto-snapshot:65:in `<main>'

If all goes right, you'll start seeing snapshots like zroot/usr/home@zfs-auto-snap_1hly-2021-12-16-12h00 when you run zfs list -t snap zroot/usr/home.

Zfstools also lets you exclude filesystems from interval-specific auto snapshotting without excluding it from other intervals. To do this, set the ZFS property com.sun:auto-snapshot:INTERVAL to false, where "INTERVAL" is the interval name. For example, I turned off the "015m" interval on zroot/ROOT and zroot/ROOT/default with the command:

root@braveplumes:~ # zfs set com.sun:auto-snapshot:015m=false zroot/ROOT
root@braveplumes:~ #

Preparing the Backup Drive

Now to turn the backup drive into a GELI-encrypted partition containing ZFS pools.

Plug in the new, unprepared backup drive, then run dmesg to find the device node name. If plugging in the drive is the most recent thing you did, then information about it should be at the end of the dump. On my system, the node name is da0. Double-check you get yours right, because the first step is destroying its partition table.

For the rest of the preparation examples, I set some variables to help me out. My root shell is tcsh, not the /bin/sh default of users, so I had to use the "set" keyword:

root@braveplumes:~ # set thishost=`hostname -s`
root@braveplumes:~ # set thispool="zroot"
root@braveplumes:~ # set backupdev="da0"
root@braveplumes:~ #

Once again, make sure you change "da0" to your backup drive's actual device node name! Data you want to keep will be deleted in the next step if you don't use your backup drive's actual device node name!

I listed the partition table to verify what I was about to destroy looked right, then I destroyed it and verified it:

root@braveplumes:~ # gpart show $backupdev
=>       63  240353217  da0  MBR  (115G)
         63  240353217       - free -  (115G)

root@braveplumes:~ # gpart destroy -F $backupdev
da0 destroyed
root@braveplumes:~ # gpart destroy -F $backupdev
gpart: arg0 'da0': Invalid argument
root@braveplumes:~ #

Then I created a GPT partition table with the label "backup" for the ZFS partition:

root@braveplumes:~ # gpart create -s gpt $backupdev
da0 created
root@braveplumes:~ # gpart add -a 1m -l backup -t freebsd-zfs $backupdev
da0p1 added
root@braveplumes:~ # gpart show -l $backupdev
=>       40  240353200  da0  GPT  (115G)
         40       2008       - free -  (1.0M)
       2048  240349184    1  backup  (115G)
  240351232       2008       - free -  (1.0M)
root@braveplumes:~ #

Next, I encrypted and attached the new partition, using what the FreeBSD installer used as a cheat sheet. See the "init" command of geli(8) for what options I used, didn't use, and why.

root@braveplumes:~ # grep "geli init" /var/log/bsdinstall_log
DEBUG: zfs_create_boot: geli init -bg -e AES-XTS -J - -l 256 -s 4096 "nvd0p4"
root@braveplumes:~ # geli init -e AES-XTS -l 256 -s 4096 "/dev/gpt/backup"
Enter new passphrase:
Reenter new passphrase:

Metadata backup for provider /dev/gpt/backup can be found in /var/backups/gpt_backup.eli
and can be restored with the following command:

        # geli restore /var/backups/gpt_backup.eli /dev/gpt/backup

root@braveplumes:~ # geli attach /dev/gpt/backup
Enter passphrase:
root@braveplumes:~ # geli status /dev/gpt/backup.eli
          Name  Status  Components
gpt/backup.eli  ACTIVE  gpt/backup
root@braveplumes:~ #

Once the partition was attached, I created the ZFS pool for the backup, then a new dataset inside "backup" for the backup of my computer named braveplumes. I could've used the "backup" pool directly, but the backup drive I'm using is big enough to hold backups for both of my computers, so that means a dataset inside for each computer.

root@braveplumes:~ # zpool create backup gpt/backup.eli
root@braveplumes:~ # zfs create backup/$thishost
root@braveplumes:~ # zpool list backup
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
backup   114G   528K   114G        -         -     0%     0%  1.00x    ONLINE  -
root@braveplumes:~ # zfs list -r backup
NAME                 USED  AVAIL     REFER  MOUNTPOINT
backup               528K   110G       96K  /backup
backup/braveplumes    96K   110G       96K  /backup/braveplumes
root@braveplumes:~ #

The final bit of preparation is backing out of the backup drive, removing it, then reinserting and getting back into it to make sure I can:

root@braveplumes:~ # zpool export backup
root@braveplumes:~ # geli detach gpt/backup.eli
root@braveplumes:~ #

At this point, it was safe to unplug the backup drive. Unplug, count to ten, plug in, then:

root@braveplumes:~ # geli attach /dev/gpt/backup
Enter passphrase:
root@braveplumes:~ # zpool import -N backup
root@braveplumes:~ # sh -c 'if zfs list backup/`hostname -s` >/dev/null; then echo true; else echo false; fi'
true
root@braveplumes:~ # zpool export backup
root@braveplumes:~ # geli detach gpt/backup.eli
root@braveplumes:~ #

Performing the Backup for the First Time

At last, it's time to actually perform the backup. This first one is going to take a very long time because it's a full backup, not an incremental from an existing snapshot to the next.

What I did resulted in some mistakes that you're likely to encounter as well, so keep that in mind as you follow along. After this first backup, I developed a more refined backup procedure to follow the next time.

First, I set up some variables to make the backup command easier to follow, modify, and if necessary repeat:

root@braveplumes:~ # set thishost=`hostname -s`
root@braveplumes:~ # set thispool="zroot"
root@braveplumes:~ #

Next, zxfer(8) has a -I switch for excluding certain ZFS properties from the backup dataset copies. I don't want zfs-auto-snapshot to snapshot, manage, and potentially destroy the already backed up filesystems, so that (and my 015m relative) is my first exclusion.

root@braveplumes:~ # set exclude="com.sun:auto-snapshot"
root@braveplumes:~ # set exclude="${exclude},com.sun:auto-snapshot:015m"
root@braveplumes:~ #

And now, I performed the backup. The first two examples in zxfer(8)'s examples section are almost exactly what I want; the only differences are that I don't have any snapshots to grandfather in and that I don't need multiple copies on a single backup drive. (I'd need to use a backup drive twice as big if I do want multiple copies per backup drive.)

Zxfer also has beeping options, -b that will play a tune on failure only, and -B that will play one of two tunes when it finishes. These go through the kernel speaker device /dev/speaker and can be quite loud on modern PCs. Add one of them if there's no one around to annoy with loud beeps and you think you'll step away and go do something else while the backup runs.

root@braveplumes:~ # geli attach /dev/gpt/backup
Enter passphrase:
root@braveplumes:~ # zpool import -N backup
root@braveplumes:~ # zxfer -dFkPv -I $exclude -R $thispool backup/$thishost
Creating destination filesystem "backup/braveplumes/zroot" with specified properties.
cannot create 'backup/braveplumes/zroot': 'objsetid' is readonly
Error when creating destination filesystem.
root@braveplumes:~ #

This is a bug in zxfer 1.1.7. The workaround is to add "objsetid" to the exclusion list. Let's try again:

root@braveplumes:~ # set exclude="${exclude},objsetid"
root@braveplumes:~ # zxfer -dFkPv -I $exclude -R $thispool backup/$thishost
Creating destination filesystem "backup/braveplumes/zroot" with specified properties.
cannot create 'backup/braveplumes/zroot': minimum pbkdf2 iterations is 100000
Error when creating destination filesysetm.
root@braveplumes:~ #

This error was annoying, and there's barely anything on the rest of the Internet hinting at what it means. I eventually discovered that ZFS datasets can be encrypted separately from the GELI partitions they're stored on, and this is one of the encryption properties. However, none of mine are using encryption:

root@braveplumes:~ # zfs get -r encryption zroot | grep -v "@"
NAME                                   PROPERTY    VALUE        SOURCE
zroot                                  encryption  off          default
zroot/ROOT                             encryption  off          default
zroot/ROOT/default                     encryption  off          default
zroot/tmp                              encryption  off          default
zroot/usr                              encryption  off          default
zroot/usr/home                         encryption  off          default
zroot/usr/ports                        encryption  off          default
zroot/usr/src                          encryption  off          default
zroot/var                              encryption  off          default
zroot/var/audit                        encryption  off          default
zroot/var/crash                        encryption  off          default
zroot/var/log                          encryption  off          default
zroot/var/mail                         encryption  off          default
zroot/var/tmp                          encryption  off          default
root@braveplumes:~ #

The workaround is to exclude three ZFS encryption properties:

root@braveplumes:~ # set exclude="${exclude},keylocation"
root@braveplumes:~ # set exclude="${exclude},keyformat"
root@braveplumes:~ # set exclude="${exclude},pbkdf2iters"
root@braveplumes:~ #

I tried again and got success, only to be black flagged on the last lap:

root@braveplumes:~ # zxfer -dFkPv -I $exclude -R $thispool backup/$thishost
Creating destination filesystem "backup/braveplumes/zroot" with specified properties.
Sending zroot@zfs-auto-snap_1hly-2021-12-20-07h00 to backup/braveplumes/zroot.
Sending zroot@zfs-auto-snap_015m-2021-12-20-07h45 to backup/braveplumes/zroot.
  (incremental to zroot@zfs-auto-snap_1hly-2021-12-20-07h00.)
Creating destination filesystem "backup/braveplumes/zroot/ROOT" with specified properties.
Creating destination filesystem "backup/braveplumes/zroot/ROOT/default" with specified properties.
Sending zroot/ROOT/default@zfs-auto-snap_1hly-2021-12-18-18h00 to backup/braveplumes/zroot/ROOT/default.
Sending zroot/ROOT/default@zfs-auto-snap_1hly-2021-12-18-19h00 to backup/braveplumes/zroot/ROOT/default.
  (incremental to zroot/ROOT/default@zfs-auto-snap_1hly-2021-12-18-18h00.)
Sending zroot/ROOT/default@zfs-auto-snap_1hly-2021-12-19-12h00 to backup/braveplumes/zroot/ROOT/default.
  (incremental to zroot/ROOT/default@zfs-auto-snap_1hly-2021-12-18-19h00.)
:
[similar snapshot sending statuses snipped]
:
Creating destination filesystem "backup/braveplumes/zroot/tmp" with specified properties.
Creating destination filesystem "backup/braveplumes/zroot/usr" with specified properties.
Creating destination filesystem "backup/braveplumes/zroot/usr/home" with specified properties.
Sending zroot/usr/home@zfs-auto-snap_1hly-2021-12-18-19h00 to backup/braveplumes/zroot/usr/home.
Sending zroot/usr/home@zfs-auto-snap_1hly-2021-12-19-12h00 to backup/braveplumes/zroot/usr/home.
  (incremental to zroot/usr/home@zfs-auto-snap_1hly-2021-12-18-19h00.)
Sending zroot/usr/home@zfs-auto-snap_1hly-2021-12-19-13h00 to backup/braveplumes/zroot/usr/home.
  (incremental to zroot/usr/home@zfs-auto-snap_1hly-2021-12-19-12h00.)
:
[similar snapshot sending statuses snipped]
:
Creating destination filesystem "backup/braveplumes/zroot/usr/ports" with specified properties.
Creating destination filesystem "backup/braveplumes/zroot/usr/src" with specified properties.
Creating destination filesystem "backup/braveplumes/zroot/var" with specified properties.
Creating destination filesystem "backup/braveplumes/zroot/var/audit" with specified properties.
Creating destination filesystem "backup/braveplumes/zroot/var/crash" with specified properties.
Creating destination filesystem "backup/braveplumes/zroot/var/log" with specified properties.
Sending zroot/var/log@zfs-auto-snap_1hly-2021-12-18-18h00 to backup/braveplumes/zroot/var/log.
Sending zroot/var/log@zfs-auto-snap_1hly-2021-12-18-19h00 to backup/braveplumes/zroot/var/log.
  (incremental to zroot/var/log@zfs-auto-snap_1hly-2021-12-18-18h00.)
Sending zroot/var/log@zfs-auto-snap_1hly-2021-12-19-12h00 to backup/braveplumes/zroot/var/log.
  (incremental to zroot/var/log@zfs-auto-snap_1hly-2021-12-18-19h00.)
:
[similar snapshot sending statuses snipped]
:
Creating destination filesystem "backup/braveplumes/zroot/var/mail" with specified properties.
Sending zroot/var/mail@zfs-auto-snap_1hly-2021-12-20-07h00 to backup/braveplumes/zroot/var/mail.
Error when zfs send/receiving.
root@braveplumes:~ #

Well, that was annoying. The command ran through the top of the hour, and the snapshot I was backing up was culled mid-backup. As annoying as that was, I kind of expected it. The fix is to re-run the backup:

root@braveplumes:~ # zxfer -dFkPv -I $exclude -R $thispool backup/$thishost
Destroying destination snapshot backup/braveplumes/zroot@zfs-auto-snap_1hly-2021-12-20-07h00.
Sending zroot@zfs-auto-snap_1hly-2021-12-20-08h00 to backup/braveplumes/zroot.
  (incremental to zroot@zfs-auto-snap_015m-2021-12-20-07h45.)
Sending zroot/ROOT/default@zfs-auto-snap_1hly-2021-12-20-08h00 to backup/braveplumes/zroot/ROOT/default.
  (incremental to zroot/ROOT/default@zfs-auto-snap_1hly-2021-12-20-07h00.)
Sending zroot/usr/home@zfs-auto-snap_1hly-2021-12-20-08h00 to backup/braveplumes/zroot/usr/home.
  (incremental to zroot/usr/home@zfs-auto-snap_015m-2021-12-20-07h45.)
Sending zroot/var/log@zfs-auto-snap_1hly-2021-12-20-08h00 to backup/braveplumes/zroot/var/log.
  (incremental to zroot/var/log@zfs-auto-snap_015m-2021-12-20-07h45.)
Sending zroot/var/mail@zfs-auto-snap_1hly-2021-12-20-08h00 to backup/braveplumes/zroot/var/mail.
Creating destination filesystem "backup/braveplumes/zroot/var/tmp" with specified properties.
Writing backup info to location /backup/braveplumes/.zxfer_backup_info.zroot
root@braveplumes:~ #

Now that's more like it. Now if I zfs mount backup/${thishost}/somefilesystem and less /backup/${thishost}/somefilesystem/.zfs/snapshot/zfs-auto-snap_1hly-2021-12-18-18h00/somefile, I'll see the contents of that file if it existed in zroot/somefilesystem at that time. (Use zfs unmount backup/${thishost}/somefilesystem to unmount it after you're done with it.)

Edit: Oops. Zxfer wrote some metadata to /backup/${thishost}/.zxfer_backup_info.zroot under the assumption it would be a directory in the backup pool. It wasn't, because the -N switch on zpool-import(8) prevented its filesystems from being mounted. I should've zfs-mount(8)ed backup/${thishost} before running zxfer. I already did that in my backup script, but I forgot to point it out below or do that in my trials above.

We could (and I did) scrub the backup drive to make sure everything's there and good. This will take at least as long as the first backup because it's every byte of every dataset in the backup pool. I added the -w switch to wait, knowing it was done when I got the shell prompt back.

root@braveplumes:~ # zpool scrub -w backup
root@braveplumes:~ #

Since we're done with the first backup attempts, let's remove the backup drive and prepare for an actual backup:

root@braveplumes:~ # zpool export backup
root@braveplumes:~ # geli detach /dev/gpt/backup.eli
root@braveplumes:~ #

Properly Performing a Backup

Set up the variables for the zxfer command. (I'm going to script this later.)

root@braveplumes:~ # set thishost=`hostname -s`
root@braveplumes:~ # set thispool="zroot"
root@braveplumes:~ # set exclude="com.sun:auto-snapshot"
root@braveplumes:~ # set exclude="${exclude},com.sun:auto-snapshot:015m"
root@braveplumes:~ # set exclude="${exclude},objsetid"
root@braveplumes:~ # set exclude="${exclude},keyformat"
root@braveplumes:~ # set exclude="${exclude},keylocation"
root@braveplumes:~ # set exclude="${exclude},pbkdf2iters"
root@braveplumes:~ #

Plug in the backup drive, then attach it (edit: then mount the backup root):

root@braveplumes:~ # geli attach /dev/gpt/backup
Enter passphrase:
root@braveplumes:~ # zpool import -N backup
root@braveplumes:~ # zfs mount "backup/${thishost}"
root@braveplumes:~ #

Perform the backup:

root@braveplumes:~ # zxfer -dFkPv -I ${exclude} -R ${thispool} backup/${thishost}
[output cut]
root@braveplumes:~ #

Detach the backup drive, then unplug it when the last shell prompt reappears. (Edited as noted above.)

root@braveplumes:~ # zfs unmount "backup/${thishost}"
root@braveplumes:~ # zpool export backup
root@braveplumes:~ # geli detach /dev/gpt/backup.eli
root@braveplumes:~ #

Adding a New Computer to the Backup Drive

I'm going to use my backup drive to back up both of my computers, since it's big enough for both. All I needed to do is plug the backup drive into my second computer and do a very simple preparation:

root@swiftpaw:~ # set thishost=`hostname -s`
root@swiftpaw:~ # set thispool="zroot"
root@swiftpaw:~ # geli attach /dev/gpt/backup
Enter passphrase:
root@swiftpaw:~ # zpool import -N backup
root@swiftpaw:~ # zfs create backup/$thishost
root@swiftpaw:~ # zpool list backup
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
backup   114G  2.75G   111G        -         -     0%     2%  1.00x    ONLINE  -
root@swiftpaw:~ # zfs list -r backup
NAME                                    USED  AVAIL     REFER  MOUNTPOINT
backup                                 2.75G   108G       96K  /backup
backup/braveplumes                     2.75G   108G       96K  /backup/braveplumes
backup/braveplumes/zroot               2.75G   108G       96K  /backup/braveplumes/zroot
backup/braveplumes/zroot/ROOT          1.92G   108G       96K  /backup/braveplumes/zroot/ROOT
backup/braveplumes/zroot/ROOT/default  1.92G   108G     1.91G  /backup/braveplumes/zroot/ROOT/default
backup/braveplumes/zroot/tmp             96K   108G       96K  /backup/braveplumes/zroot/tmp
backup/braveplumes/zroot/usr            851M   108G       96K  /backup/braveplumes/zroot/usr
backup/braveplumes/zroot/usr/home       850M   108G      847M  /backup/braveplumes/zroot/usr/home
backup/braveplumes/zroot/usr/ports       96K   108G       96K  /backup/braveplumes/zroot/usr/ports
backup/braveplumes/zroot/usr/src         96K   108G       96K  /backup/braveplumes/zroot/usr/src
backup/braveplumes/zroot/var           3.30M   108G       96K  /backup/braveplumes/zroot/var
backup/braveplumes/zroot/var/audit       96K   108G       96K  /backup/braveplumes/zroot/var/audit
backup/braveplumes/zroot/var/crash       96K   108G       96K  /backup/braveplumes/zroot/var/crash
backup/braveplumes/zroot/var/log       2.80M   108G      448K  /backup/braveplumes/zroot/var/log
backup/braveplumes/zroot/var/mail       128K   108G      128K  /backup/braveplumes/zroot/var/mail
backup/braveplumes/zroot/var/tmp         96K   108G       96K  /backup/braveplumes/zroot/var/tmp
backup/swiftpaw                          96K   108G       96K  /backup/swiftpaw
root@swiftpaw:~ # zpool export backup
root@swiftpaw:~ # geli detach gpt/backup.eli
root@swiftpaw:~ #

Next, I have to choose and enable ZFS snapshotting on the second computer just like I did on my first, potentially making different decisions, and wait for the first snapshots to be made.

Now if I run my backup commands on the second computer with the backup drive, I'll have backups of both of my computers.

Scripting the Regular Backup

I put these commands in a shellscript to make backups more convenient. The script is divided into five sections: usage comments, preparation, backup proper, clean-up, and cheat-sheet comments.

root@braveplumes:~ # touch ./backup.sh
root@braveplumes:~ # chmod u+x ./backup.sh
root@braveplumes:~ # ed ./backup.sh
0
a

Usage comments

#!/bin/sh

# Exports snapshots to the backup disk.
#
# Plug in the backup disk, then be ready to type its GELI key to attach
# it when prompted.
#
# Usage:
#
#     ./backup.sh
#                     Perform a backup.
#
#     ./backup.sh -b
#                     Perform a backup, then play a chime through the
#                     PC speaker when done.
#
# Snapshots are made automatically via zfstools.
# The transfer is via zxfer.
# See this script's end comments for preparation howto.

Preparation

First is setting up the variables for the host being backed up (thus the backup filesystem receiving the backups), the pool being backed up ("zroot" on both of my computers), and the zxfer exclusion options:

# ######################################################################

thishost="$( hostname -s )"
thispool="zroot"

                        # Don't let zfs-auto-snapshot snapshot backups
exclude="com.sun:auto-snapshot"
exclude="${exclude},com.sun:auto-snapshot:015m"
                        # Other zxfer exclusions
exclude="${exclude},special_small_blocks"       # feat. not in usb disk
exclude="${exclude},keylocation,keyformat"      # not encrypted zfs
exclude="${exclude},pbkdf2iters"                # not encrypted zfs
exclude="${exclude},objsetid"                   # zxfer bug workaround

# ######################################################################

Processing the command line switch:

playchime=0
while getopts 'b' c
do
        case $c in
        b)      playchime=1     ;;
        *)      :               ;;      # nop
        esac
done

Making sure zxfer is installed and the backup drive is plugged in:

if ! which zxfer >/dev/null 2>&1
then printf "%s\n" "Please install zxfer." >&2; exit 1
fi

if [ ! -e "/dev/gpt/backup" ]
then
        printf "%s\t" "[FAIL]"
        printf "%s\n" "Backup drive not inserted." >&2
        exit 1
fi

Attaching the backup drive:

gelidetach=1
if [ -e "/dev/gpt/backup.eli" ]
then
        printf "%s\t" "[Note]"
        printf "%s\n" "Backup drive already attached."
        gelidetach=0
else
        printf "%s\t" "[ OK ]"
        printf "%s\n" "Attaching backup drive."
        geli attach "/dev/gpt/backup"
fi
if [ ! -e "/dev/gpt/backup.eli" ]
then
        printf "%s\t" "[FAIL]"
        printf "%s\n" "Backup drive not mounted." >&2
        exit 1
fi

Importing the backup pool:

zpoolexport=1
if zfs list -H "backup" >/dev/null 2>&1
then
        printf "%s\t" "[Note]"
        printf "%s\n" "Backup pool already imported."
        zpoolexport=0
else
        printf "%s\t" "[ OK ]"
        printf "%s\n" "Importing backup pool."
        zpool import -N backup
fi
if zfs list -H "backup/${thishost}" >/dev/null 2>&1
then
        printf "%s\t" "[ OK ]"
        printf "%s\n" "Mounting 'backup/${thishost}'."
        mkdir -p "/backup/${thishost}"
        zfs mount "backup/${thishost}"
else
        printf "%s\t" "[FAIL]"
        printf "%s\n" "Backup directory is missing in backup pool." >&2
        if [ $zpoolexport -eq 1 ]
        then
                printf "%s\t" "[back]"
                printf "%s\n" "Exporting backup pool." >&2
                zpool export backup
        fi
        if [ $gelidetach -eq 1 ]
        then
                printf "%s\t" "[back]"
                printf "%s\n" "Detaching backup drive." >&2
                geli detach "/dev/gpt/backup.eli"
        fi
        exit 1
fi

Backup proper

printf "%s\t" "[ OK ]"
printf "%s\n" "Commencing backup."

if zxfer -dFkPv -I "${exclude}" -R "${thispool}" "backup/${thishost}"
then
        printf "%s\t" "[ OK ]"
        printf "%s\n" "Backup completed successfully."
        sync; sync; sync
        if zfs unmount "backup/${thishost}"
        then
                printf "%s\t" "[ OK ]"
                printf "%s\n" "Unmounted 'backup/${thishost}' cleanly."
        else
                printf "%s\t" "[WARN]"
                printf "%s\n" "Did not unmount 'backup/${thishost}' cleanly."
        fi
        rmdir "/backup/${thishost}" >/dev/null 2>&1
        rmdir "/backup" >/dev/null 2>&1
else
        printf "%s\t" "[FAIL]"
        printf "%s\n" "Backup encountered a problem."
        sync; sync; sync
fi

Clean-up

Exporting the backup pool:

if [ $zpoolexport -eq 1 ]
then
        printf "%s\t" "[ OK ]"
        printf "%s\n" "Exporting backup pool."
        zpool export backup
else
        printf "%s\t" "[Note]"
        printf "%s\n" "Leaving backup pool in place."
fi

Detaching the backup drive:

if [ $gelidetach -eq 1 ]
then
        printf "%s\t" "[ OK ]"
        printf "%s\n" "Detaching backup drive."
        geli detach "/dev/gpt/backup.eli"
else
        printf "%s\t" "[Note]"
        printf "%s\n" "Leaving backup drive attached."
fi

Announcing the end:

printf "%s\t" "[ OK ]"
printf "%s\n" "Done."

if [ "$playchime" -eq 1 ]
then
        # Jetsons doorbell chime
        echo "T208O3L4FAL8B>L4C." >/dev/speaker 2>/dev/null
fi

Cheat sheet comments

# ######################################################################
#
# Backup cheat sheet
#
# Preparation:
#
# pkg install zfstools zxfer
#
# Creation of snapshots to back up:
#
# zfs get com.sun:auto-snapshot # list auto-snapshot status
# #
# # exclude filesystems from snapshotting
# #
# zfs set com.sun:auto-snapshot=false zroot/tmp
# zfs set com.sun:auto-snapshot=false zroot/usr/ports
# zfs set com.sun:auto-snapshot=false zroot/usr/src
# zfs set com.sun:auto-snapshot=false zroot/var/audit
# zfs set com.sun:auto-snapshot=false zroot/var/crash
# zfs set com.sun:auto-snapshot=false zroot/var/tmp
# #
# # exclude filesystems from super-frequent snapshotting
# #
# zfs set com.sun:auto-snapshot:015m=false zroot/ROOT
# #
# # snapshot root filesystem and all children except excluded above
# #
# zfs set com.sun:auto-snapshot=true zroot
# #
# # Fire off automatic snapshotting via cron
# #
# crontab -e
# a
# SHELL=/bin/sh
# PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
# # zfstools automatic snapshotting
# 15,30,45 * * * * /usr/local/sbin/zfs-auto-snapshot 015m  4
# 0        * * * * /usr/local/sbin/zfs-auto-snapshot 1hly 24
# 7        0 * * * /usr/local/sbin/zfs-auto-snapshot 2dly  7
# 14       0 * * 7 /usr/local/sbin/zfs-auto-snapshot 3wky  4
# 28       0 1 * * /usr/local/sbin/zfs-auto-snapshot 4mth 12
# .
# wq
#
# Backup disk creation:
#
# thishost="$( hostname -s )"
# thispool="zroot"
# backupdev="da0"               # see dmesg after plugging in backup drive
#
# gpart destroy -F ${backupdev} # probably da0, see dmesg
# gpart destroy -F ${backupdev} # should error if already destroyed
# gpart create -s gpt ${backupdev}
# gpart add -a 1m -l backup -t freebsd-zfs "${backupdev}"
# gpart show -l ${backupdev}    # should list new backup, whole disk
# grep "geli init" /var/log/bsdinstall_log      # how was live eli was created
# geli init -e AES-XTS -l 256 -s 4096 "/dev/gpt/backup"
# geli attach /dev/gpt/backup
# geli status                   # verify
# zpool create backup gpt/backup.eli
# zfs create backup/${thishost}
# zpool list                    # verify
# zfs list                      # verify
# zpool export backup
# geli detach gpt/backup.eli
#
# Backup procedure
#
# # don't let zfs-auto-snapshot snapshot backups
# exclude="com.sun:auto-snapshot"
# # other exclusions
# exclude="${exclude},special_small_blocks"     # usb disk doesn't have feature
# exclude="${exclude},keylocation,keyformat"    # not encrypted zfs
# exclude="${exclude},pbkdf2iters"              # not encrypted zfs
# geli attach /dev/gpt/backup
# zpool import backup
# zxfer -B -dFkPv -I ${exclude} -R ${thispool} backup/${thishost}
# zpool scrub -w backup
# zpool export backup
# geli detach gpt/backup.eli
#

And that's it.

.
wq
7164
root@braveplumes:~ #

Conclusion

Now when it comes time to run a backup, all I have to do is plug in the backup drive, get root, and run ./backup.sh (or ./backup.sh -b). When it's done, I just exit, unplug, and put the backup drive away.

Download the script

Make sure you edit "da0" on line 211 (in the cheat sheet comments) to match your system and setup.