Standing on the ZFS filesystem on Ubuntu

By Patineboot

/ Update - 11 minutes read - 2136 words
Sankoyu lights its name with the features and puts out the noren with the sento symbol.

Sankoyu lights its name with the features and puts out the noren with the sento symbol.

Abstract

  1. Environment
  2. Create ZFS pool
  3. Create ZFS dataset
  4. Invent the Method
    1. Mount encrypted ZFS datasets
  5. Snapshot ZFS pool
  6. Configure ZED
  7. Archive ZFS pool
  8. Misc
  9. Reference

Environment

ZFS on Linux 2.1.4 for Filesystem.

Install ZFS on Linux

  1. Install ZFS on Linux with the ‘zfsutils-linux’ package on apt.

    sudo apt install zfsutils-linux
whereis zfs

    We can confirm to be installed correctly with whereis zfs.

Hardware List

IBM PC compatible:

External Storage:

Software Environment

Create ZFS pool

Show Disk ID

Check the disk IDs by running ls on /dev/disk/by-id/.

$ ls -la  /dev/disk/by-id/

lrwxrwxrwx 1 root root    9 Aug 30 21:19 ata-CT1000MX500SSD1_XXXXXXXXXXXX -> ../../sdb
lrwxrwxrwx 1 root root    9 Aug 30 21:19 ata-Samsung_SSD_860_EVO_1TB_XXXXXXXXXXXXXXX -> ../../sda

We got the disk IDs, ata-Samsung_SSD_860_EVO_1TB_XXXXXXXXXXXXXXX and ata-CT1000MX500SSD1_XXXXXXXXXXXX.
The disk IDs are identical even if we would connect the SSDs to other sockets on the machine.

Explain the zpool create command

Create a mirrored ZFS pool with the disk IDs from the ls command.

zpool create \
    -o ashift=12 -o autotrim=on \
    -O dnodesize=auto -O normalization=formD -O relatime=on \
    -O acltype=posixacl -O xattr=sa \
    -O compression=lz4 \
    -O dedup=on \
    <pool name> <raid type> <device id> ...

Specify the <raid type> option, and some multiple <device id>s if we create a ZFS pool with the raid.
Otherwise, Specify a single disk without the <raid type> option.

Explain the zpool create options

Specified the pairs of option and value:

Pool Options:

  • ashift=12
    The sector size of the ZFS pool is 12.
    I recommends that ashift is always 12.
    We treats multiple ZFS pools with the same ashift=12 option for sending and receiving ZFS filesystems well.
  • autotrim=on
    Send the trim command to SSD automatically.

Dataset Options:

  • acltype=posixacl and xattr=sa
    Select POSIX ACL and store extended attributes into inode of ZFS filesystem.

  • compression=lz4
    Compress files stored in a ZFS dataset.
    Set lz4 on the compression property by considering a trade-off between speed and compression ratio.

  • dedup=on
    Remove redundant data from a ZFS dataset.
    Set on on the dedup property, we reduce the used size on the ZFS dataset with the deduplication.

    Deduplication needs a huge memory.

    e.g., Deduplication requires approximately 2.6 GiB memory if the size of the deduplication ZFS dataset is 1TiB. The size of the block is 128 KiB, which is the default value on ZFS on Linux.

    Formula: 1 TiB / 128 KiB * 320 Byte = 2 684 354 560 Byte.

Options more:

  • Setting normalization=formD eliminates some corner cases relating to UTF-8 filename normalization. It also implies utf8only=on, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see The problems with enforced UTF-8 only filenames.

  • Setting relatime=on is a middle ground between classic POSIX atime behavior (with its significant performance impact) and atime=off (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, relatime has been the default for other filesystems. See RedHat’s documentation for further information.

  • The above and others options specified at Ubuntu 22.04 Root on ZFS.

Create two ZFS pools

Actually, we will create two pools named root.pool and storage.pool.

  • Create a mirrored ZFS pool named storage.pool with the mirror option meaning RAID5 and the disk IDs identifying the SSDs.

    zpool create \
    -o ashift=12 -o autotrim=on \
    -O dnodesize=auto -O normalization=formD -O relatime=on \
    -O acltype=posixacl -O xattr=sa \
    -O compression=lz4 \
    -O dedup=on \
    storage.pool mirror ata-Samsung_SSD_860_EVO_1TB_XXXXXXXXXXXXXXX ata-CT1000MX500SSD1_XXXXXXXXXXXX

  • Create a basic ZFS pool named root.pool_ with a single disk.

    zpool create \
    -o ashift=12 -o autotrim=on \
    -O dnodesize=auto -O normalization=formD -O relatime=on \
    -O acltype=posixacl -O xattr=sa \
    -O compression=lz4 \
    -O dedup=on \
    root.pool ubuntu--vg-rootpool--vg

Check ZFS pools

Check the status of the ZFS pools with the zpool status command.

The command show the root.pool and storage.pool as the pool key.

Experience

Important: I enable the encryption property when creating a ZFS dataset, then I can transport the encryptionroot property with the zfs send and zfs recv commands.

Do NOT create a ZFS pool with the encryption property on ZFS dataset. In this case, the zfs send and zfs recv commands do not transport the encryptionroot property on the ZFS pool and display an error message on the terminal.

Create ZFS dataset

Explain ZFS create with encryption

Create the ZFS dataset encrypted with the AES cryptographic algorithm of 256 bits and the GCM mode.

zfs create \
    -o encryption=aes-256-gcm \
    -o keylocation=prompt \
    -o keyformat=passphrase \
    <pool name>/<dataset name>
  • encryption=aes-256-gcm
    Choose AES 256 bits cryptographic algorithm with GCM mode.
  • -o keylocation=prompt and keyformat=passphrase
    Choose to enter the passphrase to load the encryption key on the prompt.

keylocation and keyformat are available as the following:

  • keylocation:
    • prompt: The ZFS filesystem will ask for the key at the command prompt when an encrypted dataset is mounted.
    • file:///absolute/file/path: the key or a passphrase file location.
    • https://address: The key or a passphrase file location.
    • http://address: The key or a passphrase file location.
  • keyformat:
    • raw: the raw key bytes
    • hex: a hexadecimal key string
    • passphrase: a character sentences that generates a key

Explain Set Mount Point

Set a mount point into a ZFS dataset

zfs set mountpoint=<a mount point> <pool name>/<dataset name>

Create two ZFS datasets

Actually, we will create the ZFS datasets, which are home on the root.pool, and storage on the storage.pool.

Create the home ZFS dataset and mount it on the home system directory as the procedure.

  1. Create the encrypted ZFS dataset named home with AES 256 bits and the GCM mode, and a passphrase to load the encryption key on the prompt.

    zfs create \
    -o encryption=aes-256-gcm \
    -o keylocation=prompt \
    -o keyformat=passphrase \
    root.pool/home

  2. Copy recursively the files and directories involved in the /home directory to the /root.pool/home directory.

    rsync -avh --delete /home/ /root.pool/home/

  3. Set /home to the mountpoint property on the home ZFS dataset.

    zfs set mountpoint=/home root.pool/home

Create the encrypted ZFS dataset named storage with AES 256 bits with the GCM mode.

zfs create \
    -o encryption=aes-256-gcm \
    -o keylocation=prompt \
    -o keyformat=passphrase \
    storage.pool/storage

Mount Manually

We manually mount an encrypted dataset with a passphrase.

  • Load an encryption key and mount an encrypted ZFS dataset separately.

    zfs load-key <pool name>/<dataset name>
zfs mount -a

  • Mount an encrypted ZFS dataset after loading an encryption on the one command.

    zfs mount -l <pool name>/<dataset name>

Note: Ubuntu does NOT mount the encrypted ZFS datasets automatically.

Invent the Method

Mount encrypted ZFS datasets

We register the service to load encryption keys in the boot sequence with systemd.

Create the .zfs-dataset.key passphrase file, and move the passphrase file to the /root directory.

# Create the passphrase file involving the zfs datasets passphrase.
# Press the CTRL + C keys after typing the passphrase.
cat - | tee .zfs-dataset.key

# Change the permission of the passphrase file to read only.
sudo chmod 400 .zfs-dataset.key

# Change the owner of the passphrase file to the root user.
sudo chown root:root .zfs-dataset.key

# Move the file to the root directory.
sudo mv .zfs-dataset.key /root/

Create, on the root user, the new zfs-load-enckey.service service file for loading the encryption key on the boot sequence.

[Unit]
Description=Load the encryption key for the ZFS datasets
DefaultDependencies=no
RequiresMountsFor=/root/

# Do itself after importing ZFS pool and before mount ZFS filesystems
After=zfs-import.target
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=%s -c 'yes `cat %h/.zfs-dataset.key` | /sbin/zfs load-key -a'

[Install]
WantedBy=zfs.target

Register the zfs-load-enckey.service service with systemd set to run the service on boot, and start it now.

# copy the service file to the user directory of the systemd.
cp zfs-load-enckey.service /etc/systemd/system/

# Enable and start now the service.
systemctl enable --now zfs-load-enckey.service

Note: The systemctl disable command unregisters the service from systemd.

Snapshot ZFS pool

Install Elephant Backup

Elephant Backup is an easy backup CLI to archive any number of ZFS filesystems containing lots of snapshots to another ZFS filesystem.

Install Elephant Backup package:

npm install elephant-backup

Note: Install npm with the apt package manager if npm does not exist on Ubuntu.
Install node with npm and reboot after installing npm.

apt install npm
npm --global install node
reboot

Take snapshot automatically

Take snapshots automatically with the systemd timer.

Start to take snapshots automatically.

elephant-backup systemd-install root.pool storage.pool

Stop taking snapshots automatically.

elephant-backup systemd-uninstall

Share the ZFS snapshots on Samba

We configure Samba to share ZFS snapshots with an SMB client.

Add the snapshot section to the /etc/samba/smb.conf configure file.

[snapshot]
   # Assign xattr to fruit:resource if xattr is not assigned in the global section.
   fruit:resource = xattr

   path = /storage.pool/storage/.zfs/snapshot
   valid users = patine

   access based share enum = yes
   browseable = yes

   writable = no

We share the ZFS snapshots directly and access them on an SMB client while mounting /storage.pool/storage.

Configure ZED

Install an SMTP client and Configure ZED. ZED stands for ZFS Event Daemon.

ZED sends a notification email when a scrub starts or our pool becomes unhealthy.

I add the following pairs of the key and the value to the configure file located at /etc/zfs/zed.d/zed.rc.

ZED_EMAIL_PROG="mail"
ZED_EMAIL_OPTS="-s '@SUBJECT@' @ADDRESS@" 
ZED_NOTIFY_VERBOSE=1

Restart the ZED service.

systemctl restart zed.service

We complete setting up the ZFS filesystem.

Archive ZFS pool

We archive the primary ZFS pools, named root.pool and the storage.pool, to a archive ZFS pool to be named archive.pool.

Create archive ZFS pool

Create the archive.pool ZFS pool with a single ZFS pool from the disk ID.

zpool create \
    -o ashift=12 -o autotrim=on \
    -O dnodesize=auto -O normalization=formD -O relatime=on \
    -O acltype=posixacl -O xattr=sa \
    -O compression=lz4 \
    -O dedup=on \
    archive.pool scsi-SRealtek_RTL9210B-CG_0000000000000000

Run Elephant Backup

We archive the primary ZFS pools to the archive.pool ZFS pool.
For example:

elephant-backup backup -a archive.pool root.pool storage.pool

I provide the bash (Bourne Again Shell) script, example-getting-started.sh, which archives ZFS pools, and print the difference of the primary ZFS pools. Fit the bash script to your environment.

Misc

Helpful ZFS Commands

# Import/Export a ZFS pool
# Import a ZFS pool after we attach an external storage device.
zpool import <pool>
# Export the ZFS pool before we detach the device.
zpool export <pool>

# Display the status of all ZFS pools on Ubuntu.
zpool status
# Display the capacity of all ZFS pools on Ubuntu.
zpool list

# Display the capacity of all ZFS datasets on Ubuntu.
zfs list
# Display all the snapshots of all  ZFS datasets on Ubuntu.
zfs list -t snapshot 
# Display all the snapshots of the specified ZFS dataset with the date.
zfs list -r -t snapshot -o name,creation <pool>/<dataset>
# Display the variable size of the ZFS datasets.
zfs list -o space,mountpoint

# Mount/Unmount a ZFS pool
# Load all of the keys on Ubuntu
zfs load-key -a
# Mount all of the datasets on Ubuntu
zfs mount -a
# Mount a ZFS dataset with an encryption key
zfs mount -l <dataset>
# Unmount the ZFS dataset forcefully.
zfs unmount -u <dataset>

# Destroy a ZFS pool
zpool destroy <pool>

Helpful ZFS filesystem Properties

# Choose lz4 compression on a ZFS dataset property.
zfs set compression=lz4 <pool>/<dataset>

# Enable deduplication of a ZFS dataset property.
zfs set dedup=on <pool>/<dataset>

# Choose posixacl acl type on a ZFS dataset property.
zfs set acltype=posixacl <pool>/<dataset>

# Enable extended attribute stored on i-node of a ZFS dataset property.
zfs set xattr=sa <pool>/<dataset>

# Assign a mount point on a ZFS dataset property.
zfs set mountpoint=<a mount point> <pool>/<dataset>

# Disable auto-snapshot for zfs-auto-snapshot.
zfs set com.sun:auto-snapshot=false <pool>/<dataset>

System Directory

I recommend Ubuntu 22.04 Root on ZFS if we use system directories on the ZFS filesystem.

I found the following problems to mount ZFS filesystem after booting.

  • /etc
    There are the configuration files for the ZFS filesystem in etc. At least, The process importing ZFS pools is dependent on /etc/zfs/zpool.cache.

  • /var
    /var contains the log files. journalctl refers to the journal log after mounting the ZFS filesystem.

Security Consideration

Do not write the raw passphrase for loading the encryption key in the auto-mount service file. The passphrase is exposed to every user on your Ubuntu server.

The right way exists on this page.

Reference