Standing on the ZFS filesystem on Ubuntu
By Patineboot

Sankoyu lights its name with the features and puts out the noren with the sento symbol.
Abstract
- Environment
- Create ZFS pool
- Create ZFS dataset
- Invent the Method
- Mount encrypted ZFS datasets
- Snapshot ZFS pool
- Configure ZED
- Archive ZFS pool
- Misc
- Reference
Environment
ZFS on Linux 2.1.4 for Filesystem.
Install ZFS on Linux
-
Install ZFS on Linux with the ‘zfsutils-linux’ package on
apt
.sudo apt install zfsutils-linux whereis zfs
We can confirm to be installed correctly with
whereis zfs
.
Hardware List
IBM PC compatible:
- MiniPC Barebone: ASRock DeskMini X300
- CPU: AMD Ryzen 5 PRO 4650G
- Storage: Samsung 980 M.2 SSD 500GB
- Memory: Crucial 16GB (8GB x 2)
- Storage(ZFS): Samsung 860 EVO 1TB SATA
- Storage(ZFS): Crucial MX500 1TB SATA
External Storage:
-
USB 3.0 Storage
- Case Converter Adapter: NFHK USB M.2 Converter
The official name is ‘USB 3.0 to NVME M-key M.2 NGFF SSD External PCBA Converter Adapter RTL9210 Chipset with Case.’ - M.2 SSD: Lexar NM610 M.2 2280 NVMe SSD
- Case Converter Adapter: NFHK USB M.2 Converter
-
SATA Hard Drive Adapter: StarTech.com USB312SAT3CB
Software Environment
- Operating System: Ubuntu 22.04 LTS Server
Create ZFS pool
Show Disk ID
Check the disk IDs by running ls
on /dev/disk/by-id/.
$ ls -la /dev/disk/by-id/
lrwxrwxrwx 1 root root 9 Aug 30 21:19 ata-CT1000MX500SSD1_XXXXXXXXXXXX -> ../../sdb
lrwxrwxrwx 1 root root 9 Aug 30 21:19 ata-Samsung_SSD_860_EVO_1TB_XXXXXXXXXXXXXXX -> ../../sda
We got the disk IDs, ata-Samsung_SSD_860_EVO_1TB_XXXXXXXXXXXXXXX and ata-CT1000MX500SSD1_XXXXXXXXXXXX.
The disk IDs are identical even if we would connect the SSDs to other sockets on the machine.
Explain the zpool create
command
Create a mirrored ZFS pool with the disk IDs from the ls
command.
zpool create \
-o ashift=12 -o autotrim=on \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O acltype=posixacl -O xattr=sa \
-O compression=lz4 \
-O dedup=on \
<pool name> <raid type> <device id> ...
Specify the <raid type> option, and some multiple <device id>s if we create a ZFS pool with the raid.
Otherwise, Specify a single disk without the <raid type> option.
Explain the zpool create
options
Specified the pairs of option and value:
Pool Options:
ashift=12
The sector size of the ZFS pool is 12.
I recommends thatashift
is always 12.
We treats multiple ZFS pools with the sameashift=12
option for sending and receiving ZFS filesystems well.autotrim=on
Send the trim command to SSD automatically.
Dataset Options:
-
acltype=posixacl
andxattr=sa
Select POSIX ACL and store extended attributes into inode of ZFS filesystem. -
compression=lz4
Compress files stored in a ZFS dataset.
Setlz4
on thecompression
property by considering a trade-off between speed and compression ratio. -
dedup=on
Remove redundant data from a ZFS dataset.
Set on on the dedup property, we reduce the used size on the ZFS dataset with the deduplication.Deduplication needs a huge memory.
e.g., Deduplication requires approximately 2.6 GiB memory if the size of the deduplication ZFS dataset is 1TiB. The size of the block is 128 KiB, which is the default value on ZFS on Linux.
Formula: 1 TiB / 128 KiB * 320 Byte = 2 684 354 560 Byte.
Options more:
-
Setting
normalization=formD
eliminates some corner cases relating to UTF-8 filename normalization. It also impliesutf8only=on
, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see The problems with enforced UTF-8 only filenames. -
Setting
relatime=on
is a middle ground between classic POSIX atime behavior (with its significant performance impact) andatime=off
(which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, relatime has been the default for other filesystems. See RedHat’s documentation for further information. -
The above and others options specified at Ubuntu 22.04 Root on ZFS.
Create two ZFS pools
Actually, we will create two pools named root.pool and storage.pool.
-
Create a mirrored ZFS pool named storage.pool with the
mirror
option meaning RAID5 and the disk IDs identifying the SSDs.zpool create \ -o ashift=12 -o autotrim=on \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O acltype=posixacl -O xattr=sa \ -O compression=lz4 \ -O dedup=on \ storage.pool mirror ata-Samsung_SSD_860_EVO_1TB_XXXXXXXXXXXXXXX ata-CT1000MX500SSD1_XXXXXXXXXXXX
-
Create a basic ZFS pool named root.pool_ with a single disk.
zpool create \ -o ashift=12 -o autotrim=on \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O acltype=posixacl -O xattr=sa \ -O compression=lz4 \ -O dedup=on \ root.pool ubuntu--vg-rootpool--vg
Check ZFS pools
Check the status of the ZFS pools with the zpool status
command.
The command show the root.pool and storage.pool as the pool key.
Experience
Important: I enable the encryption
property when creating a ZFS dataset, then I can transport the encryptionroot
property with the zfs send
and zfs recv
commands.
Do NOT create a ZFS pool with the encryption
property on ZFS dataset.
In this case, the zfs send
and zfs recv
commands do not transport the encryptionroot
property on the ZFS pool and display an error message on the terminal.
Create ZFS dataset
Explain ZFS create with encryption
Create the ZFS dataset encrypted with the AES cryptographic algorithm of 256 bits and the GCM mode.
zfs create \
-o encryption=aes-256-gcm \
-o keylocation=prompt \
-o keyformat=passphrase \
<pool name>/<dataset name>
encryption=aes-256-gcm
Choose AES 256 bits cryptographic algorithm with GCM mode.-o keylocation=prompt
andkeyformat=passphrase
Choose to enter the passphrase to load the encryption key on the prompt.
keylocation
and keyformat
are available as the following:
keylocation
:prompt
: The ZFS filesystem will ask for the key at the command prompt when an encrypted dataset is mounted.- file:///absolute/file/path: the key or a passphrase file location.
- https://address: The key or a passphrase file location.
- http://address: The key or a passphrase file location.
keyformat
:raw
: the raw key byteshex
: a hexadecimal key stringpassphrase
: a character sentences that generates a key
Explain Set Mount Point
Set a mount point into a ZFS dataset
zfs set mountpoint=<a mount point> <pool name>/<dataset name>
Create two ZFS datasets
Actually, we will create the ZFS datasets, which are home on the root.pool, and storage on the storage.pool.
Create the home ZFS dataset and mount it on the home system directory as the procedure.
-
Create the encrypted ZFS dataset named home with AES 256 bits and the GCM mode, and a passphrase to load the encryption key on the prompt.
zfs create \ -o encryption=aes-256-gcm \ -o keylocation=prompt \ -o keyformat=passphrase \ root.pool/home
-
Copy recursively the files and directories involved in the /home directory to the /root.pool/home directory.
rsync -avh --delete /home/ /root.pool/home/
-
Set /home to the
mountpoint
property on the home ZFS dataset.zfs set mountpoint=/home root.pool/home
Create the encrypted ZFS dataset named storage with AES 256 bits with the GCM mode.
zfs create \
-o encryption=aes-256-gcm \
-o keylocation=prompt \
-o keyformat=passphrase \
storage.pool/storage
Mount Manually
We manually mount an encrypted dataset with a passphrase.
-
Load an encryption key and mount an encrypted ZFS dataset separately.
zfs load-key <pool name>/<dataset name> zfs mount -a
-
Mount an encrypted ZFS dataset after loading an encryption on the one command.
zfs mount -l <pool name>/<dataset name>
Note: Ubuntu does NOT mount the encrypted ZFS datasets automatically.
Invent the Method
Mount encrypted ZFS datasets
We register the service to load encryption keys in the boot sequence with systemd.
Create the .zfs-dataset.key passphrase file, and move the passphrase file to the /root directory.
# Create the passphrase file involving the zfs datasets passphrase.
# Press the CTRL + C keys after typing the passphrase.
cat - | tee .zfs-dataset.key
# Change the permission of the passphrase file to read only.
sudo chmod 400 .zfs-dataset.key
# Change the owner of the passphrase file to the root user.
sudo chown root:root .zfs-dataset.key
# Move the file to the root directory.
sudo mv .zfs-dataset.key /root/
Create, on the root user, the new zfs-load-enckey.service service file for loading the encryption key on the boot sequence.
[Unit]
Description=Load the encryption key for the ZFS datasets
DefaultDependencies=no
RequiresMountsFor=/root/
# Do itself after importing ZFS pool and before mount ZFS filesystems
After=zfs-import.target
Before=zfs-mount.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=%s -c 'yes `cat %h/.zfs-dataset.key` | /sbin/zfs load-key -a'
[Install]
WantedBy=zfs.target
Register the zfs-load-enckey.service service with systemd set to run the service on boot, and start it now.
# copy the service file to the user directory of the systemd.
cp zfs-load-enckey.service /etc/systemd/system/
# Enable and start now the service.
systemctl enable --now zfs-load-enckey.service
Note: The systemctl disable
command unregisters the service from systemd.
Snapshot ZFS pool
Install Elephant Backup
Elephant Backup is an easy backup CLI to archive any number of ZFS filesystems containing lots of snapshots to another ZFS filesystem.
Install Elephant Backup package:
npm install elephant-backup
Note:
Install npm
with the apt
package manager if npm
does not exist on Ubuntu.
Install node with npm
and reboot after installing npm
.
apt install npm
npm --global install node
reboot
Take snapshot automatically
Take snapshots automatically with the systemd
timer.
Start to take snapshots automatically.
elephant-backup systemd-install root.pool storage.pool
Stop taking snapshots automatically.
elephant-backup systemd-uninstall
Share the ZFS snapshots on Samba
We configure Samba to share ZFS snapshots with an SMB client.
Add the snapshot section to the /etc/samba/smb.conf configure file.
[snapshot]
# Assign xattr to fruit:resource if xattr is not assigned in the global section.
fruit:resource = xattr
path = /storage.pool/storage/.zfs/snapshot
valid users = patine
access based share enum = yes
browseable = yes
writable = no
We share the ZFS snapshots directly and access them on an SMB client while mounting /storage.pool/storage.
Configure ZED
Install an SMTP client and Configure ZED. ZED stands for ZFS Event Daemon.
ZED sends a notification email when a scrub starts or our pool becomes unhealthy.
I add the following pairs of the key and the value to the configure file located at /etc/zfs/zed.d/zed.rc.
ZED_EMAIL_PROG="mail"
ZED_EMAIL_OPTS="-s '@SUBJECT@' @ADDRESS@"
ZED_NOTIFY_VERBOSE=1
Restart the ZED service.
systemctl restart zed.service
We complete setting up the ZFS filesystem.
Archive ZFS pool
We archive the primary ZFS pools, named root.pool and the storage.pool, to a archive ZFS pool to be named archive.pool.
Create archive ZFS pool
Create the archive.pool ZFS pool with a single ZFS pool from the disk ID.
zpool create \
-o ashift=12 -o autotrim=on \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O acltype=posixacl -O xattr=sa \
-O compression=lz4 \
-O dedup=on \
archive.pool scsi-SRealtek_RTL9210B-CG_0000000000000000
Run Elephant Backup
We archive the primary ZFS pools to the archive.pool ZFS pool.
For example:
elephant-backup backup -a archive.pool root.pool storage.pool
I provide the bash (Bourne Again Shell) script, example-getting-started.sh, which archives ZFS pools, and print the difference of the primary ZFS pools. Fit the bash script to your environment.
Misc
Helpful ZFS Commands
# Import/Export a ZFS pool
# Import a ZFS pool after we attach an external storage device.
zpool import <pool>
# Export the ZFS pool before we detach the device.
zpool export <pool>
# Display the status of all ZFS pools on Ubuntu.
zpool status
# Display the capacity of all ZFS pools on Ubuntu.
zpool list
# Display the capacity of all ZFS datasets on Ubuntu.
zfs list
# Display all the snapshots of all ZFS datasets on Ubuntu.
zfs list -t snapshot
# Display all the snapshots of the specified ZFS dataset with the date.
zfs list -r -t snapshot -o name,creation <pool>/<dataset>
# Display the variable size of the ZFS datasets.
zfs list -o space,mountpoint
# Mount/Unmount a ZFS pool
# Load all of the keys on Ubuntu
zfs load-key -a
# Mount all of the datasets on Ubuntu
zfs mount -a
# Mount a ZFS dataset with an encryption key
zfs mount -l <dataset>
# Unmount the ZFS dataset forcefully.
zfs unmount -u <dataset>
# Destroy a ZFS pool
zpool destroy <pool>
Helpful ZFS filesystem Properties
# Choose lz4 compression on a ZFS dataset property.
zfs set compression=lz4 <pool>/<dataset>
# Enable deduplication of a ZFS dataset property.
zfs set dedup=on <pool>/<dataset>
# Choose posixacl acl type on a ZFS dataset property.
zfs set acltype=posixacl <pool>/<dataset>
# Enable extended attribute stored on i-node of a ZFS dataset property.
zfs set xattr=sa <pool>/<dataset>
# Assign a mount point on a ZFS dataset property.
zfs set mountpoint=<a mount point> <pool>/<dataset>
# Disable auto-snapshot for zfs-auto-snapshot.
zfs set com.sun:auto-snapshot=false <pool>/<dataset>
System Directory
I recommend Ubuntu 22.04 Root on ZFS if we use system directories on the ZFS filesystem.
I found the following problems to mount ZFS filesystem after booting.
-
/etc
There are the configuration files for the ZFS filesystem in etc. At least, The process importing ZFS pools is dependent on /etc/zfs/zpool.cache. -
/var
/var contains the log files.journalctl
refers to the journal log after mounting the ZFS filesystem.
Security Consideration
Do not write the raw passphrase for loading the encryption key in the auto-mount service file. The passphrase is exposed to every user on your Ubuntu server.
The right way exists on this page.
Reference
- Elephant Backup
Elephant Backup is an easy backup CLI to archive any number of ZFS filesystems containing lots of snapshots to another ZFS filesystem.
https://github.com/patineboot/elephant-backup - Setup a ZFS storage pool
https://ubuntu.com/tutorials/setup-zfs-storage.pool#1-overview