ZFS is the file system you almost always meet in a homelab: Proxmox VE, TrueNAS and many Linux distributions offer it. Some choices made on day one cannot be changed later, so it pays to understand its structure before creating the pool.
File system and volume manager in one
With ext4 or XFS there are separate layers: a RAID controller or mdadm combines the disks, LVM creates volumes, and the file system organises files. ZFS does all of it. This lets it know exactly which blocks are in use, verify every read with a checksum and, after a failure, rebuild only the real data rather than the whole disk.
The hierarchy is:
- pool (zpool): the total space you see;
- vdevs: the groups of disks that make up the pool;
- datasets and zvols: logical subdivisions of the pool, respectively file systems and block devices (the latter used for virtual machine disks).
The key point: redundancy lives in the vdevs
The rule to memorise is that redundancy exists inside each vdev, not across vdevs. The pool spreads data over all vdevs; if a whole vdev fails, the entire pool is lost.
Typical vdev layouts:
| Type | Minimum disks | Failures tolerated | Notes |
|---|---|---|---|
| Mirror | 2 | all but one per vdev | excellent random performance, fast resilver |
| RAIDZ1 | 3 | 1 | not recommended with large disks |
| RAIDZ2 | 4 | 2 | good compromise for storage |
| RAIDZ3 | 5 | 3 | for very wide arrays |
For virtual machine storage, where random reads and writes matter, mirrors are almost always the best choice: several mirrors side by side behave like RAID 10. For an archive of large files such as photos and videos, RAIDZ2 gives more usable space for the same number of disks.
A classic mistake is adding a single disk as a new vdev to a RAIDZ2 pool: with no redundancy, it becomes the weak point of the whole pool.
Choices to make before creating the pool
ashift
The ashift parameter sets the minimum sector size ZFS uses. Nearly all modern disks have 4 KiB physical sectors, so ashift=12 is used. It is set when the vdev is created and never changes: a value that is too low on 4 KiB disks permanently hurts performance.
Expansion
Adding a vdev to a pool has always been possible. Widening a RAIDZ vdev with an extra disk arrived only in recent OpenZFS versions, and data already written keeps the old data-to-parity ratio until it is rewritten. A mirror, on the other hand, grows by replacing its disks one at a time with larger models. If you expect to grow, mirrors are more flexible.
Suitable disks
Hard drives using SMR recording can make a rebuild extremely slow or cause it to fail through timeouts. For ZFS choose CMR drives, checking the datasheet of the exact model.
ARC: why ZFS uses so much RAM
ZFS keeps an in-memory cache called the ARC (Adaptive Replacement Cache). Seeing a large share of RAM in use is normal: it is released when other processes need memory, though not always as quickly as you would like. On a virtualisation host it is wise to cap it, for example with the zfs_arc_max parameter, so the virtual machines have guaranteed room.
Extra devices should be used with judgement:
- L2ARC: an SSD read cache; it only makes sense when the ARC is already large and the workload rereads the same data often;
- SLOG: speeds up only synchronous writes, such as those from NFS or some databases; it must be an SSD with power-loss protection;
- special vdev: holds metadata on SSD; if it is lost, the pool is lost, so it must always be mirrored.
Features to use every day
- Compression:
lz4orzstdshould almost always be on. They reduce space used and often improve performance, since less data is read from disk. - Snapshots: near-free point-in-time copies, perfect before an upgrade. Tools such as Sanoid create and rotate hourly, daily and monthly snapshots.
- Replication:
zfs sendandzfs receivetransfer snapshots to another machine incrementally. It is the basis of a good backup, but the copy must live on different hardware.
An example of creating a pool from two mirrors:
zpool create -o ashift=12 tank \
mirror /dev/disk/by-id/ata-DISK1 /dev/disk/by-id/ata-DISK2 \
mirror /dev/disk/by-id/ata-DISK3 /dev/disk/by-id/ata-DISK4
zfs set compression=lz4 tank
The /dev/disk/by-id/ paths are used because names like /dev/sda can change between boots.
Scrubs and health checks
A scrub rereads all data in the pool and verifies its checksums, repairing errors thanks to redundancy. Many distributions already schedule one a month; in my work I consider that the bare minimum. The zpool status command shows the state of the vdevs and any read, write or checksum errors: check it, or hook it up to a notification system such as ZED, the ZFS event daemon, which can send email when a disk degrades.
In short
- Redundancy lives in the vdevs: if one vdev fails, the whole pool is lost.
- Mirrors for virtual machines, RAIDZ2 for storing large files.
ashift=12, CMR drives and by-id paths must be decided at creation time.- The ARC uses a lot of RAM by design; on a hypervisor it is worth capping.
- Compression on, automatic snapshots, replication to another machine and a monthly scrub.
