ZFS Parameters and Optimizations

The TL;DR

For root pool/dataset:

zfs set atime=off xattr=sa compression=lz4 recordsize=128k $POOLNAME

For datasets with ‘hot’ data:

zfs set atime=off dnodesize=auto xattr=sa compression=lz4 recordsize=128k $POOLNAME

For purely ‘cold’ storage:

zfs set atime=off dnodesize=auto xattr=sa compression=zstd recordsize=1M $POOLNAME

General Recommendations

Pool-level

• When adding vdevs, I explicitly set ashift to at least 12, maybe 13 for SSDs.
  • ashift=12 corresponds to 4K block size, ashift=13 is 8K; default is ashift=9 (512b).
• For HDD pools, add a special vdev on a fast SSD that is 0.3% (3/1000) of the total size of the HDDs.
  • If using special_small_blocks, this will need to be much higher.¹
  • If using RAID, the special vdev can be of a different topology than the main array.
• For max ARC size, there is a loose consensus of at least 1 + N GiB, where N is every TiB of useable storage space.
  • This is RAM dedicated only to ARC, not counting the host OS and other apps. If you have a lot of free RAM, just give it to ARC.
  • ‘Useable storage space’ refers to space used after factoring in RAID, not total HDD space.

Dataset-level

These are the properties I would typically set for every dataset:

atime = off
dnodesize = auto
xattr = sa

For compatibility reasons, dnodesize should be left at legacy when a ZFS dataset is used as the boot pool for the host OS (i.e. Proxmox). I believe this is only a problem when using GRUB as the bootloader; systemd-boot could be more compatible but I haven’t tested it and would rather stay safe.

dnodesize = legacy # ONLY for boot pool

For general ‘cold’ storage (i.e. media like pictures, videos, etc.) I would add:

compression = zstd
recordsize = 1M

Explanations

Ashift

Quoting High-Availability.com²:

The ashift property determines the block allocation size that ZFS will use per vdev (not per pool as is sometimes mistakenly thought). Ideally this value should be set to the sector size of the underlying physical device (the sector size being the smallest physical unit that can be read or written from/to that device).

Traditionally hard drives had a sector size of 512 bytes; nowadays most drives come with a 4KiB sector size and some even with an 8KiB sector size (for example modern SSDs).

Supposedly ZFS can automatically detect the sector size of the drive hardware and adjust ashift accordingly, but IMO its unrelieable; setting it explicitly is also more reassuring.

Note that ashift CAN NOT be changed after pool creation; it is immutable once set.

Atime

Keep atime=off to minimize writes when accessing files; I don’t really care about keeping track of access times either.

Compression

It’s generally recommended to enable compression on every dataset because you can save space for minimal performance loss.³ The modern default is LZ4, so compression=on is basically the same as compression=lz4. There is also ZSTD, which has better compression but is slightly slower. The general consensus is to use lz4 if you need the higher speed, and zstd for everything else. You can also set higher or lower compression levels for ZSTD for further space savings or speed.

Even on datasets with mostly incompressible or already compressible data, compression is beneficial since it can automatically detect and skip this data and only apply the compression on the data that needs it.⁴

It’s also important to note that enabling compression after a dataset has already been created does not retroactively compress the data that has already been written, so it’s best to set compression during dataset creation.

Dnodesize and Xattr

What is a dnode?

A ZFS dnode is a data structure which represents an object. An object can be a ZPL file or directory, a ZVOL volume, or several other types of internal metadata. A ZPL-type dnode serves a similar function to an inode in UFS and other filesystems.⁵

From official OpenZFS documentation regarding ZFS properties⁶:

System-attribute-based xattrs can be enabled by setting the value to sa (default and equal to on ) . The key advantage of this type of xattr is improved performance. Storing extended attributes as system attributes significantly decreases the amount of disk I/O required.

Consider setting dnodesize to auto if the dataset uses the xattr=sa property setting and the workload makes heavy use of extended attributes. This may be applicable to SELinux-enabled systems, Lustre servers, and Samba servers, for example. Literal values are supported for cases where the optimal size is known in advance and for performance testing.

Recordsize

The recordsize property gives the maximum size of a logical block in a ZFS dataset. Unlike many other file systems, ZFS has a variable record size, meaning files are stored as either a single block of varying sizes, or multiple blocks of recordsize blocks.⁷

Bigger recordsizes (1M or higher) are better for large, contiguous files like compressed archives and videos, and are also more compressible. However, for datasets with lots of small files, larger recordsizes can cause disk space to balloon out of proportion, as something like a 1 KB file will be saved as e.g. 1 MB on the actual disk. Therefore, for mixed use-cases, 128K is generally recommended. For database folders in particular, you want to set the recordsize equal to the database block size (16K–64K).

How to Set Properties

While a handful of properties cannot be changed after dataset creation, the vast majority of them can be modified. To set properties at creation, we add -o <property=value> to the zfs create command for every property we want to set.

zfs create -o property1=value -o property2=value dataset

To modify the properties on an existing dataset, we use the zfs set command:

zfs set property1=value property2=value dataset

There is also the zfs inherit command. When child datasets are created they should automatically inherit properties from their parents, so this is more useful to ‘reset’ that inheritance after manually setting a property.

zfs inherit property dataset

Finally, we can use zfs get to check the current value for the given properties for the given dataset(s)

zfs get property1,property dataset1 dataset2

Sources

1. https://forums.truenas.com/t/special-vdev-svdev-planning-sizing-and-considerations/5086 ↩︎
2. https://www.high-availability.com/docs/ZFS-Tuning-Guide/#alignment-shift-ashiftn ↩︎
3. https://web.archive.org/web/20250112222606/https://www.reddit.com/r/zfs/comments/svnycx/a_simple_real_world_zfs_compression_speed_an/ ↩︎
4. https://www.truenas.com/community/threads/game-changer-for-zfs-2-2-intelligent-compression-that-saves-cpu-and-space.113247/ ↩︎
5. https://openzfs.org/wiki/Documentation/DnodeSync#What_is_a_dnode? ↩︎
6. https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html ↩︎
7. https://www.high-availability.com/docs/ZFS-Tuning-Guide/#record-size-recordsizen ↩︎