Part 5 - Feature Flags in Code¶

Feature flags are OpenZFS's compatibility contract for on-disk format evolution.

Why This Chapter Matters¶

Feature flags are where three concerns meet:

on-disk compatibility
runtime behavior gates in kernel code
user-visible policy (feature@..., compatibility, zpool upgrade)

If you want to add or reason about a format-affecting change, this is the system you have to understand.

Scope¶

This chapter focuses on:

how features are declared and registered (spa_feature_t, zfeature_info_t, zpool_feature_init())
how state is represented on disk (disabled/enabled/active via refcount in MOS ZAPs)
how import compatibility is enforced (label precheck plus MOS checks)
how userland policy (compatibility) shapes create/upgrade behavior
a practical "add a feature" workflow

Line-number anchors in this chapter are based on the reference revision listed in README.md. If you are on a newer OpenZFS commit, expect drift and relocate by symbol names.

Diagram 1 - Big Picture¶

flowchart LR
    A["zpool_feature_init()<br/>register all features"] --> B["spa_feature_table[SPA_FEATURES]"]
    B --> C["zpool create/upgrade/set<br/>feature@...=enabled"]
    C --> D["spa_feature_enable()"]
    D --> E["feature_enable_sync()<br/>enable deps + write ZAPs"]

    E --> F["MOS: features_for_read"]
    E --> G["MOS: features_for_write"]
    E --> H["MOS: feature_descriptions"]

    F --> I["pool import/open checks"]
    G --> I
    I --> J["read-only allowed?"]
    I --> K["read-write allowed?"]

Core Data Structures¶

Primary anchors:

feature enum: include/zfeature_common.h:44
flags: include/zfeature_common.h:98
feature type enum: include/zfeature_common.h:120
descriptor: include/zfeature_common.h:126
global feature table declaration: include/zfeature_common.h:142
global feature table definition: module/zcommon/zfeature_common.c:54

The central table is:

zfeature_info_t spa_feature_table[SPA_FEATURES];

spa_feature_t is the internal index. fi_guid is the on-disk identity. That distinction matters:

enum order is code-local bookkeeping
GUID string is the persistent compatibility contract

Registration Flow in `zpool_feature_init()`¶

Primary anchors:

zfeature_register(...): module/zcommon/zfeature_common.c:323
zpool_feature_init(...): module/zcommon/zfeature_common.c:369

Registration pattern:

zfeature_register(SPA_FEATURE_ASYNC_DESTROY,
    "com.delphix:async_destroy", "async_destroy",
    "Destroy filesystems asynchronously.",
    ZFEATURE_FLAG_READONLY_COMPAT, ZFEATURE_TYPE_BOOLEAN, NULL,
    sfeatures);

Each call sets:

enum ID (SPA_FEATURE_*)
on-disk GUID (org.vendor:feature)
user-visible short name (feature@short_name)
human description
behavior flags
feature type
dependency list (... SPA_FEATURE_NONE)

zfeature_type_t currently has two values:

ZFEATURE_TYPE_BOOLEAN
ZFEATURE_TYPE_UINT64_ARRAY (for example SPA_FEATURE_REDACTED_DATASETS)

Concrete examples worth reading:

simple read-only compatible: SPA_FEATURE_ASYNC_DESTROY (module/zcommon/zfeature_common.c:374)
activate-at-enable: SPA_FEATURE_LZ4_COMPRESS (module/zcommon/zfeature_common.c:386)
dependency on enabled-txg: SPA_FEATURE_HOLE_BIRTH (module/zcommon/zfeature_common.c:414)
opt-out from blanket zpool upgrade: SPA_FEATURE_DYNAMIC_GANG_HEADER (module/zcommon/zfeature_common.c:795)

Flags That Drive Behavior¶

Primary definitions: include/zfeature_common.h:100.

ZFEATURE_FLAG_READONLY_COMPAT: unsupported feature may still allow read-only import.
ZFEATURE_FLAG_MOS: feature is required to read the MOS/feature list at all; when active it is written into the vdev label's features_for_read so support can be checked before pool open.
ZFEATURE_FLAG_ACTIVATE_ON_ENABLE: initial refcount starts at 1 when enabled.
ZFEATURE_FLAG_PER_DATASET: feature usage is tracked per dataset context.
ZFEATURE_FLAG_NO_UPGRADE: zpool upgrade does not auto-enable it.

Important registration constraint:

zfeature_register() asserts READONLY_COMPAT and MOS are not combined (module/zcommon/zfeature_common.c:334).

State Model: Disabled, Enabled, Active¶

The canonical explanation is in the header comment of module/zfs/zfeature.c (starting near module/zfs/zfeature.c:40).

The state machine is refcount-based:

disabled: no ZAP entry
enabled: ZAP entry exists, refcount = 0
active: ZAP entry exists, refcount > 0

This is exactly how spa_feature_is_enabled() and spa_feature_is_active() behave (see module/zfs/zfeature.c, functions spa_feature_is_enabled and spa_feature_is_active).

Diagram 2 - Feature Lifecycle¶

stateDiagram-v2
    [*] --> Disabled
    Disabled --> Enabled: spa_feature_enable()
    Enabled --> Active: spa_feature_incr() or ACTIVATE_ON_ENABLE
    Active --> Enabled: spa_feature_decr() to 0

    note right of Enabled
      refcount == 0
      feature is allowed,
      but not currently in-use
    end note

    note right of Active
      refcount > 0
      on-disk effects present
    end note

Where Feature State Lives On Disk¶

Constants:

DMU_POOL_FEATURES_FOR_WRITE: include/sys/dmu.h:384
DMU_POOL_FEATURES_FOR_READ: include/sys/dmu.h:385
DMU_POOL_FEATURE_DESCRIPTIONS: include/sys/dmu.h:386
DMU_POOL_FEATURE_ENABLED_TXG: include/sys/dmu.h:387

Creation path:

spa_feature_create_zap_objects(...) in module/zfs/zfeature.c (around the mid-400s in the reference revision)

Mutation path:

feature_sync(...) writes refcount entry: module/zfs/zfeature.c:305
feature_enable_sync(...) creates description entry and initial state: module/zfs/zfeature.c:340

Placement rule:

if feature has READONLY_COMPAT, it is stored in features_for_write
otherwise it is stored in features_for_read

That rule is visible in both feature_sync() and feature_enable_sync() via the zapobj selection.

Diagram 3 - MOS Feature Objects¶

flowchart TD
    DIR["DMU_POOL_DIRECTORY_OBJECT"] --> R["features_for_read<br/>GUID -> refcount"]
    DIR --> W["features_for_write<br/>GUID -> refcount"]
    DIR --> D["feature_descriptions<br/>GUID -> string"]
    DIR --> T["feature_enabled_txg<br/>GUID -> txg (optional)"]

    R --> IMP1["required for read-open support checks"]
    W --> IMP2["required for write-open support checks"]

Enable Path and Dependency Handling¶

Primary control entry points:

spa_feature_enable(...) in module/zfs/zfeature.c
spa_feature_incr(...) in module/zfs/zfeature.c
spa_feature_decr(...) in module/zfs/zfeature.c

feature_enable_sync() does the heavy lifting:

chooses initial refcount (1 with ACTIVATE_ON_ENABLE, else 0)
returns early if already enabled
recursively enables dependencies (feature->fi_depends)
writes feature_descriptions entry
writes initial refcount in read/write feature ZAP
records the enabling txg if the enabled_txg feature is itself enabled (the gate is spa_feature_is_enabled(), not activity)

Anchors: module/zfs/zfeature.c:340 through module/zfs/zfeature.c:382.

Dependency query helper:

zfeature_depends_on(...): module/zcommon/zfeature_common.c:149

Per-Dataset Feature Activation¶

ZFEATURE_FLAG_PER_DATASET features track activity per dataset, not just per pool. The API is declared in include/sys/dsl_dataset.h:500:

dsl_dataset_activate_feature(): module/zfs/dsl_dataset.c:1124
dsl_dataset_deactivate_feature(): module/zfs/dsl_dataset.c:1170
dsl_dataset_feature_is_active(): module/zfs/dsl_dataset.c:1099

How it works (all in module/zfs/dsl_dataset.c):

dsl_dataset_activate_feature() VERIFYs the PER_DATASET flag, calls spa_feature_incr() once, zapifies the dataset object (dmu_object_zapify()), and zap_add()s an entry keyed by the feature GUID (fi_guid) directly on the dataset object.
So the pool-level refcount equals the number of datasets with the feature active; the per-dataset state is the GUID-keyed ZAP entry on the dataset object itself.
Entry payload follows fi_type: BOOLEAN stores a single zero uint64, UINT64_ARRAY stores the array (redacted datasets).
Deactivation is the mirror image: zap_remove() plus spa_feature_decr(), and the in-memory ds->ds_feature[f] slot is cleared.
At dataset open, load_zfeature() (module/zfs/dsl_dataset.c:402) populates ds->ds_feature[]; dsl_dataset_feature_is_active() only consults that in-memory state.

Because activation zapifies the dataset object, PER_DATASET features have a hard dependency requirement covered in the add-a-feature steps below.

Import-Time Compatibility Enforcement¶

Import/open checks are two-stage:

pre-MOS check from label features_for_read list
MOS check via spa_features_check() for read and optionally write mode

Primary anchors:

early label feature check: module/zfs/spa.c:4840
load-time feature reconciliation: spa_ld_check_features(...) at module/zfs/spa.c:5142
core validator: spa_features_check(...) at module/zfs/zfeature.c:175

What this means operationally:

unsupported active feature in read set blocks pool open
unsupported active feature only in write set can still allow read-only open
spa_ld_check_features() also builds userland-visible nvlists of enabled/unsupported features

Diagram 4 - Import Decision Model¶

flowchart TD
    A["select uberblock + read label"] --> B{"label features_for_read has unknown active feature?"}
    B -->|Yes| X["fail import/open"]
    B -->|No| C["open MOS + load feature ZAP object IDs"]
    C --> D["spa_features_check(for_read)"]
    D --> E["spa_features_check(for_write) if needed"]
    E --> F{"for_write unsupported?"}
    F -->|No| G["allow read-write open"]
    F -->|Yes| H["allow read-only only (if read path is compatible)"]

Userland State and Policy¶

How `feature@...` strings are computed¶

zpool_prop_get_feature() maps raw refcount to property text:

absent -> disabled
present, 0 -> enabled
present, >0 -> active

Anchor: lib/libzfs/libzfs_pool.c:1145.

Compatibility property parsing¶

zpool_load_compat(...) parses the compatibility property and feature files:

unset/off/empty -> all features allowed
legacy -> all features disallowed
comma list of file names -> intersection of listed feature sets
search paths include /etc/zfs/compatibility.d and /usr/share/zfs/compatibility.d

Anchor: lib/libzfs/libzfs_pool.c:5176.

Where compatibility is enforced in workflows¶

zpool create applies compatibility set and auto-adds allowed features: cmd/zpool/zpool_main.c:2236
zpool upgrade skips NO_UPGRADE features and respects compatibility: cmd/zpool/zpool_main.c:11486

Practical Inspection Commands¶

Quick commands that are usually high-signal:

# Show per-feature state (disabled/enabled/active) for a pool
zpool get all POOL | rg 'feature@'

# See pool compatibility policy
zpool get compatibility POOL

# List importable pools; the scan reports unsupported-feature status
zpool import

When tracing code behavior, these anchors are usually the best first breakpoints:

module/zcommon/zfeature_common.c: zpool_feature_init
module/zfs/zfeature.c: feature_enable_sync
module/zfs/zfeature.c: feature_sync
module/zfs/zfeature.c: spa_features_check
module/zfs/spa.c: spa_ld_check_features
lib/libzfs/libzfs_pool.c: zpool_prop_get_feature

How To Add a New Feature Flag¶

This is the shortest reliable workflow for contributors.

Add enum value in include/zfeature_common.h just before SPA_FEATURES.
Register it in zpool_feature_init() in module/zcommon/zfeature_common.c.
Choose flags deliberately.
Add dependency list ending with SPA_FEATURE_NONE. Hard constraint: a ZFEATURE_FLAG_PER_DATASET feature must list SPA_FEATURE_EXTENSIBLE_DATASET in its dependency list; zfeature_register() VERIFYs this (module/zcommon/zfeature_common.c:341).
Gate behavior with spa_feature_is_enabled() and increment/decrement usage with spa_feature_incr()/spa_feature_decr() in syncing context.
Add/extend tests in the owning subsystem's suite under tests/zfs-tests/tests/functional/. Recent feature commits do not add tests under tests/functional/features/; that directory holds only the legacy async_destroy and large_dnode tests.
Update man/man7/zpool-features.7.
Consider compatibility file implications in cmd/zpool/compatibility.d/ (installed to /usr/share/zfs/compatibility.d/; /etc/zfs/compatibility.d/ is only the admin-local override path).
Add the feature@short_name string to the properties array in tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg (the feature-name list the zpool get CLI tests check; the array starts at tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg:34).
Regenerate the libzfs ABI dump with make storeabi and commit the updated lib/libzfs/libzfs.abi (target defined in lib/Makefile.am:107; requires libabigail >= 2.0.0 and only runs on x86_64). CI runs the matching checkabi target, so a stale .abi file fails CI.

Steps 9 and 10 are easy to verify empirically: recent feature commits such as large_microzap (224393a32) and physical rewrites (60f714e6e) each touch exactly these two files alongside the code change, and neither touches tests/functional/features/ (git show --stat 224393a32).

Two design rules from module/zfs/zfeature.c comments are worth keeping front-and-center:

do not assume enable-time metadata initialization has happened
treat "enabled but not active" as a first-class state your code can encounter

Key Takeaways¶

Feature flags are refcounted, not binary toggles.
GUID strings are the durable compatibility identity.
Import compatibility is enforced in two stages: label-first, MOS-second.
compatibility is policy; feature ZAP state is reality.
NO_UPGRADE features intentionally require explicit operator intent.

Next¶

-> Part 6 - Contributing Guide: build, test, debug tools, and patch workflow.

Part 5 - Feature Flags in Code¶

Why This Chapter Matters¶

Scope¶

Diagram 1 - Big Picture¶

Core Data Structures¶

Registration Flow in zpool_feature_init()¶

Flags That Drive Behavior¶

State Model: Disabled, Enabled, Active¶

Diagram 2 - Feature Lifecycle¶

Where Feature State Lives On Disk¶

Diagram 3 - MOS Feature Objects¶

Enable Path and Dependency Handling¶

Per-Dataset Feature Activation¶

Import-Time Compatibility Enforcement¶

Diagram 4 - Import Decision Model¶

Userland State and Policy¶

How feature@... strings are computed¶

Compatibility property parsing¶

Where compatibility is enforced in workflows¶

Practical Inspection Commands¶

How To Add a New Feature Flag¶

Key Takeaways¶

Next¶

Registration Flow in `zpool_feature_init()`¶

How `feature@...` strings are computed¶