Skip to content

fde: persistent dm-snapshot fails after clean reboot on Alinux because offline resize precedes fsck #140

Description

@wdsun1008

Cryptpilot's persistent dm-snapshot path runs an offline resize2fs on every boot, before dracut gets to run its root filesystem check. With ext4/e2fsprogs 1.46.0 this can reject a clean filesystem after its first read/write mount: s_lastcheck < s_mtime is sufficient to produce Please run 'e2fsck -f /dev/mapper/rootfs' first.

Verified results (official RPMs, no Shelter/CA components)

Built on a fresh Alibaba Cloud Linux ECS using the documented Alinux image, official cryptpilot-enhance --mode partial (to retain key-only SSH for the probe), and the matching release's cryptpilot-convert + guest RPM. The enhance script is identical between v0.9.1 and v0.9.2. Guest kernel: 5.10.134-19.2.al8.x86_64; e2fsprogs-1.46.0-1.0.3.al8.x86_64.

Environment Guest/configuration First boot Normal reboot Forced stop/start
Real ECS ecs.g8i.2xlarge, imported QCOW2, UEFI, 40 GiB system disk Official 0.9.1, stable exec key, disk-persist, integrity=false Pass Fails with fsck request Fails with fsck request
Same real ECS setup Official 0.9.2, same persistent configuration Pass Fails with fsck request Fails with fsck request
Same real ECS setup Official 0.9.2 quick-start: disk, integrity=true Pass Pass; sentinel absent as expected for recreated delta Pass; sentinel absent
QEMU 10.0.6 from the upstream CI image ghcr.io/qemus/qemu:7.29, TCG, 4 vCPU/4 GiB Official 0.9.2, stable exec key, disk-persist, integrity=false Pass Same fsck failure Not tested

Normal reboot used systemctl --no-block reboot. Forced stop/start used independent fresh instances, wrote and synced the sentinel first, then ECS StopInstance(ForceStop=true) followed by StartInstance. Real ECS failures were confirmed from the platform's console screenshots, not merely an SSH timeout. The first-boot guest RPM version and binary digest were also recorded.

Console excerpt, reproduced by both releases:

Failed to start Cryptpilot FDE Service (before /sysroot mount).
0: Failed to resize ext4 filesystem on /dev/mapper/rootfs
cmd: LC_ALL="C" "resize2fs" "/dev/mapper/rootfs"
exit code: 1
resize2fs 1.46.0 (29-Jan-2020)
Please run 'e2fsck -f /dev/mapper/rootfs' first.

Dracut subsequently runs its filesystem check and can mount /sysroot, but the failed Cryptpilot unit/dependency has already left the boot in emergency mode. This is an ordering problem, not evidence that every affected filesystem is corrupt.

A separate controlled ext4 test also reproduced the rejection with Filesystem state: clean and lastcheck < mtime. A forced preen check (e2fsck -f -p, exit 0) made resize2fs succeed. This is not a substitute for testing a complete upstream fix.

Reproduction using the documented Alinux workflow

Use the official Alinux 3 image aliyun_3_x64_20G_nocloud_alibase_20251030.qcow2, the official release host/guest RPMs, and cryptpilot-enhance --mode partial --ssh-key ... followed by cryptpilot-convert. No Shelter or CA initrd/build components are used.

Use the documented stable exec key provider, with persistent delta:

[rootfs]
delta_location = "disk-persist"
[rootfs.encrypt.exec]
command = "echo"
args = ["-n", "AAAaaawewe222"]
[delta]
integrity = false
[delta.encrypt.exec]
command = "echo"
args = ["-n", "AAAaaawewe222"]

The key above is the public documentation's demonstration key, not a production credential. integrity=false isolates the filesystem-check issue from integrity initialization.

cryptpilot-enhance --mode partial --image alinux3-hardened.qcow2 --ssh-key guest-key.pub
cryptpilot-convert --in alinux3-hardened.qcow2 --out encrypted.qcow2 \
  -c config --rootfs-passphrase AAAaaawewe222 \
  --package cryptpilot-fde-guest-VERSION-1.al8.x86_64.rpm

Boot the converted disk, write and sync a sentinel under /var/lib, then execute systemctl reboot and retain the same disk. Check that the boot ID changes and the sentinel survives. The official quick-start delta_location="disk" has different semantics: it recreates the delta on every boot and cannot serve as a persistence control.

Code path and suggested fix

before_sysroot.rs at v0.9.2 builds the snapshot chain, then calls resize_ext4_filesystem() unconditionally. This helper runs tune2fs -O ^read-only followed by resize2fs, without a filesystem check. In dracut, the Cryptpilot service runs before initrd-root-device.target; the normal root fsck happens only after Cryptpilot has completed, so it cannot recover from this earlier failure.

Suggested minimal correction:

  • Check/recover the unmounted writable root mapper before offline resize, for example using e2fsck -f -p with explicit exit-status handling (0/1 may proceed; errors or a required reboot must not be ignored).
  • Explicitly include the checker in the guest's dracut runtime and declare its dependency.
  • Optionally avoid unnecessary resizing when the device size has not changed, but retain a correct filesystem recovery path for unclean shutdown and actual disk growth. Do not bypass the check with resize2fs -f.
  • Test the encrypted COW/sentinel across clean reboot, forced power loss, and disk growth; ensure uncorrectable errors fail closed.

The same helper is used by persistent overlayfs delta resizing, so that path should be covered too; the reproductions above use dm-snapshot.

The ext4 check is explicit in e2fsprogs v1.46.0 resize/main.c, and occurs before the no-size-change early return.

Coverage gap

tests/test-convert.sh configures [delta.encrypt.otp], even for the disk-persist matrix entry. The temporary key provider recreates the delta, so this does not exercise persistence. The boot check also succeeds at the first login prompt. Please add a stable-key, same-disk, second-boot test that verifies data preservation.

The quick-start currently describes delta_location="disk" as retaining data after reboot, which conflicts with its implementation and the configuration documentation. Please correct that sentence so it does not hide the need for a stable-key disk-persist regression test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions