Skip to content

file: recover legacy checksums and validate shadow header fallback - #118

Merged
edsiper merged 7 commits into
masterfrom
fix/checksum-recovery-validation
Oct 31, 2026
Merged

edsiper merged 7 commits into
masterfrom
fix/checksum-recovery-validation

Conversation

@edsiper

@edsiper edsiper commented Oct 31, 2026

Copy link
Copy Markdown
Member

Stored chunks can fail checksum validation when an older writer leaves an unfinished CRC or uses the legacy 64-bit big-endian checksum layout. Recover those layouts after validating the content, and add a shadow content-length/CRC tuple in the existing header padding as a fallback when the primary header fails validation. Writable recovery repairs the primary header; read-only recovery leaves the file unchanged.

A valid primary tuple always takes precedence over the shadow. Otherwise, an older writer can append and sync 18 bytes while leaving a valid 9-byte shadow, and reopening the chunk would silently restore the 9-byte length. The same ambiguity can undo a truncation. Without a generation marker, two valid tuples cannot establish which is newer, so recovery preserves the primary even if the shadow describes additional bytes. This is a recovery fallback, not an atomic commit protocol.

The changes also persist zero-length write_at truncations correctly and validate shadow lengths by subtraction to prevent 32-bit arithmetic overflow. A crafted UINT32_MAX shadow length previously passed the size check under Emscripten and could be accepted with a checksum over the wrapped length.

Coverage includes legacy format fixtures, read-only recovery, append/truncation with stale shadows, corrupt payloads and headers, maximum shadow lengths, and Fluent Bit storage lifecycle behavior.

Validation:

  • Native GCC debug build: all 8 CTest suites passed.
  • Emscripten wasm32 build: all 9 CTest suites passed.
  • Clang ASan/UBSan with leak detection: all 8 CTest suites passed. Function-type sanitization was disabled because the existing Acutest callbacks use legacy function prototypes; all other enabled sanitizer checks remained active.
  • Strict Valgrind on the complete filesystem suite: zero errors and zero leaked bytes.
  • Cross-version reproduction using the actual origin/master library: create 9 bytes with this branch, append/sync another 9 bytes with upstream, reopen with this branch; all 18 bytes preserved.
  • The five primary-precedence regressions failed before the fix. The crafted maximum-length regression failed under wasm32 before the bounds fix.
  • git diff --check passed.

Windows and macOS execution is left to the repository's PR CI.

Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-31T21:47:56.430246Z a614c86 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@edsiper
edsiper merged commit 7225b87 into master Oct 31, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant