feat: diff generated corpus output against an earlier ref - #73
Merged
Conversation
Changing the generator can silently move the code emitted for the 58 real-world specs in specs/, and nothing showed what moved. Generation is byte-deterministic and the specs are pinned, so the old output does not need to be stored — it is rebuilt on demand. The full corpus regenerates in about 30 seconds, against 300 MB if it were checked in. scripts/gen-diff.sh <base-ref> builds the generator at that ref in a throwaway worktree, regenerates both corpora, and reports per-spec churn plus any public item that appeared or disappeared; full diffs land in tmp/gen-diff/report/. The base side is cached per commit. tests/corpus-manifest.txt is the committed tripwire: one hashed line per generated file (21 KB), verified in CI by corpus-manifest.sh --check, so a change that moves real-world output cannot land unnoticed. The "Generated by openapi-to-rust vX.Y.Z" stamp is normalized away so a version bump alone never touches it. Both share scripts/lib/corpus.sh, which owns everything that decides the generated bytes — the per-spec TOML mirrors spec-compile.sh, so the manifest describes the same code CI compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X3eDyWLLqUtCzynJeLMAXL
A pass/fail tripwire tells a reviewer that real-world output moved but not what moved. The corpus-diff job regenerates the corpus at the base of the pull request and at its head, puts the per-spec table in the job summary, and uploads the full diffs as an artifact. That job already generates the head corpus, so the manifest gate now runs off it (`--check --from tmp/gen-diff/head`) instead of generating a third copy; the standalone manifest job is left to pushes and scheduled runs. Both sides also build into one cargo target dir now, so the dependency graph compiles once per run rather than twice. Fixes a cache bug the tool found in itself: the base corpus was keyed on the commit alone, so a corpus left behind by an earlier GEN_DIFF_SPECS run was reused for a wider run and every spec it lacked read as new. The marker now records which specs the cache holds, and the comparison walks the requested spec list rather than whatever directories happen to exist. Per-spec diffs are capped at 5 MB (GEN_DIFF_MAX_DIFF_BYTES=0 to disable) so a sweeping change cannot produce a report nobody can open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X3eDyWLLqUtCzynJeLMAXL
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
CI caught the manifest failing on a pull request that changes no generated code. The lines were identical — same bytes, same hashes — but in a different order: `sort` collates by locale, so macOS orders anthropic/REQUIRED_DEPS.toml after anthropic/client.rs and a C-locale runner orders it before. LC_ALL=C in the shared library fixes every ordering the tooling produces, the manifest included; it is regenerated here in byte order. Worth recording what the failure also proved: the generated code itself is byte-identical on macOS and Linux, and identical between debug and release builds of the generator. Every hash matched across all three. spec-compile.sh wrote its own copy of the generator config, identical to the corpus tooling's by copy-paste alone. Both now call corpus_config, so the code the manifest hashes cannot drift from the code the compile gate checks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X3eDyWLLqUtCzynJeLMAXL
A warm local run goes from 11m40s to 7m25s: nextest runs each test in its own process, where the built-in harness runs one process per test binary and serializes across binaries. All 679 tests pass unchanged, so nothing depended on state shared within a binary. Doctests still go through `cargo test --doc`. nextest cannot run them, and this crate has seven — swapping the command outright would have dropped them silently. The conformance report step converts as-is: each of those tests builds its results and writes the report inside one test function, so process isolation does not split it. .config/nextest.toml raises the slow-test threshold to 120s, since tests that generate a crate and compile it legitimately run for minutes (the longest takes 428s), and adds a ci profile that reports every failure in one run instead of stopping at the first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X3eDyWLLqUtCzynJeLMAXL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changing the generator can silently move the code emitted for the 58 real-world specs in
specs/, and nothing showed what moved.Generation is byte-deterministic and the specs are pinned, so the old output does not need to be stored — it is rebuilt on demand. The full corpus regenerates in about 30 seconds locally, against 300 MB if it were checked in (microsoft-graph alone is 113 MB).
scripts/gen-diff.sh <base-ref>builds the generator at that ref in a throwaway worktree, regenerates both corpora, and reports per-spec churn plus any public item that appeared or disappeared. Full diffs land intmp/gen-diff/report/, capped at 5 MB each. The base side is cached per commit, keyed on the spec set it holds.tests/corpus-manifest.txt(21 KB) is the committed tripwire: one hashed line per generated file. TheGenerated by openapi-to-rust vX.Y.Zstamp is normalized away, so a version bump alone never touches it.CI: pull requests get the
corpus-diffjob — it diffs against the PR base, puts the per-spec table in the job summary, uploads the full diffs as an artifact, and then verifies the manifest against the corpus it already generated. Pushes tomainand scheduled runs verify the manifest on its own.Both cover types and client output. Server scaffolding is generated from per-spec
[server].operationsselectors, so it has no uniform corpus pass yet.For reference,
gen-diff.sh v0.15.0on the merged union fixes reports 24 specs changed, the largest being 3,688 microsoft-graph aliases moving fromserde_json::ValuetoBTreeMap<String, serde_json::Value>with no public item added or removed.🤖 Generated with Claude Code
https://claude.ai/code/session_01X3eDyWLLqUtCzynJeLMAXL