Repository navigation
Feat/phase 2 evaluation readiness - #19
Merged
Merged
Conversation
Phase 2.1 of docs/roadmap-production-ready.md: drop in an external implementation (a library under validation, an AI-generated candidate, a rewrite) and evaluate it against a fixed contract with a fixed suite, without touching the code under test. - examples/eval-harness/ (new, standalone workspace — empty [workspace] table, the fuzz/ detachment pattern, so the default 'cargo test --workspace' is untouched): the Candidate contract (pinned to reference semantics), the FnCandidate adapter, the GOLDEN_CASES table, and the three-part suite — golden cases, proptest properties, and a trybuild compile-fail check (a candidate that doesn't implement the contract fails to build loudly instead of at test time). - examples/sample-candidate/ (new): reference candidate — a hand-rolled digit-loop parser deliberately different from the reference semantics, so equivalence is checked, not assumed. - just eval <path> / scripts/eval-harness.sh: copies both crates into a scratch workspace, substitutes the __HARNESS_PATH__/__CANDIDATE_PATH__ placeholders, runs the suite; keeps the scratch workspace (with its path printed) on failure. Guards: clear errors for a missing Cargo.toml and for a candidate not named 'candidate' (the name is part of the contract). - CI's adoption job gained a third step evaluating the sample candidate end-to-end, catching harness rot the same way the two adoption legs catch adoption rot. - The dependency cycle (harness dev-depends on the candidate) is deliberate and legal in Cargo — it's what lets the suite test the candidate while the candidate implements the harness's trait. - .gitignore: standalone example targets and the Phase 1.3 report outputs (nextest.json, lcov.info, clippy.json, deny.json, mutants.json, mutants.out/) — previously running those recipes littered the repo. - New doc docs/evaluation-harness.md; examples/README.md documents the standalone-example-workspace pattern. Verified against real execution: the sample candidate passes the full suite (golden 1/1, properties 4/4, compile-fail 1/1, doc-test 1/1); a deliberately broken candidate (accepts '-') fails the golden cases with a discriminating message and exit 101, scratch workspace kept. Two bugs caught pre-commit: cargo won't match an underscore dependency key against a hyphenated package for path deps, and a rustdoc link to ../../justfile would break doc-check.
…se 2.2) Phase 2.2 of docs/roadmap-production-ready.md: the "I rewrote the hot path — prove it still behaves identically" primitive, as code. - crates/core-tests/src/differential.rs (new, ungated, std-only): assert_impls_agree(reference, candidate, inputs) asserts two implementations of the same function agree on every fixed input, naming the input and both results on disagreement. Worked examples ship as real tests — closed-form sum vs loop sum, naive trial-division primality vs sqrt-bounded optimization. - crates/fuzz-tests/src/differential.rs (new, behind the existing 'fuzz' feature — no new dependency): the property-based companion, same agreement check over proptest-generated inputs with automatic shrinking, plus the same two worked pairs over generated inputs. - Docs: best-practices.md gains the differential bullet under "Match the test to the failure mode"; docs/fuzzing.md's differential-fuzzing section now points at the in-crate helpers as the first step before escalating to a continuous cargo-fuzz target; README module responsibilities updated. Verified against real execution: both helpers pass agreeing pairs (10 unit + 1 doc-test with the fuzz feature) and both *catch* a deliberately disagreeing pair — the fixed-input form with a discriminating panic message, the property form with a shrunk minimal input. Miri covers the new core-tests module (8 + 5 tests interpreted, ~24s for the primality test's naive loop — acceptable for the nightly-only job). Three compile errors caught pre-commit: prop_assert_eq! can't capture locals in its format string (macro expansion), TestRunner isn't in proptest's prelude, and the macro moves the compared values so they can't be reused as format args.
Phase 2.3 of docs/roadmap-production-ready.md — mutation polish that stays exploratory by design. - New justfile recipes: 'mutants-crate <crate>' (cargo mutants -p), 'mutants-file <glob>' (--file), 'mutants-diff <diff>' (--in-diff) — each variadic, so they compose with extra cargo-mutants arguments and the --json/-o machine-readable output from Phase 1.3. - docs/best-practices.md: the mutation section gains "Scoping it to what changed" (the three recipes, what each flag does) and a documented *soft* reporting pattern — a complete opt-in workflow that runs mutants scoped to a PR's diff and posts the kill rate as a PR comment, with continue-on-error: true on every step so nothing can block a merge. Deliberately NOT added to RustForge's own CI: a fresh copy of the template gets no failing jobs it didn't ask for, and the exploratory stance is unchanged. - docs/machine-readable-outputs.md points at the scoped recipes; README recipe list updated. Verified against real execution: 'cargo mutants -p core-tests' runs scoped (23 mutants, 42s — exit 2 is cargo-mutants' missed-mutants signal, same as the existing 'mutants' recipe; informational), and 'cargo mutants --file <path>' filters the list to exactly that file's mutants.
Owner
Author
Added
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Checklist
cargo fmt --all --checkpassescargo clippy --workspace --all-targets --all-features -- -D warningspassescargo test --workspace --all-featurespassescargo test -p syntax-tests --features compile-failpasses on stable, ifsyntax-tests' UI fixtures changedREADME.md/docs/adoption.mdupdated if this adds or changes a category, feature flag, or CI jobCHANGELOG.mdupdated under[Unreleased]