Replay V1 against the predecessor pilot - #3608
Conversation
Freeze the V1 skill package and replay the synthetic, historical, and current-source targets from the exploratory predecessor pilot. Preserve the manifests, raw reports, scores, and limitations needed to compare the replay with the earlier procedural run. The replay recovers every known synthetic and historical defect and accepts every fixed control. It improves version applicability, literal contract discovery, and exposure of reconstructed proofs, while still missing an admissible indirect Copy and UnsafeCell derivation on the current-source challenge. Treat this as legacy confirmation rather than release evidence: the replay is not byte-identical to the pilot, uses one replicate per target, and retains the pilot's procedural-isolation and unavailable model-identity limitations. gherrit-pr-id: Gup6qexxvahgx6uken5vngpapqsaio22h
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## G6f4nmapfoz5ur4camwh2yhyazexc2lct #3608 +/- ##
==================================================================
Coverage 91.85% 91.85%
==================================================================
Files 20 20
Lines 6093 6093
==================================================================
Hits 5597 5597
Misses 496 496 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6829891a84
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - Skill package tree digest: | ||
| `d97b9ace50109216614fbb7c975ac9c97508bfa928381d247869699593a2bcdd` |
There was a problem hiding this comment.
The archived package does not verify against this identity: from the package root, the documented GNU-tar procedure (tar --sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner -cf - . | sha256sum) produces 004f3f0c83eabc8b8f04524d6c5e061bcd80dda3c13627678b6c2371ee6cbeb6, not d97b9ace…. Consequently the manifest and content-addressed directory name cannot identify the checked-in package, undermining reproducibility and any later check that this was the skill used for the replay; record the actual digest or specify and preserve the different canonicalization that yields the stated value.
Useful? React with 👍 / 👎.
| | SIMD | Found universal gap | Found universal gap | Tie | | ||
| | Generated `KnownLayout` provenance | Found `NonNull::cast` documentation gap | Missed | Treatment improvement | | ||
| | Test-only `assume_initialized` | Correctly `cfg(test)` and `UNPROVED`, not production UB | Not emphasized in final findings | Treatment scope correct | | ||
| | Safe transmute-delegation macro | Not reported | Found latent size-equality contract hole; no current bad invocation | Valid baseline-only maintenance finding | |
There was a problem hiding this comment.
When this change is merged into the public repository, this line publishes an independently adjudicated, previously baseline-only unsafe-macro contract hole together with its trigger and remediation. The evaluation protocol explicitly requires potential novel defects in maintained code—including reports and fixture material—to remain in coordinated-disclosure quarantine until fixed, disclosed, or authorized (evals/unsafe-rust/testing-plan.md:944-948), but neither the manifest nor the result records any such disposition. Move the current-source reports/results to the restricted triage location or document the maintainer authorization before publishing them.
Useful? React with 👍 / 👎.
|
|
||
| Fixed controls: | ||
|
|
||
| - **A: fails the control requirement.** It reproduces none of the repaired vulnerabilities and its PASS is scoped and supported, but it misses the absent adjacent proof-grade comment for `item_unchecked`. |
There was a problem hiding this comment.
Report A satisfies the frozen control requirement: it reproduces none of the six repaired findings, and its scoped pass is supported. The fixture oracle explicitly says reporting the retained proof-comment defect is optional and that the required condition is not reproducing a repaired finding (evals/unsafe-rust/fixtures/pilot/README.md:36-40), so failing A solely for omitting that comment changes the scoring rule after the reports were observed and creates a spurious treatment advantage. Mark A as passing the fixed control and report proof-comment discovery only as an optional secondary distinction.
Useful? React with 👍 / 👎.
| The synthetic reports were copied under labels A–D and scored by a fresh agent | ||
| which knew vulnerable versus fixed membership but not condition identity. It | ||
| saw no target source, skill, prior results, or condition mapping. The raw score | ||
| is [`blind-synthetic-score.md`](blind-synthetic-score.md). |
There was a problem hiding this comment.
The archived blind score refers only to A–D, while the manifest identifies reports only by agent/runtime names and never records which file was assigned each blind label. After blinding is complete, omitting that mapping prevents an independent reader from binding the scorer's fixed-control decisions and vulnerable scores to the skill and no-skill cells, so claims such as the treatment's proof-artifact win cannot be reproduced from the committed evidence. Add the post-scoring A–D-to-report mapping to the manifest.
Useful? React with 👍 / 👎.
| The first historical-vulnerable baseline agent found the correct defect but | ||
| stalled while waiting for a helper and produced no final report. That attempt | ||
| was invalidated. An accidentally over-specific replacement prompt was aborted | ||
| before it returned any result and was not scored. The recorded `g2r6` report | ||
| came from a new neutral-prompt agent forbidden to spawn helpers. |
There was a problem hiding this comment.
The recorded historical baseline was uniquely forbidden to spawn helpers after an earlier baseline stalled, while the skill cell retained the original agent policy. Because helper access can directly affect audit coverage and report completeness, the resulting 14/14 versus 12/14 comparison no longer isolates the skill condition, particularly where the reported difference is authority and configuration detail. Rerun the paired historical cells with the same helper/tool policy, or treat this pair as invalid rather than using it as a treatment regression result.
Useful? React with 👍 / 👎.
| Long-running cells received only target-neutral steering such as “complete | ||
| with your current source-only findings” or “do not widen scope.” No finding, | ||
| location, contract, or expected verdict was supplied. |
There was a problem hiding this comment.
The manifest says the reconstructed prompts were frozen, but it stores only prompt fragments and summaries: the baseline's added prohibition is not quoted, target requests are paraphrased, placeholders contain conditional text, and follow-up steering is recorded only as examples. This makes it impossible to reconstruct or hash the actual per-cell prompts, verify that paired conditions differed only by the skill instruction, or reproduce the replay as described. Archive the literal initial and follow-up messages for every cell, preferably with their hashes.
Useful? React with 👍 / 👎.
2 participants
Freeze the V1 skill package and replay the synthetic, historical, and
current-source targets from the exploratory predecessor pilot. Preserve the
manifests, raw reports, scores, and limitations needed to compare the replay
with the earlier procedural run.
The replay recovers every known synthetic and historical defect and accepts
every fixed control. It improves version applicability, literal contract
discovery, and exposure of reconstructed proofs, while still missing an
admissible indirect Copy and UnsafeCell derivation on the current-source
challenge.
Treat this as legacy confirmation rather than release evidence: the replay is
not byte-identical to the pilot, uses one replicate per target, and retains the
pilot's procedural-isolation and unavailable model-identity limitations.
⬇️ Download this PR
Branch
git fetch origin refs/heads/Gup6qexxvahgx6uken5vngpapqsaio22h && git checkout -b pr-Gup6qexxvahgx6uken5vngpapqsaio22h FETCH_HEADCheckout
git fetch origin refs/heads/Gup6qexxvahgx6uken5vngpapqsaio22h && git checkout FETCH_HEADCherry Pick
git fetch origin refs/heads/Gup6qexxvahgx6uken5vngpapqsaio22h && git cherry-pick FETCH_HEADPull
Stacked PRs enabled by GHerrit.