[omsherikar]

What

Withhold SAFE when a diff weakens the very tests that would judge it. A trusted
would-be-SAFE whose covering evidence rests on tests the same diff relaxed
(removed assertion, deleted/renamed test, added skip/xfail) now degrades to
UNPROVEN, naming the weakened file. Degrade-only: it can never grant SAFE.

Closes #163.

Supersedes #166. That branch was retired because it also carried an internal
decision record; the design rationale is kept in internal dev-docs, out of the
public repo. This branch is the same reviewed code, minus that file.

How

• src/verify/test-weakening.ts (new): pure detector over changed test files.
  Strips strings/comments, then flags a net drop in assertion count (pytest
  assert, self.assertX, self.fail, inline ; assert), a disappearing test
  name (def/async def, delete or rename), and added skip/xfail markers.
• src/verify/verdict-fuse.ts: testWeakening is a required fuseVerdict input
  and an additive VerdictReport field; the downgrade lives in the trusted branch.
• src/verify/verify-diff.ts: reads each changed test file's pre-diff content and
  fails safe — a repo-escaping path is refused before the read, a non-ENOENT read
  error is recorded as weakening, only ENOENT counts as a new (strengthening) test.

Evidence

• Red-first integration (--trusted): behavior change + gutted covering assertion
  reads SAFE on main, UNPROVEN here, with changedLinesCovered === true
  pinned so a coverage-gap UNPROVEN can't pass it green. Strengthening stays SAFE.
• Unit: detector cases (removed/loosened count, deleted/renamed test incl. async,
  each skip/xfail family, docstring-mask, inline assert, known-negatives) and the
  intake fail-safe (read-error → weakening, escaping path → refused, ENOENT → new).
• On the reviewed branch this passed the full gate and CI 14/14 across
  macOS/Ubuntu/Windows x Node 18/20/22 (660 tests). Same code here; CI re-runs.

Known limitations

Heuristic bar-raiser, not a proof, and degrade-only, so every gap leaves the
pre-change verdict, never a new false SAFE. Count-preserving evasions (#164) and
broad-vs-covering scope (#165) are tracked as follow-ups. Python-shaped signals.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Verification now detects when a change reduces test assertions, removes or renames test functions, or adds skip markers.
  • Verification reports which tests were weakened and why. If weakened tests would otherwise support a SAFE verdict, the result is downgraded to UNPROVEN.
• Bug Fixes
  • Unsafe or unreadable test paths and other read errors no longer pass silently; verification treats them as unresolved evidence. New test files are excluded from weakening checks.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 676c6134-52c0-4355-b206-fa67d7bbb339

📥 Commits

Reviewing files that changed from the base of the PR and between 36a316b and c4fa16a.

📒 Files selected for processing (8)

• src/verify/diff-input.ts
• src/verify/test-weakening.ts
• src/verify/verdict-fuse.ts
• src/verify/verify-diff.ts
• tests/integration/self-weakening-tests.test.ts
• tests/unit/verify/detect-weakened-tests.test.ts
• tests/unit/verify/test-weakening.test.ts
• tests/unit/verify/verdict-fuse.test.ts

Files not reviewed due to moderation or processing errors (4)

• src/verify/test-weakening.ts
• tests/unit/verify/test-weakening.test.ts
• src/verify/diff-input.ts
• src/verify/verify-diff.ts

Included review availability: Your plan provides up to 2 included reviews per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The verifier now compares changed test files with their pre-diff versions and detects selected weakening signals. Trusted would-be-SAFE results become UNPROVEN when findings exist. The report includes non-empty findings. Unit and integration tests cover detection, base-content reads, and verdict outcomes.

Changes

Test weakening verification

Layer / File(s)	Summary
Detect test weakening src/verify/test-weakening.ts, tests/unit/verify/test-weakening.test.ts	The detector compares assertion counts, test names, and skip/xfail markers in existing test files. Tests cover these signals, additive changes, and known detection limits.
Read pre-diff test content src/verify/diff-input.ts, src/verify/verify-diff.ts, tests/unit/verify/detect-weakened-tests.test.ts	detectWeakenedTests reads prior content for changed test files and checks repository boundaries. It skips absent base files and records other read failures as findings. Weakening findings also prevent the verifier from entering mutation or stability deep checks.
Fuse findings into verdicts src/verify/verdict-fuse.ts, src/verify/verify-diff.ts, tests/unit/verify/verdict-fuse.test.ts, tests/integration/self-weakening-tests.test.ts	fuseVerdict includes non-empty findings in the report and returns UNPROVEN for a trusted would-be-SAFE result with findings. Unit and integration tests cover weakening and non-weakening verdicts.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant verifyDiff
  participant detectWeakenedTests
  participant Repository
  participant detectTestWeakening
  participant fuseVerdict
  verifyDiff->>detectWeakenedTests: repoRoot and edits
  detectWeakenedTests->>Repository: read pre-diff test content
  Repository-->>detectWeakenedTests: content or read error
  detectWeakenedTests->>detectTestWeakening: old and new test content
  detectTestWeakening-->>detectWeakenedTests: weakening findings
  detectWeakenedTests-->>verifyDiff: findings
  verifyDiff->>fuseVerdict: verification result and findings
  fuseVerdict-->>verifyDiff: final verdict and report

Loading

Merge Risk: ⚪ Minimal · up to c4fa1

The reviewed change has no identified merge-blocking issue. It is ready for normal merge checks.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: preventing SAFE verdicts when a diff weakens its own tests.
Linked Issues check	✅ Passed	Issue #163 requires a trusted would-be-SAFE result to downgrade to UNPROVEN when changed tests show clear weakening. The detector reports net assertion drops, deleted or renamed test functions, and ad…
Out of Scope Changes check	✅ Passed	The changes stay within Issue #163. The resolvesInsideRepo export supports the required safe pre-diff reads. Detector, verdict-fusion, intake, and test changes directly implement or verify self-weak…
Docstring Coverage	✅ Passed	Docstring coverage is 86.67% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 15 functions across 8 files.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}