Repository navigation
docs: LLM-detection coverage audit (recall clean, precision over-fire) - #4
Merged
Merged
Conversation
Scope-B audit of provider detection over the corpus. Runs the real markers.scan_text and a broad shape-based heuristic over the SAME bytes. v1 (persisted bodies, 7.78% coverage) was rejected as selection-biased — marker hits feed file-selection ranking, so the sample was partly chosen by the detector under test. v2 re-ran over the acquire-cache bare repos (46,170 blobs, 10.4x, pre-selection) to remove the circularity. Recall: 0 misses on the unbiased sample; shape-detection adds noise and no recall (do not build). One real blind spot — a fully config-driven LLM client with no literals (the Assessor's own client.py) — defeats every string scanner equally, so it is not a vocabulary gap. Precision: the detector over-fires. 12 of 33 fired repos are false positives, all in the <=80-hit tail, in 7 identifiable classes (MCP-server connect docs, eval harnesses, test fixtures, BOM reference data, dual-use GOOGLE_API_KEY, CI bots, editor-extension recs). 5 of 7 close via path exclusion in is_scannable. Severity is quality/efficiency, not correctness: the bespoke probe already rejects provider hits, so a false hit cannot misclassify. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Findings from the Scope-B audit of provider/LLM detection across the corpus.
Method: real
markers.scan_textvs a broad shape-based heuristic over the same bytes. v1 (persisted bodies, 7.78% coverage) was rejected as selection-biased — marker hits feed file-selection ranking, so the sample was partly chosen by the detector under test. v2 re-ran over the acquire-cache bare repos (46,170 blobs, 10.4×, pre-selection). Every candidate adjudicated against real file content, default false-positive.Recall: 0 misses on the unbiased sample. Shape-detection measured to add noise and no recall — do not build. One real blind spot (config-driven client with no literals — the Assessor's own
client.py) defeats every static scanner equally; not a vocabulary gap.Precision: the detector over-fires — 12 of 33 fired repos are false positives, all in the ≤80-hit tail, in 7 structural classes. 5 close via path exclusion in
is_scannable(same mechanism it already uses for vendored dirs). Severity is quality/efficiency, not correctness: the bespoke-agent probe already rejectsprovider_identifier/provider_model_idhits, so a false hit cannot misclassify.Full taxonomy, per-class mitigations, and reproduction in the doc. No code change here — findings only; the optional precision fix is a separate task.
🤖 Generated with Claude Code