Repository navigation
fix(scoring): tighten Web Fetch rules with source-backed controls - #12
Conversation
Maisiqi
changed the title
![lexmount-ocr[bot]](https://avatars.githubusercontent.com/u/277637559?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Re-review complete · 1ff30e9
No findings in this incremental review.
There was a problem hiding this comment.
This review found 1 issue(s), including 1 blocking issue(s).
There was a problem hiding this comment.
This review found 2 issue(s), including 2 blocking issue(s).
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
Re-reviewed e3c0757 against base 4aa3cb2, including the latest review-boundary and mixed-record fixes and the 59 document-body conversions.
The previous outside-footer case is fixed; complete controls for all ten fixed-source tasks still pass, and fabricated quotations/authors remain rejected. The updated distribution documentation and its direct-character-floor clarification match the current declarations. The 59 body conversions use the existing version-3 scorer with five regions, an 80% floor in every region and all five required; the other 491 task definitions are unchanged from 613cd73. Projection/export binding checks passed. This review does not claim to have independently replayed the 59 private current-source captures.
Two new parser regressions remain (inline), both reproduced using public scoring and actual task contracts: ordinary div-based review cards lose their bodies, and the mixed-list fallback can borrow a description from a separate sibling section. Both stem from losing structural ownership before field matching; adding more phrase exceptions would not fix them.
The earlier output-exclusion finding remains unresolved, as the PR now acknowledges: the unchanged tasks still exclude Cookie banners/cart/recommendation material, while acceptance intentionally permits it. The documented non-adoption explains the chosen limitation, but does not make the task and scorer agree. Keep that existing finding distinct from the two new regressions rather than describing it as fixed.
Focused parser checks passed 178 tests and 34 subtests. No source edits, provider calls or historical score changes were made.
There was a problem hiding this comment.
This review found 2 issue(s), including 2 blocking issue(s).
There was a problem hiding this comment.
This review found 1 issue(s), including 1 blocking issue(s).
There was a problem hiding this comment.
This review found 2 issue(s), including 2 blocking issue(s).
There was a problem hiding this comment.
This review found 1 issue(s), including 1 blocking issue(s).
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
Reviewed the latest sample definitions and acceptance method at 9f34919, including real official-page positive/deletion controls and integration with PR65's certified runner. Three remaining issues are reproducible below. The distribution documentation now agrees with the 550-case definitions, duplicate-task acceptance gates agree, and the previously reported record-ownership cases pass their controls. The earlier Cookie/cart/recommendation exclusion finding remains unimplemented as already disclosed; this review does not duplicate it.
The fixes can stay within existing owners: source-qualified body requirements, substantive task facts, and the shared reference lifecycle. Additional provider-specific scoring or case-ID branches would not address these causes. No repository files were changed and no paid provider calls were made.
There was a problem hiding this comment.
This review found 1 issue(s), including 1 blocking issue(s).
There was a problem hiding this comment.
✅ Re-review complete · c2b468c
No findings in this incremental review.
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
Completed a bounded P0/P1-only re-review of the current 550 sample definitions and acceptance method at c2b468c, including the companion source lifecycle in browser-eval PR75 at d6f3665. No new P0/P1 was confirmed.
The previous substantive-documentation finding is fixed: the complete official Python HTML documentation passes, while removing the whole html.escape explanation now fails. The current Scrapy README passes with Python 3.11 wording, and the current PostgreSQL tutorial passes without the obsolete release banner. These are paired source controls, not provider measurements. The 59 newly mandatory live-body cases now all declare live_body and enter the companion consumer's pre/post source checks; missing or invalid source evidence prevents normal certification.
Population checks still agree with the maintained documentation: 550 entries, 535 exact URLs and 540 URL+prompt tasks. Ten duplicate tasks have equal acceptance gates. Empty-answer controls reject all 248 cases testable without external body references; 143 targeted acceptance/projection tests passed. No provider-result-driven sample selection or new method-specific scoring exception was found.
This review is limited to P0/P1. It does not waive lower-priority findings, deferred frozen-export checks, or the requirement to use a compatible consumer for live_body. No source edits, paid provider calls, historical rescoring or merge.
Summary
Tightens acceptance for
review-550-20261023across five suites and 550 cases: titles, navigation, duplicate links or incomplete article endings must not substitute for the requested content. Select the revised rules with--revision review-550-20261023. Frozen input files, task URLs, prompts and original reference bytes are preserved.Changes
html.escapeexplanation; remove obsolete Scrapy/PostgreSQL version labels. Preserve literal plain text and record-local dates, titles and descriptions across nested/sibling wrappers.--review-sourceevidence is verified when supplied; missing evidence remains unverified. Remove 232 external evidence dependencies from policy-only forbidden-word corrections.Maintained details: document-body decisions, review decisions and evidence, and sample composition/scoring coverage.
V8 review disposition
live-019,live-027and Reduxlive-070at their documented 82.26–83.72% minimum margins. Removing internal prose solely to increase scores is not justified.Formal scoring uses saved references without an added live-page admission gate. Investigate source drift and extraction when a correct-looking answer fails; any reference correction must be reviewed, versioned and source-backed, preserving frozen inputs and keeping rules independent of provider outputs.
Validation
Commit
7967a12: local tests passed 3,546 tests and 353 subtests; registered-definition validation and the documentation distribution check passed.CI for
7967a12passed on 2026-10-28, including source and package checks. The bot review for the same commit also passed.Compared all 550 registered revised cases against
949a2fe: only the Push API first-region boundary/bindings change; the other 549 cases are identical. Other Push requirements and later regions remain unchanged.Reused hash-matching saved Push reference/HTML: complete reference and full-content source controls pass; headings/navigation only, excluded prefix only, half-body and all five omitted-region controls fail. No fresh capture or provider call.
Historical provider rescoring predates the document-body follow-up: 4,820 outputs replayed at
03d23e7, with targeted table checks through19ba48a. Those totals are not results for the current revision.Remaining limits and release work
webfetch,popular-sites,dynamic-contentandaccess-restrictionsremain deferred at the requester's direction: exact hash-matching original bundles were unavailable. Complete export/reference acceptance remains unverified; CI and source controls do not replace those checks.bench-v2.1.0. No tag or merge is performed here.Status: open for maintainer review; unmerged.