fix(webfetch): qualify current sources against frozen body rules

.agents/skills/benchmark-report/SKILL.md

@@ -37,7 +37,7 @@ Version changes require an explicit human release request. Code synchronization,
 Public report CLIs fall back to English. `BROWSER_EVAL_REPORT_LANGUAGE` changes the environment default; an explicit `--language` wins, and a rebuild preserves the saved primary language. For an agent-driven build, select the language from the current conversation. English builds retain the matching Chinese edition; a Chinese selection does not generate an English translation unless explicitly requested. Keep measured material in its source language. Historical Chinese-only renderers must disclose that limit and reject unsupported requests. Bind generated languages and the selected entry in report metadata. English navigation opens a not-generated notice when absent; never publish stale English pages. Validate each generated edition and retain route, value and evidence-link parity checks when both exist.
 
 1. Establish the decision being supported, the measured unit, cohort, versions, execution location, freshness policy, and intended deliverable. Inspect the actual dataset, runner, outputs, scorer, prior review evidence, and current report. Do not infer them from screenshots or prose.
-2. Freeze a machine-readable evaluation contract before expensive calls. It must identify the exact cases and methods, run count, timeout, concurrency policy, input and output contract, scoring provenance, reference state, and publication target when one exists. Existing compiled reference answers are frozen by default. Source metadata, capture dates, and absent newer acceptance-contract fields do not imply a live-reference task. Only tasks explicitly declaring `source_verification.reference_mode: live_exact` require live checks. For those tasks, fail closed: verify and freeze references immediately before calls, verify the same declared scope afterward, and certify the cohort only when both checks pass. An unchanged compiled acceptance-contract dataset is already the benchmark input: freeze its task and rule bytes for execution, without requiring historical source-qualification raw files or re-auditing cohort admission. Qualify sources when adding tasks or changing references/acceptance rules; do not turn an ordinary version rerun into dataset reconstruction. Partial or uncertified cohorts remain diagnostic evidence and cannot enter a release.
+2. Freeze a machine-readable evaluation contract before expensive calls. It must identify the exact cases and methods, run count, timeout, concurrency policy, input and output contract, scoring provenance, reference state, and publication target when one exists. Existing compiled reference answers are frozen by default. Source metadata, capture dates, and absent newer acceptance-contract fields do not imply a live-reference task. Only tasks explicitly declaring `source_verification.reference_mode: live_exact` or `live_body` require live checks. `live_body` applies the frozen body contract to independent pre/post source text without replacing the reference. For those tasks, fail closed: verify and freeze references immediately before calls, verify the same declared scope afterward, and certify the cohort only when both checks pass. An unchanged compiled acceptance-contract dataset is already the benchmark input: freeze its task and rule bytes for execution, without requiring historical source-qualification raw files or re-auditing cohort admission. Qualify sources when adding tasks or changing references/acceptance rules; do not turn an ordinary version rerun into dataset reconstruction. Partial or uncertified cohorts remain diagnostic evidence and cannot enter a release.
 3. Let each method use its strongest documented route to produce the required output. Equalize the task, inputs, output contract, call budget, and scoring. Do not require identical internals. For fixed-native cohorts, declare the input media type independently of URL patterns and expected answers, freeze the class-level route and transport parameters before calls, and verify the retained request receipts at aggregation and release. Record material asymmetries such as local versus remote execution, cold versus warm start, bundled conversion, network transit, account tier, and billable time.
 4. Preserve evidence as an append-only chain by default: task definition, reference, provider call, raw artifact, normalized artifact, score, review, revision, and release. If the user explicitly requests deletion of superseded history, validate the chain first and use the existing owner's current-only authority described in [audit and revision](references/audit-and-revision.md); remove obsolete full snapshots rather than storing reverse patches. Keep current provider outputs, call identities, real scoring sources and necessary review evidence immutable. Routine refresh and publication reuse current inputs without redundant historical snapshots. A presentation or scoring change must not masquerade as a new provider result. Declare the native answer fields and validate their structural projection before accepting a scored artifact. Adapters must not remove returned candidates by URL meaning or consult expected answers when selecting candidate fields. Apply privacy redaction only to separately identified public views after scoring, preserving the private originals and distinguishing original, scored and public-view hashes.
 5. Separate retrieval success, task acceptance, content quality, latency, cost, and freshness. Treat missing telemetry as unavailable. A request that received no first byte contributes to total latency, but not to first-byte or body-transfer stage means; show each stage’s observed population. A missing or partial reference makes the affected metric unavailable or scoped; it does not prove success or failure. Do not derive binary acceptance from an unrelated aggregate similarity score.

AGENTS.md

@@ -44,7 +44,7 @@ These requirements apply to **all** report tracks; browser-specific rules below
 - For benchmark design, execution, scoring, result audit, report generation or release, read `.agents/skills/benchmark-report/SKILL.md`. Use `.agents/skills/benchmark-editorial/SKILL.md` additionally only for a new or substantially rewritten standalone interpretation article, official blog or public-account article.
 - Keep one report implementation in `harness/web_fetch/report.py`. The default `scripts/build_report.py` build uses persistent measured evidence (`~/.browser-eval/evidence/current/` by default, or `WEB_FETCH_EVIDENCE_DIR`) and writes `reports/webfetch/`; the independently bound WCXB build uses `--wcxb` and writes `reports/wcxb/`. Do not create versioned report implementations, compatibility renderers or redirect builds.
 - Run repository Python only through `make` targets or `uv run --frozen python`. `.python-version`, `uv.lock`, the Makefile and CI are the runtime contract; never fall back to system Python.
-- Run every publishable provider cohort through `scripts/run_certified_benchmark.py`, which freezes executable task inputs. Existing compiled reference answers stay frozen; pre/post source checks require an explicit `source_verification.reference_mode: live_exact` task declaration. Missing newer acceptance-contract fields never imply a live-reference task. Rerunning an unchanged compiled dataset must not require historical source-qualification raw files or reopen cohort admission; source qualification belongs to adding tasks or changing references/acceptance rules. Direct `run_benchmark.py` output and incomplete cohorts are diagnostic evidence only; never assemble or publish them.
+- Run every publishable provider cohort through `scripts/run_certified_benchmark.py`, which freezes executable task inputs. Existing compiled reference answers stay frozen; pre/post source checks require an explicit `source_verification.reference_mode: live_exact` or `live_body` task declaration. `live_body` qualifies current source text against unchanged frozen body requirements and never refreshes the reference. Missing newer acceptance-contract fields never imply a live-reference task. Rerunning an unchanged compiled dataset must not require historical source-qualification raw files or reopen cohort admission; source qualification belongs to adding tasks or changing references/acceptance rules. Direct `run_benchmark.py` output and incomplete cohorts are diagnostic evidence only; never assemble or publish them.
 - Select report instances explicitly with `--profile` (or an external `--authority-root` bundle for historical full-cohort formats). Store their matrices, publication targets and mutable version receipts outside Git alongside the frozen evidence. Each report and track owns its own publication binding; never borrow another report's target.
 - For the complete WebMainBench report, `evaluation/webmainbench-full/current-report.json` inside the selected external authority bundle is the immutable baseline pre-call profile bound by its cohort certificate. Independently certified variant cohorts are registered in `evaluation/webmainbench-full/`; discover the current variant registry and each variant's profile and certificate from that directory rather than hard-coding names. The method-binding file joins the baseline and all certified variant cohorts through locked profile hashes. Their `publication_state` records frozen pre-call state. The live site, share URL, and expected remote version are bound in `evaluation/webmainbench-full/publication-target.json` together with the baseline and all variant profile hashes; update this target after publication without changing or re-certifying historical calls.
 - For the current WCXB report, the source-qualified manifest is the entire evaluation population for quality, delivery, and latency. Keep source-invalid cases outside the current manifest, scored evidence, public package, and future provider calls; preserve earlier physical calls only in private historical evidence. The dataset page owns frozen task inputs, source URLs and human reference scope; it must not show measured browser scores. The home page owns aggregate comparisons, the result analysis owns the filterable method-case inventory, and case result pages own full responses. Keep a direct link from each dataset row to a safe frozen-page preview and its human reference. Default result filters must show a nonempty, clearly labeled diagnostic population when one exists, and filtered tables must shrink rather than leave a large blank viewport.

harness/web_fetch/README.md

@@ -127,6 +127,17 @@ body regions, record contracts, composite fields and patterns as well as facts;
 a fact group is optional when another
 content requirement establishes the source scope.
 
+For live URLs whose revised acceptance depends on frozen body text, explicitly
+select `source_verification.reference_mode: live_body`. The existing certified
+entry captures current source text before and after calls. Both observations
+must satisfy the original body requirements; neither observation replaces the
+frozen reference, regions or thresholds. The freezer retains a hash-bound
+`source-qualification-<sha256>.txt` beside task sources. Preflight checks its freshness
+and contents, and certification rechecks retained pre/post evidence. Missing or
+incompatible sources reject the cohort; they are not provider failures. Small
+changes within the declared coverage tolerance remain valid. Unsupported
+consumers must reject the mode rather than omit its source checks.
+
 ## WCXB scoring
 
 `wcxb_scoring.py` owns certified frozen-HTML scoring. The maintained command is
@@ -1248,7 +1259,7 @@ Lightpanda 使用 `--dump markdown --json` 的原生正文；Obscura 使用其
 不重新审核入选资格，不读取历史资格调查原文，也不因旧归档缺失而改成诊断结果。
 新增题目、修改参考或验收规则时，仍须通过相应构建与修订流程核验来源。
 固定答案不会因为有来源核验信息、旧核验日期或缺少新版验收契约而自动变成实时参考。
-只有题目明确声明 `source_verification.reference_mode: live_exact`，才执行调用前冻结和调用后核验。
+题目明确声明 `source_verification.reference_mode: live_exact` 时，执行调用前冻结和调用后精确核验；`live_body` 则在调用前后用原正文规则核验当前来源，保留原参考、分区和门槛。来源核验缺失或不通过时，不能认证为正式结果。
 没有该声明时，执行器保持题库中的答案和评分规则不变。明确修订参考答案仍走独立的参考修订入口。
 
 已授权的错误修正应自动完成：先查明共享原因并添加必要的正反测试，再冻结受影响的完整调用范围，经认证入口补跑。

harness/web_fetch/artifacts.py

@@ -18,7 +18,9 @@
 from harness.record_io import file_sha256
 from harness.web_fetch import native_routes
 from harness.web_fetch.publication_privacy import original_evidence_size, read_evidence_binding
+from harness.web_fetch.reference_inputs import input_tree
 from harness.web_fetch.reference_lifecycle import (
+    body_qualification_path,
     latency_correction_original_elapsed,
     preserve_legacy_reference_dataset,
 )
@@ -408,7 +410,8 @@ def materialize_bound_artifacts(output: Path, source_root: Path, records: list[d
         verify_record_public_call_logs(output, record)
 
 
-def materialize_reference_checks(output: Path, source: Path, batches: list[dict]) -> None:
+def materialize_reference_checks(output: Path, source: Path, batches: list[dict], *,
+                                 qualification_source: Path | None = None) -> None:
     """Retain hash-bound source observations and native execution contracts."""
     native_routes.materialize_native_route_contracts(output, source, batches)
     for batch in batches:
@@ -421,18 +424,28 @@ def materialize_reference_checks(output: Path, source: Path, batches: list[dict]
                 or any(char not in '0123456789abcdef' for char in digest)
                 or relative != f'reference-checks/observations-{digest}.json'):
             raise ValueError('Reference check has an invalid evidence binding')
-        path = contained_path(source, relative, reject_symlinks=True)
-        if not path.is_file() or file_sha256(path) != digest:
-            raise ValueError('Reference check evidence differs from its certificate')
-        target = contained_path(output, relative, reject_symlinks=True)
-        if target.exists():
-            if not target.is_file() or file_sha256(target) != digest:
-                raise ValueError('Conflicting reference check evidence')
-        else:
-            target.parent.mkdir(parents=True, exist_ok=True)
-            shutil.copy2(path, target)
-        if file_sha256(target) != digest:
-            raise ValueError('Copied reference check evidence changed')
+        files = [(relative, digest)]
+        for identifier, binding in batch.get('reference_checks', {}).get('cases', {}).items():
+            verification = binding.get('source_verification', {})
+            if verification.get('reference_mode') == 'live_body':
+                digest = verification.get('qualification_sha256')
+                files.append(('sources/' + body_qualification_path(identifier, digest), digest))
+        for relative, digest in files:
+            path = (contained_path(input_tree((qualification_source or source) / 'dataset.json', 'sources'),
+                                   relative.removeprefix('sources/'), reject_symlinks=True)
+                    if relative.startswith('sources/')
+                    else contained_path(source, relative, reject_symlinks=True))
+            if not path.is_file() or file_sha256(path) != digest:
+                raise ValueError('Reference check evidence differs from its certificate')
+            target = contained_path(output, relative, reject_symlinks=True)
+            if target.exists():
+                if not target.is_file() or file_sha256(target) != digest:
+                    raise ValueError('Conflicting reference check evidence')
+            else:
+                target.parent.mkdir(parents=True, exist_ok=True)
+                shutil.copy2(path, target)
+            if file_sha256(target) != digest:
+                raise ValueError('Copied reference check evidence changed')
 
 
 def materialize_task_references(output: Path, trees: dict[str, Path], case_ids: Iterable[str]) -> None:

harness/web_fetch/evidence.py

@@ -726,7 +726,7 @@ def finalize_aggregate(run_dir: Path, observations: Path) -> None:
         raise ValueError("Finalized worker metadata changed")
 
     for path, manifest in zip(manifests, updated, strict=True):
-        materialize_reference_checks(run_dir, path.parent, [manifest])
+        materialize_reference_checks(run_dir, path.parent, [manifest], qualification_source=run_dir)
 
     rebuilt = {**execution, "batches": [updated_by_key[_attempt_key(batch)] for batch in original]}
     temporary = None

harness/web_fetch/reference_freeze.py

@@ -31,6 +31,8 @@
 )
 from harness.web_fetch.reference_lifecycle import (
     _completion,
+    body_qualification_path,
+    body_source_qualifies,
     check_case_reference_binding,
     completed_case_records,
     legacy_reference_cases,
@@ -253,6 +255,18 @@ def compute_reference_update(dataset, captures, observations=None, *, refresh_ca
         if missing_facts:
             raise ValueError('Required fact missing from independent reference: '
                              +cid+'/'+missing_facts[0])
+        if case.get('source_verification', {}).get('reference_mode') == 'live_body':
+            if not body_source_qualifies(case, text):
+                raise ValueError('Current source no longer satisfies frozen body: ' + cid)
+            case['source_verification'].update(
+                state='valid', checked_at=timestamp, selector=selector, method=method,
+                observation_route='browser' if cid in observations else 'http',
+                reference_sha256=hashlib.sha256(case['ground_truth_main_content'].encode()).hexdigest(),
+                qualification_sha256=hashlib.sha256(text.encode()).hexdigest())
+            contexts[cid] = {'row': row, 'missing_facts': missing_facts,
+                             'browser_observation': cid in observations,
+                             'qualification_text': text}
+            continue
         old=case['ground_truth_main_content']
         if normalize_for_match(old)==normalize_for_match(text):text=old
         old_requirements=case.get('body_requirements')
@@ -370,6 +384,8 @@ def main(argv=None):
                                        and not capture_record['quality_issues'] else 'needs_review')
             (target/'current-capture.json').write_text(
                 json.dumps(capture_record,ensure_ascii=False,indent=2)+'\n')
+        if 'qualification_text' in contexts[cid]:
+            (args.output/'sources'/body_qualification_path(cid, case['source_verification']['qualification_sha256'])).write_bytes(contexts[cid]['qualification_text'].encode('utf-8'))
         (target/'reference.txt').write_text(case['ground_truth_main_content'])
     (args.output/'dataset.json').write_text(json.dumps(dataset,ensure_ascii=False,indent=2)+'\n')
     refresh={'schema_version':1,'changed':changes,
@@ -391,7 +407,7 @@ def main(argv=None):
             json.dumps(dataset,ensure_ascii=False,indent=2)+'\n')
     (args.output/'reference-refresh.json').write_text(
         json.dumps(refresh,ensure_ascii=False,indent=2)+'\n')
-    print('Frozen exact-reference cases:',sum(requires_fresh_reference(c) for c in dataset['cases']),'; changed full-text references:', len(changes))
+    print('Qualified live-reference cases:',sum(requires_fresh_reference(c) for c in dataset['cases']),'; changed full-text references:', len(changes))
 
 
 def load_resumable_observations(checkpoint, cases):