feat(benchmarks): enforce shared evaluation integrity policy

AGENTS.md

@@ -35,6 +35,13 @@ cache import, and the `--install-only` dependency preflight option.
 - Keep setup failures and verifier errors separate from scored failures. Only
   verifier timeouts may retry, at most three total attempts using the same agent
   output and the task's original per-attempt time limit.
+- New standard runs require a registered `integrity` policy shared across agent
+  arms. `--legacy-open-network` is only for explicitly registered historical
+  protocol reproductions. Never silently relax a failed isolation preflight.
+- Freeze completed outputs before integrity review. Keep original rewards and
+  errors, record reviewed violations separately, count violations as effective
+  zeroes without removing them from the denominator, and keep unreviewed results
+  pending. Do not mix restricted and historical protocols into one baseline.
 
 
 ## Language

benchmarks/harbor/README.md

@@ -15,21 +15,21 @@ endpoint, configure the API key and base URL; select the model with Harbor's
 
 ```bash
 export ANTHROPIC_API_KEY="..."
-export ANTHROPIC_BASE_URL="https://gateway.example"
+export ANTHROPIC_BASE_URL="https://api.deepseek.com/anthropic"
 ```
 
 Run new benchmarks through the standard entry point, from the repository root:
 
 ```bash
 uv run --project benchmarks/harbor python -m harbor_adapter run \
-  --config benchmarks/harbor/configs/qemu-alpine-ssh.json
+  --config benchmarks/harbor/configs/restricted-wal.json
 ```
 
-This example pins the QEMU task revision and uses `deepseek/deepseek-flash`.
+This example pins the WAL task revision and uses `deepseek/deepseek-flash`.
 Set the endpoint and credentials for that model, or copy the configuration and
 change `agents[].model_name`, task selection, and budgets for another run. These
-are ordinary Harbor JSON configurations; the entry point supplies the standard
-verifier and cache policy. It accepts the repository's `NanoPyCodeAgent` adapter
+are Harbor JSON configurations with an additional `integrity` section; the entry
+point supplies the standard verifier, integrity, and cache policies. It accepts the repository's `NanoPyCodeAgent` adapter
 with budget kwargs; credentials belong in the environment.
 
 The entry point builds the current working tree, including uncommitted changes,
@@ -48,6 +48,138 @@ exercise dependency installation and preflight without calling the model or
 executing the official tests. A full run still performs setup and preflight in
 its own fresh container; an install-only check does not resume into model work.
 
+### Evaluation integrity protocol
+
+New standard runs require an explicit `integrity` section:
+
+```json
+{
+  "integrity": {
+    "protocol": "restricted-v1",
+    "allowed_hosts": ["api.deepseek.com"],
+    "probe_url": "https://example.com/"
+  }
+}
+```
+
+The same policy and English integrity instructions apply to Nano, Pi, and the
+DeepSeek Harness adapter. Their native tools and model settings are preserved.
+The actual inference host must be listed explicitly; provider endpoints are not
+silently added. Exact DNS hosts are accepted; wildcards, IP literals, CIDRs,
+extra agent network exceptions, injected skills, resumed sessions, external
+tools, and filtered logs are rejected. Do not allowlist general code hosting or
+search sites: that would reopen access to reference answers. Task-required data
+or reviewed dependency/documentation services need explicit, preregistered hosts.
+An allowed service can itself expose reference material; an allowlist is not a
+content classifier.
+
+Installation and registered dependency preflight use the task's original public
+network. Before model work, a public HTTPS probe must succeed, the same probe
+must fail under the allowlist, and allowed hosts must remain reachable. Redirects
+are not followed by probes. A failed positive control is an infrastructure
+failure, never proof of blocking. `--install-only` exercises these checks too.
+The run boundary rechecks enforcement within the native agent time limit.
+Harbor's external egress sidecar enforces the restriction; shell proxy variables
+are not the security boundary. The task container loses `NET_RAW`, `NET_ADMIN`,
+`SYS_ADMIN`, and `SYS_PTRACE`, and cannot gain privileges through setuid binaries.
+Host directories, including verifier logs, are not mounted into the container.
+UDP DNS forwarding is permitted only to the exact upstream resolver addresses
+reported by Docker's trusted sidecar; those addresses are recorded. DNS failure
+does not count as a successful HTTPS blocking probe.
+
+Version one supports single-step Linux Dockerfile/prebuilt-image tasks with
+public native network policies. Custom Compose services, extra mounts,
+environment overrides, and task MCP servers are unsupported and fail closed.
+These limits deliberately exclude the historical dependency-cache Compose
+overlays until they have a reviewed isolation design. The protocol preserves
+task images, package sources, official tests, resources, and native deadlines.
+CPU, memory, storage, GPU, and TPU overrides are rejected; CPU and memory
+enforcement must be `auto` (Docker's native limits) or `limit`.
+Missing task dependencies remain setup or task failures according to the
+original verifier; network restrictions do not authorize changing the task.
+
+All three agents use process supervision under this protocol. Network access
+stays restricted through cancellation cleanup. Controller-collected output,
+network-rule snapshots, preflight outcomes, and sidecar diagnostic logs are saved
+under the job's host-only `.integrity/<trial-id>/` directory, outside Harbor's
+native trial-log rewriting. These records are private
+because output can contain credentials. They preserve collected bytes, but do
+not prove complete observation of every tool action or network request.
+
+Existing shared verifiers remain shared: after quiescence checks and evidence
+collection, their native network is restored for official verification. They
+remain vulnerable to malicious submitted code and retained services influencing
+grading. Tasks already configured with separate verifiers retain that boundary;
+the adapter records the mode without rewriting tasks. Migrating existing tasks,
+isolating submitted code inside the grader, private held-out tasks, and DNS/covert
+channel defenses are separate work. `restricted-v1` blocks the observed direct
+web answer retrieval; it is not a proof against every adversarial strategy or
+training-data contamination.
+
+For an explicitly registered historical-protocol reproduction, pass
+`--legacy-open-network` with the original configuration. This opt-out is recorded
+as `legacy-open-network`, cannot be combined with `integrity`, and supplies no
+integrity enforcement. Historical configurations and reports are unchanged.
+Do not combine restricted and historical results into one comparable baseline.
+
+### Freeze, review, and integrity-adjusted scores
+
+Completed restricted runs automatically freeze output hashes and write candidate
+flags to `jobs/<job-name>-input/audit/audit.json`. They do not automatically earn
+an integrity-cleared score. The same audit can be invoked explicitly:
+
+```bash
+uv run --project benchmarks/harbor python -m harbor_adapter audit \
+  --job-dir jobs/<job-name> --output-dir jobs/<job-name>-input/audit
+```
+
+The output directory must be outside the job. Auditing refuses unfinished jobs,
+symbolic links, special files, and changes to an existing freeze. Scanning covers
+native text/JSON logs across adapters plus collected output and egress evidence;
+findings contain file/line locations, not raw credential-bearing excerpts.
+Patterns are triage hints: inspect surrounding calls and results, including
+blocked versus successful retrieval, before deciding. No matches do not prove a
+clean attempt. `clear` means no violation observed in the reviewed evidence.
+
+Create a review bound to the exact `freeze.json` SHA256, using actual trial IDs
+and evidence paths from that manifest:
+
+```json
+{
+  "freeze_sha256": "<SHA256 of freeze.json>",
+  "trials": {
+    "<trial-id>": {
+      "verdict": "violation",
+      "reviewer": "<reviewer identifier>",
+      "reason": "Downloaded and used the task's reference solution.",
+      "evidence": ["<trial-id>/agent/pi.txt"]
+    }
+  }
+}
+```
+
+Pass `--review /path/to/review.json` to the audit command. Verdicts are `clear`
+or `violation`; every verdict requires a reviewer, reason, and frozen evidence.
+Each review and resulting audit is saved under its hash, preserving previous
+decisions. Raw Harbor rewards and exceptions are never overwritten. Confirmed
+violations receive an effective zero and stay in the denominator. Pending
+reviews have no effective score, and no adjusted pass rate is published until
+all eligible trials are reviewed. Setup/verifier errors remain unscored. A native
+agent timeout is eligible only when cancellation was confirmed and an official
+binary reward exists. Missing required enforcement is an integrity error.
+
+Run the optional model-free Docker probe with an existing image containing
+Python 3 and CA certificates:
+
+```bash
+NANOPY_INTEGRITY_TEST_IMAGE=<image> uv run --project benchmarks/harbor pytest \
+  -c benchmarks/harbor/pyproject.toml benchmarks/harbor/tests/test_integrity_docker.py
+```
+
+The probe builds Harbor's pinned egress sidecar, tests allowed and denied HTTPS,
+checks that packet marks cannot bypass filtering, and removes only its own
+containers. It does not call a model or run benchmark solutions/tests.
+
 ### Verified dependencies and preflight
 
 The standard adapter automatically uses `.cache/nanopy-harbor/`, shared across

benchmarks/harbor/README.zh-CN.md

@@ -14,19 +14,19 @@ URL；模型由下方 Harbor 命令中的 `--model` 选择：
 
 ```bash
 export ANTHROPIC_API_KEY="..."
-export ANTHROPIC_BASE_URL="https://gateway.example"
+export ANTHROPIC_BASE_URL="https://api.deepseek.com/anthropic"
 ```
 
 新的 benchmark 统一从仓库根目录通过标准入口运行：
 
 ```bash
 uv run --project benchmarks/harbor python -m harbor_adapter run \
-  --config benchmarks/harbor/configs/qemu-alpine-ssh.json
+  --config benchmarks/harbor/configs/restricted-wal.json
 ```
 
-此示例固定了 QEMU 题目版本，模型为 `deepseek/deepseek-flash`。请配置对应的
+此示例固定了 WAL 题目版本，模型为 `deepseek/deepseek-flash`。请配置对应的
 endpoint 和凭证；测试其他模型或题目时，复制配置并修改 `agents[].model_name`、
-题目选择及预算。这是普通的 Harbor JSON 配置，入口会补齐标准验证器和缓存策略。
+题目选择及预算。这是增加了 `integrity` 段的 Harbor JSON 配置，入口会补齐标准验证器、完整性和缓存策略。
 入口接受本仓库的 `NanoPyCodeAgent` adapter 和预算参数，凭证通过环境变量传入。
 
 入口会将当前工作区（包括未提交修改）构建为 wheel，并核对其 Python 源码与保存的
@@ -40,6 +40,114 @@ endpoint 和凭证；测试其他模型或题目时，复制配置并修改 `age
 不启动容器；`--install-only` 只执行依赖安装和预检，不调用模型、不运行官方测试。
 完整运行仍会在自己的新容器里执行安装和预检，不能把 install-only 容器续跑为解题。
 
+### 评测完整性协议
+
+新的标准运行必须显式配置 `integrity`：
+
+```json
+{
+  "integrity": {
+    "protocol": "restricted-v1",
+    "allowed_hosts": ["api.deepseek.com"],
+    "probe_url": "https://example.com/"
+  }
+}
+```
+
+Nano、Pi 和 DeepSeek Harness adapter 使用相同策略与英文完整性规则，保留各自
+原生工具和模型设置。实际推理服务域名必须显式列入，入口不会静默增加 provider
+端点。只接受精确 DNS 主机名；拒绝通配符、IP、CIDR、额外 agent 网络例外、
+注入的 skills、恢复会话、外部工具和日志过滤。不要允许通用代码托管或搜索站点，
+否则会重新开放参考答案访问。题目明确需要的数据、经过审阅的依赖或文档服务必须
+事先登记具体域名。允许的服务本身仍可能暴露参考材料；白名单并不是内容分类器。
+
+安装与已登记的依赖预检使用题目原有的公开网络。模型开始工作前，公开 HTTPS
+探针必须成功；应用白名单后同一探针必须失败，同时允许的主机仍须可达。探针不
+跟随重定向。正向探针失败属于基础设施错误，不能当作阻断成功。
+`--install-only` 也会执行这些检查。运行边界会在原生 agent 时限内再次检查限制。
+网络限制由 Harbor 的外部 egress sidecar 执行，shell 代理环境变量不是安全边界。
+题目容器移除 `NET_RAW`、`NET_ADMIN`、`SYS_ADMIN`、`SYS_PTRACE` 权限，禁止
+通过 setuid 程序提升权限。包括 verifier 日志在内的宿主目录不会挂载进容器。
+UDP DNS 转发仅允许到 Docker 可信 sidecar 报告的精确上游解析器地址，并记录
+这些地址。DNS 解析失败不会被当作 HTTPS 阻断探针成功。
+
+第一版支持原生网络策略为公开网络的单步 Linux Dockerfile/预构建镜像任务。
+自定义 Compose 服务、额外挂载、环境覆盖和题目 MCP 服务暂不支持，遇到时直接
+拒绝运行。因此，历史依赖缓存 Compose 覆盖配置需要经过隔离设计审阅后才能接入。
+协议保留题目镜像、软件源、官方测试、资源和原生时限。CPU、内存、存储、GPU
+和 TPU 的资源覆盖配置会被拒绝；CPU 和内存的执行策略必须为 `auto`（Docker
+原生限制）或 `limit`。题目缺失依赖仍按原验证器
+表现分类为安装错误或任务失败；网络限制不授权修改题目。
+
+该协议下三个 agent 都使用进程托管，取消及清理期间网络仍受限。控制器采集的
+输出、网络规则快照、预检结果和 sidecar 诊断日志保存到 job 的宿主专属
+`.integrity/<trial-id>/` 目录，避开 Harbor 对原生 trial 日志的改写。
+输出可能含凭据，因此这些记录保持私有。它们保全已采集的
+字节，但不能证明完整观察了每次工具调用或网络请求。
+
+现有共享验证器仍是共享模式：检查进程结束边界并收集证据后，恢复其原生网络以
+执行官方验证。恶意提交代码及保留的服务仍可能干预评分。已经配置独立验证器的
+题目保留原有边界，adapter 记录其模式，不改写题目。现有题目的迁移、评分器内部
+对提交代码的隔离、私有保留集，以及 DNS/隐蔽通道防护属于后续工作。
+`restricted-v1` 阻断本次观察到的直接网络答案下载途径，并不证明能抵抗所有
+对抗策略或排除训练数据污染。
+
+对于明确登记的历史协议复现，可用原配置加 `--legacy-open-network`。这一退出
+选项会记录为 `legacy-open-network`，不能与 `integrity` 合用，也不提供完整性
+防护。历史配置和报告保持原样。受限协议与历史结果不应合并成可直接比较的基线。
+
+### 冻结、复核和完整性修正成绩
+
+完成的受限运行会自动冻结输出哈希，并将候选线索写入
+`jobs/<job-name>-input/audit/audit.json`，不会自动获得完整性复核通过的成绩。
+也可以显式执行相同审计：
+
+```bash
+uv run --project benchmarks/harbor python -m harbor_adapter audit \
+  --job-dir jobs/<job-name> --output-dir jobs/<job-name>-input/audit
+```
+
+输出目录必须位于 job 之外。未结束的 job、符号链接、特殊文件，以及冻结后发生
+变化的证据都会被拒绝。扫描覆盖各 adapter 原生文本/JSON 日志、采集输出与网络
+证据；线索只记录文件及行号，不摘录可能含凭据的原文。模式匹配仅用于初筛：
+需要查看相关调用与结果，区分请求被阻断和实际获取成功，再作判断。未匹配到内容
+不证明尝试干净；`clear` 表示在复核证据中未观察到违规。
+
+复核文件须绑定 `freeze.json` 的精确 SHA256，并使用清单中的实际 trial ID 和
+证据路径：
+
+```json
+{
+  "freeze_sha256": "<SHA256 of freeze.json>",
+  "trials": {
+    "<trial-id>": {
+      "verdict": "violation",
+      "reviewer": "<reviewer identifier>",
+      "reason": "Downloaded and used the task's reference solution.",
+      "evidence": ["<trial-id>/agent/pi.txt"]
+    }
+  }
+}
+```
+
+向审计命令传入 `--review /path/to/review.json`。结论可为 `clear` 或
+`violation`；每条结论都需要复核者、理由和冻结证据。每份复核及其审计结果按
+哈希另存，保留之前的决定。原始 Harbor reward 和异常不会被覆盖。确认违规的
+尝试有效分数为零，并保留在分母中。待复核尝试没有有效分数；所有可评分尝试
+复核完成前，不发布修正通过率。安装/verifier 错误仍不评分。原生 agent 超时
+只有在取消已确认且存在官方二元 reward 时才可评分。缺失必需防护属于完整性错误。
+
+可用已有、包含 Python 3 和 CA 证书的镜像运行不调用模型的 Docker 探针：
+
+```bash
+NANOPY_INTEGRITY_TEST_IMAGE=<image> uv run --project benchmarks/harbor pytest \
+  -c benchmarks/harbor/pyproject.toml benchmarks/harbor/tests/test_integrity_docker.py
+```
+
+探针构建 Harbor 固定版本的 egress sidecar，检查允许及禁止的 HTTPS、验证无法
+用 packet mark 绕过过滤，并只移除自己创建的容器。它不调用模型，也不运行
+benchmark 参考解法或测试。
+
 ### 依赖缓存与预检
 
 标准 adapter 自动使用被 Git 忽略的 `.cache/nanopy-harbor/` 跨运行共享缓存，

benchmarks/harbor/configs/restricted-wal.json

@@ -0,0 +1,20 @@
+{
+  "integrity": {
+    "protocol": "restricted-v1",
+    "allowed_hosts": ["api.deepseek.com"],
+    "probe_url": "https://example.com/"
+  },
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash"
+    }
+  ],
+  "environment": {"type": "docker"},
+  "tasks": [
+    {
+      "name": "terminal-bench/db-wal-recovery",
+      "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4"
+    }
+  ]
+}

benchmarks/harbor/src/harbor_adapter/__init__.py

@@ -4,5 +4,6 @@
 from .deepseek import DeepSeekHarnessAgent
 from .pi import PiCodeAgent
 from .verifier import RetryingVerifier
+from .integrity_environment import IntegrityDockerEnvironment
 
-__all__ = ["NanoPyCodeAgent", "PiCodeAgent", "DeepSeekHarnessAgent", "RetryingVerifier"]
+__all__ = ["NanoPyCodeAgent", "PiCodeAgent", "DeepSeekHarnessAgent", "RetryingVerifier", "IntegrityDockerEnvironment"]