feat(benchmarks): evaluate official deepseek harness against pi

benchmarks/harbor/README.md

@@ -88,13 +88,57 @@ pytest 8.4.1, torch 2.7.1, and pytest-json-ctrf 0.3.5 requirements, then runs on
 `pytest --version` from `/tmp`. It does not import the submitted model or run
 the task's tests. Other revisions require profile review.
 
+Reviewed revisions of `multi-source-data-merger`, `pytorch-model-cli`,
+`query-optimize`, `torch-pipeline-parallelism`, and `torch-tensor-parallelism`
+also have dependency-only profiles. They preserve each original wrapper's
+Python version, package pins, APT dependencies, and index options, and run only
+`pytest --version` from `/tmp`. In particular, only the CLI profile adds the
+original PyTorch CPU index and `unsafe-best-match` strategy; the other PyTorch
+profiles keep their original default index. Task metadata and verifier-wrapper
+hashes are recorded with these five profiles.
+
 Every installation writes `agent/bootstrap.json`, including partial progress
 on failure. Preflight failure stops the trial before any model call. The
 official verifier script remains unchanged and may still access the network
 when it runs; a successful preflight is not a guarantee against later outages.
 Full standard runs enable the timeout retry policy below by default, with zero
 whole-trial retries. CI checks these contracts without model calls or Docker.
 
+### Verified PyPI wheel transport
+
+Large original PyPI wheels can use the optional loopback cache:
+
+```bash
+python benchmarks/harbor/scripts/pypi_wheel_cache.py \
+  --cache .cache/nanopy-harbor/pypi-ranges --bind 127.0.0.1 --port 18744
+```
+
+For Docker tasks, run it in a sidecar sharing the task's network namespace and
+mount the cache only in that sidecar. Register
+`UV_DEFAULT_INDEX=http://127.0.0.1:18744/simple` in the task environment.
+The service always fetches current PyPI index metadata, preserving all versions,
+requirements, yanks, hashes, and extra-index settings. Only wheel URLs change to
+the loopback cache. Source-distribution URLs and PEP 658 metadata bytes remain
+original; metadata is forwarded fresh. Every wheel request rechecks its
+filename, size, SHA256, and original HTTPS URL against current PyPI release
+metadata before restoring any cached bytes. Metadata outages and corrupt files
+fail closed. The cache is not an offline or alternative package index.
+
+A cold wheel is fully downloaded and verified before its body is served.
+Allow a suitable uv HTTP timeout (for example `UV_HTTP_TIMEOUT=600`) within
+the separately registered setup deadline. Native agent and verifier timeouts
+remain unchanged.
+
+The existing bounded range downloader verifies every range and the complete
+wheel digest. File locks protect publication across task sidecars. Register the
+transport and its scripts before model work, run dependency-only preflight,
+and preserve original task images, scripts, package versions, sources, and
+per-attempt agent/verifier limits.
+
+Use `--audit-log /cache/run-events.jsonl` to retain current release metadata
+and verified cache-hit events after the sidecars exit. The append-only log is
+locked across processes and includes the originating container instance.
+
 ### Pi agent baselines
 
 The same entry point also supports `harbor_adapter:PiCodeAgent`. Register a local
@@ -148,6 +192,49 @@ preserves all 34 attempts, the supervision amendment, two unscored legacy
 timeouts, public-reference exposure, and partial accounting. It does not claim
 a complete, uncontaminated 34-task pass rate.
 
+### DeepSeek Harness baselines
+
+`harbor_adapter:DeepSeekHarnessAgent` runs the upstream Python SDK's
+`sdk-minimal` profile, as recommended by DeepSeek Harness's `BENCHMARK.md`.
+Register `distribution_path`, `distribution_sha256`, and optional
+`reasoning_effort` (default `high`) in the agent kwargs. The distribution is an
+offline `wheels/` directory containing the SDK, native runtime, and all Python
+dependencies, plus `distribution.json` with `version`, `source_commit`,
+`source_archive_sha256`, `sdk_wheel`, and `runtime_wheel`. The digest covers every
+file and permission; the standard workflow snapshots and rechecks the bytes.
+
+Build the runtime and SDK from the selected upstream checkout using its documented
+Python contributor workflow. Task installation uses Python 3.13 and the frozen
+wheelhouse. Installation-only checks initialize the actual SDK profile and model
+without sending a task prompt, and run the existing verifier dependency preflight.
+Model work requires `DEEPSEEK_API_KEY` and uses the official Messages endpoint.
+Each task receives a fresh Harness home and session, with upstream prompts,
+tools, compaction, retries, and generation defaults. Native logs remain in
+`agent/dsh/`; SDK events, completion status, and diagnostics remain beside it.
+
+Configurations must set `verifier.kwargs.require_agent_quiescence=true`.
+The shared Linux supervisor confirms cancellation of the agent process tree
+before verification. A non-completed SDK finish reason is an execution error.
+Reported tokens include native assistant and compaction usage; missing records
+and interrupted runs remain partial, and cost remains unknown. This baseline
+does not impose Pi's generation settings or Nano's turn and work budgets.
+
+The first [official Harness dev34 report](reports/en/deepseek-harness-dev34-20261030.md)
+records 22 passes, six scored failures, and six unscored errors on Pi's existing
+34-task pool. Normal SDK closure in the pinned upstream runtime stops managed
+background services, including the probe's `nohup` server. This affects service
+tasks: KV and PyPI failed official checks after successful agent self-tests.
+The report retains those failures and documents the
+[keyless lifecycle probe](scripts/probe_deepseek_lifecycle.py). Treat these
+results as measurements of the registered SDK configuration.
+
+The separately authorized [six-task supplement](reports/en/deepseek-harness-dev6-supplement-20261101.md)
+reran the unscored tasks with reviewed dependency preflight and verified original
+PyPI wheel transport: four passed, two failed, and none remained unscored. The
+original results stay unchanged. Combining the 28 original accepted attempts
+with these six new attempts gives mixed coverage of 26 passes and eight failures;
+this outcome-selected supplement is not an independent benchmark.
+
 ### Cache original task images
 
 If Docker image pulls repeatedly stall, the optional cache utility downloads the

benchmarks/harbor/README.zh-CN.md

@@ -71,11 +71,46 @@ uv 0.9.5、Python 3.13、pytest 8.4.1 和 pytest-json-ctrf 0.3.5，不执行题
 0.3.5，随后仅在 `/tmp` 执行 `pytest --version`。它不导入提交的模型，也不运行
 题目测试。其他题目版本需要重新检查对应配置。
 
+已检查的 `multi-source-data-merger`、`pytorch-model-cli`、`query-optimize`、
+`torch-pipeline-parallelism` 和 `torch-tensor-parallelism` 版本也有仅准备依赖的
+配置。它们保留各自原始脚本的 Python 版本、软件包版本、APT 依赖和索引选项，
+仅在 `/tmp` 运行 `pytest --version`。其中仅 CLI 配置添加原脚本中的 PyTorch
+CPU 索引和 `unsafe-best-match` 策略；其他 PyTorch 配置保留原始默认索引。
+每个新检查的配置都记录题目元数据与验证器脚本的哈希。
+
 每次安装都会写入 `agent/bootstrap.json`，失败时也保留已完成的步骤。预检失败会
 在模型调用前终止 trial。官方验证脚本保持原样，执行时仍可能访问网络，所以预检
 通过不能保证后续没有网络故障。完整标准运行默认启用下述验证超时重试，整题重试
 次数为零。CI 会验证这些约定，不调用模型、不启动 Docker。
 
+### 校验后的 PyPI wheel 传输
+
+较大的原始 PyPI wheel 可使用可选的本地回环缓存：
+
+```bash
+python benchmarks/harbor/scripts/pypi_wheel_cache.py \
+  --cache .cache/nanopy-harbor/pypi-ranges --bind 127.0.0.1 --port 18744
+```
+
+Docker 题目中，让 sidecar 与题目共享网络命名空间，并仅向 sidecar 挂载缓存。
+在题目环境中注册 `UV_DEFAULT_INDEX=http://127.0.0.1:18744/simple`。
+服务始终获取当前 PyPI 索引元数据，保留所有版本、依赖要求、撤回标记、哈希和
+额外索引设置，仅将 wheel URL 指向回环缓存。源码分发 URL 与 PEP 658 元数据
+内容保持原样，元数据每次从上游转发。每次 wheel 请求都先根据当前 PyPI 发布
+元数据重新核对文件名、大小、SHA256 和原始 HTTPS URL，再恢复缓存内容。
+元数据不可用或文件损坏时直接失败，不使用离线或替代软件源。
+
+首次请求的 wheel 会先完整下载并校验，再返回内容。应在单独注册的安装时限
+范围内设置合适的 uv HTTP 等待时间（例如 `UV_HTTP_TIMEOUT=600`）。原生
+agent 和验证器时限保持不变。
+
+已有的有界分段下载器会校验每个范围及整个 wheel 的摘要。文件锁保护多个
+题目 sidecar 之间的缓存发布。模型运行前应注册传输配置及脚本，执行仅依赖
+预检，并保留原始题目镜像、脚本、软件包版本、来源以及每次 agent / 验证器时限。
+
+使用 `--audit-log /cache/run-events.jsonl`，可在 sidecar 退出后保留当前发布
+元数据及已校验的缓存命中事件。追加日志使用跨进程锁，并记录来源容器实例。
+
 ### Pi agent baseline
 
 同一个标准入口也支持 `harbor_adapter:PiCodeAgent`。在 agent kwargs 中用
@@ -116,6 +151,43 @@ Docker 空间受限时，可以通过 `batch_images` 计划按批恢复已注册
 保留了全部 34 次尝试、进程托管修正、两次不评分的旧超时、公开参考材料接触和
 不完整用量统计，不宣称完整且无答案接触的 34 题通过率。
 
+### DeepSeek Harness baseline
+
+`harbor_adapter:DeepSeekHarnessAgent` 使用上游 Python SDK 的 `sdk-minimal`
+配置运行，这是 DeepSeek Harness 的 `BENCHMARK.md` 推荐的基准配置。
+在 agent kwargs 中注册 `distribution_path`、`distribution_sha256` 和可选的
+`reasoning_effort`（默认为 `high`）。分发目录包含离线 `wheels/` 目录，内有
+SDK、原生运行时以及所有 Python 依赖；另有 `distribution.json`，记录
+`version`、`source_commit`、`source_archive_sha256`、`sdk_wheel` 和
+`runtime_wheel`。摘要覆盖每个文件及其权限；标准流程会保存快照并重新校验。
+
+按照选定上游检出的 Python 贡献者文档构建运行时和 SDK。任务安装使用
+Python 3.13 和冻结的 wheel 集合。仅安装检查会初始化真实 SDK 配置和模型，
+但不发送任务提示，并执行已有的验证器依赖预检。模型执行需要
+`DEEPSEEK_API_KEY`，使用官方 Messages 端点。每道题都使用全新的 Harness
+目录与会话，保留上游提示词、工具、上下文压缩、重试及生成默认值。原生日志
+保存在 `agent/dsh/`，SDK 事件、完成状态和诊断信息保存在旁边。
+
+配置必须设置 `verifier.kwargs.require_agent_quiescence=true`。
+共享的 Linux 进程监督器在验证前确认 agent 进程树已被取消。SDK 返回非
+`completed` 的结束原因时记为执行错误。报告的 token 包含原生助手消息和
+压缩请求用量；缺失记录和中断运行仍标记为部分统计，费用保持未知。
+此 baseline 不强制套用 Pi 的生成设置或 Nano 的轮数和工作时间预算。
+
+首份[官方 Harness dev34 报告](reports/zh-CN/deepseek-harness-dev34-20261030.md)
+在 Pi 已使用的 34 道题上记录了 22 道通过、六道计分失败、六道因错误不计分。
+固定版本的上游运行时在正常关闭 SDK 时会停止托管的后台服务，包括探针中
+通过 `nohup` 启动的服务。这会影响服务题：KV 和 PyPI 在 agent 自测成功后，
+官方检查仍然失败。报告保留这些失败，并记录了
+[无密钥生命周期探针](scripts/probe_deepseek_lifecycle.py)。这些结果衡量的是
+注册的 SDK 配置。
+
+另行授权的[六题补测](reports/zh-CN/deepseek-harness-dev6-supplement-20261101.md)
+使用经过审核的依赖预检和经过校验的原始 PyPI wheel 下载通道，重跑未计分题目：
+四题通过、两题失败，无剩余未计分项。原始结果保持不变。将原始 28 次有效尝试
+与这六次新尝试结合，得到 26 题通过、八题失败的混合覆盖；这种根据原始结果
+选取的补测不构成独立基准。
+
 ### 缓存原始任务镜像
 
 如果 Docker 拉取镜像反复停滞，可用缓存工具从同一 Docker Hub 公共仓库分段并行下载：