chore(benchmarks): record flash 1017 dev34 baseline and recovery

benchmarks/harbor/README.md

@@ -82,6 +82,12 @@ assertions or solution code. Unknown QEMU revisions fail setup pending profile
 review; tasks without a profile explicitly report `not_configured`. Add a
 profile and regression tests when another task needs verifier preflight.
 
+The reviewed `pytorch-model-recovery` revision also has a dependency-only
+profile. It uses the original uv 0.9.5 installer and the verifier's Python 3.13,
+pytest 8.4.1, torch 2.7.1, and pytest-json-ctrf 0.3.5 requirements, then runs only
+`pytest --version` from `/tmp`. It does not import the submitted model or run
+the task's tests. Other revisions require profile review.
+
 Every installation writes `agent/bootstrap.json`, including partial progress
 on failure. Preflight failure stops the trial before any model call. The
 official verifier script remains unchanged and may still access the network
@@ -295,6 +301,57 @@ valid scored failures stop immediately. Exhausted dependency failures raise
 canonical reward. The default remains timeout-only; record the authorization
 and selected policy in the experiment plan.
 
+For separately authorized recovery of pytest collection failures, also set
+`retry_collection_errors=true`. This retries only an explicit collection abort
+before any test case runs. A successful reward, executed test cases, or an
+ambiguous CTRF report prevents this recovery. Exhaustion raises
+`VerifierCollectionError` and preserves archived rewards without publishing a
+canonical score. The default is false, and the three-attempt limit is shared
+with timeout and dependency recovery.
+
+When future verification may be needed, set Docker's
+`environment.kwargs.keep_containers=true` to retain the stopped task containers
+and register solution paths such as `/app` in `artifacts`.
+`environment.delete=false` alone still removes Docker containers. Harbor collects
+the registered files before verification. Keep both the
+artifact manifest and file hashes with the run evidence; trajectories alone do
+not preserve binaries or the original system state.
+
+For an explicitly authorized additional verifier batch on a retained Docker
+trial, use the repository entry point without starting an agent:
+
+```bash
+uv run --project benchmarks/harbor python -m harbor_adapter verify-retained \
+  --trial-dir jobs/example/task__trial \
+  --task-dir /path/to/the/original/task \
+  --container task__trial__env-main-1 \
+  --output-dir jobs/example-additional-verifier-batch
+```
+
+All original project services must be stopped. The command starts those same
+containers, waits for service health checks, verifies frozen `/app` artifacts
+and original test hashes, and runs at most three attempts with the native task
+timeout. The optional `--retry-dependency-errors` and
+`--retry-collection-errors` flags require the same separate authorization as
+their verifier kwargs. Any completed score, including zero, stops the batch.
+The original trial logs are temporarily set aside inside the retained container
+and restored afterward. New attempt logs, implementation sources, policy,
+hashes, and integrity checks go into the separate output directory. Containers
+are stopped and retained after the batch. This does not change the original
+trial result or authorize repeated batches automatically.
+
+To give dependency preparation its own budget on a retained solution, add
+`--dependency-timeout-sec 3600`. This requires a reviewed profile for the exact
+task revision and checks its registered task-file hashes. Preparation runs once
+in the same container before verification; its logs and status are recorded
+under `dependency-preparation/`. A preparation timeout or nonzero exit is a
+setup failure with no score, and verification does not start. On success, the
+unchanged official verifier script runs with the original per-attempt timeout
+(900 seconds for `pytorch-model-recovery`). Its own remaining dependency checks
+are still included in those 900 seconds. The larger preparation deadline never
+extends the verifier deadline. Record this preparation policy separately from
+earlier baseline results, even when the solution is identical.
+
 ## Benchmark reports
 
 Reports are available in [English](reports/en/) and

benchmarks/harbor/README.zh-CN.md

@@ -66,6 +66,11 @@ uv 0.9.5、Python 3.13、pytest 8.4.1 和 pytest-json-ctrf 0.3.5，不执行题
 参考解。未知的 QEMU 版本会在安装阶段报错，要求先检查对应预检配置；没有预检配置
 的其他题目明确记录为 `not_configured`。其他题目需要预检时，在这里新增配置及回归测试。
 
+已检查的 `pytorch-model-recovery` 版本也有仅准备依赖的配置。它使用原始 uv 0.9.5
+安装器和验证器所需的 Python 3.13、pytest 8.4.1、torch 2.7.1、pytest-json-ctrf
+0.3.5，随后仅在 `/tmp` 执行 `pytest --version`。它不导入提交的模型，也不运行
+题目测试。其他题目版本需要重新检查对应配置。
+
 每次安装都会写入 `agent/bootstrap.json`，失败时也保留已完成的步骤。预检失败会
 在模型调用前终止 trial。官方验证脚本保持原样，执行时仍可能访问网络，所以预检
 通过不能保证后续没有网络故障。完整标准运行默认启用下述验证超时重试，整题重试
@@ -238,6 +243,45 @@ Harbor 原生的 provider 凭证和已经配置的 base URL 也会转换为 nano
 评分也会立即停止。依赖错误耗尽尝试后抛出 `VerifierDependencyError`，保留每次
 证据并清除当前评分。默认仍只重试超时；实验计划应记录授权和实际选用的策略。
 
+对单独授权的 pytest 收集失败恢复，还可设置
+`retry_collection_errors=true`。它只重试尚未运行任何测试用例时明确发生的收集中断。
+成功评分、已执行的测试用例或含义不明确的 CTRF 报告都会阻止这种恢复。耗尽次数后
+抛出 `VerifierCollectionError`，保留归档评分，但不发布当前评分。默认值为 false，
+与超时及依赖恢复共享最多三次的尝试上限。
+
+如果之后可能需要继续验证，应设置 Docker 的
+`environment.kwargs.keep_containers=true` 保留已停止的任务容器，并在 `artifacts`
+中登记 `/app` 等解答路径。仅设置 `environment.delete=false` 仍会删除 Docker
+容器。Harbor 会在验证前收集已登记的文件。
+将产物清单和文件哈希与运行证据一起保存；仅有轨迹无法保留二进制文件和原始系统状态。
+
+对于明确授权、使用已保留 Docker trial 的额外 verifier 批次，可通过仓库入口执行，
+无需启动 agent：
+
+```bash
+uv run --project benchmarks/harbor python -m harbor_adapter verify-retained \
+  --trial-dir jobs/example/task__trial \
+  --task-dir /path/to/the/original/task \
+  --container task__trial__env-main-1 \
+  --output-dir jobs/example-additional-verifier-batch
+```
+
+原项目的所有服务必须处于停止状态。该命令启动原来的容器，等待服务健康检查通过，
+核对冻结的 `/app` 产物和原始测试的哈希，并使用题目原生时限执行最多三次尝试。
+可选的 `--retry-dependency-errors` 和 `--retry-collection-errors` 标志与对应的
+verifier 参数一样，需要单独授权。任何已完成的评分都会结束该批次，包括得分为 0。
+原 trial 日志会暂存在保留容器内，并在结束后恢复。新增尝试日志、执行源码、策略、
+哈希和完整性检查写入单独的输出目录。批次结束后，容器会被停止并继续保留。
+该命令不会修改原 trial 的结果，也不会自动授权后续重复批次。
+
+若要为保留解答的依赖准备单独分配时间，可添加 `--dependency-timeout-sec 3600`。
+这要求精确题目版本已有检查过的依赖配置，并核对其中登记的任务文件哈希。依赖准备
+在同一容器中、正式验证前执行一次，日志和状态单独记录到 `dependency-preparation/`。
+准备超时或以非零状态退出时，结果为安装失败、不评分，也不会启动正式验证。准备
+成功后，保持原样的官方验证脚本按原生单次时限执行（`pytorch-model-recovery` 为
+900 秒）。脚本内剩余的依赖检查仍计入这 900 秒。更长的准备时限不会延长正式验证
+时限。即使解答相同，也应将这一准备策略与先前 baseline 分开记录。
+
 ## Benchmark 报告
 
 报告提供[英文](reports/en/)和[简体中文](reports/zh-CN/)两个版本，文件名一一对应。

benchmarks/harbor/configs/dev34-flash1017-main-20261019/README.md

@@ -0,0 +1,113 @@
+# Current-main dev34 baseline with DeepSeek Flash 1017
+
+**Completed:** 26 passes, five scored failures, and three unscored verifier errors. All 34 tasks started a fresh agent trial. See the [English report](../../reports/en/dev34-flash1017-main-20261019.md), [complete Simplified Chinese report](../../reports/zh-CN/dev34-flash1017-main-20261019.md) and [audited results](../../results/dev34-flash1017-main-20261019.json). Native zero rewards for pre-assertion verifier failures remain preserved; analytical scores are null. The CLI dependency failure followed agent-forced package installations. A passing Scheme evaluator uses a host depth-counter workaround, recorded separately.
+
+This registration runs all 34 existing development tasks once with fresh model
+outputs from current main `ce28b1e826a292d796ba64f13e79199b66f65758`. The experiment branch and its independent
+runtime checkout are separate, so recording results cannot change the evaluated
+source. Historical scores and outputs remain separate.
+
+## Frozen inputs and limits
+
+- [Active plan](experiment-v2.json): source, model mapping, task and infrastructure hashes,
+  budgets, installation preflight, and the complete fixed queue.
+- [Selection](selection.json): the unchanged dev34 pool and pinned task revisions.
+- [Prompt](main-prompt.txt): current main's 385-word headless prompt.
+- [Images](image-digests.json): image identities checked before execution.
+- Runtime checkout: `jobs/main-source`, detached at `ce28b1e826a292d796ba64f13e79199b66f65758`.
+
+The model is `deepseek/deepseek-flash` at the official DeepSeek Anthropic endpoint.
+[Official documentation](https://api-docs.deepseek.com/quick_start/agent_integrations/)
+was checked on 2026-10-19 and maps this alias to DeepSeek-V4.1-Flash-1017. The
+alias does not pin an immutable checkpoint. Temperature and reasoning effort are
+unset. Each task allows 100 replies, 65,536 output tokens per reply, and its native
+agent timeout minus 180 seconds for model work; setup separately allows 1,800
+seconds. Original task images, resources, package sources, official scripts, and
+per-attempt verifier limits stay unchanged.
+
+Run fresh installation checks for all 34 tasks before starting any model work.
+Only QEMU has a registered verifier dependency profile; the other 33 report
+`not_configured`. Agent installation success does not certify those dependencies.
+The verified APT index and original uv/Python archive caches retain upstream
+filename, size, and SHA256 checks. The [archive manifest](dependency-release-manifest.json)
+is reused unchanged from the October 18 run, including all five original archives.
+
+The model queue sorts by ascending work budget and task name, with consecutive
+pair barriers and maximum concurrency two. Every task generates exactly one new
+output. Only verifier timeouts may retry, at most three total attempts with the
+same output and original per-attempt limits. Dependency-error recovery remains
+disabled. Setup failures and verifier errors are recorded separately from scored
+failures; a zero without executed assertions is analytically unscored.
+
+## Installation preflight history
+
+All 34 tasks now have fresh, successful native installation-only records from
+this frozen checkout. The first batch completed 33 installations and recorded
+one `NetworkConnectionError`: TLS failed while downloading the unchanged
+`https://astral.sh/uv/0.9.11/install.sh` script for the Scheme evaluator. A new,
+separately named installation-only trial for that task succeeded. No model or
+official verifier ran during either preflight job.
+
+The [original registration](experiment.json) and all 35 attempts remain intact.
+[Native evidence](preflight-evidence.json) binds all results, bootstrap records,
+source manifests, and input configurations by hash. The active v2 plan accepts
+exactly one successful native record per task through the existing strict
+orchestrator checks. Model configurations, task revisions, environment transport,
+source, budgets, and timeout-only verifier policy remain unchanged. QEMU's
+registered verifier profile passed; the other 33 remain `not_configured`.
+
+## Execute
+
+From this experiment worktree's repository root, supply credentials through the
+process environment, set `ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic`,
+and leave `ANTHROPIC_MODEL` and `ANTHROPIC_MAX_TOKENS` unset.
+
+```bash
+python benchmarks/harbor/scripts/run_comparison.py \
+  --plan benchmarks/harbor/configs/dev34-flash1017-main-20261019/experiment-v2.json \
+  --phase preflight
+
+python benchmarks/harbor/scripts/run_comparison.py \
+  --plan benchmarks/harbor/configs/dev34-flash1017-main-20261019/experiment-v2.json \
+  --phase model
+```
+
+The existing orchestrator calls only the standard
+`uv run --project benchmarks/harbor python -m harbor_adapter run --config ...`
+entry point from the frozen runtime checkout. Native jobs and input manifests
+live under `jobs/main-source/jobs/`; orchestration records live under
+`jobs/dev34-1017-main-20261019-orchestration/`. Existing jobs cannot be overwritten.
+
+Inspect operational stages and completion metadata during execution, then freeze
+all 34 outputs before reviewing new trajectories or hidden-test content. Raw logs
+remain local; publish derived findings and evidence hashes after review. All 34
+tasks have prior development exposure. The remaining 55 dataset tasks stay
+unopened, and this run does not isolate a model-checkpoint effect.
+
+## Host launch failures and supplemental jobs
+
+Three primary jobs failed before any native trial or model output existed:
+Git multibranch and gRPC KV store could not connect to the Harbor registry;
+log-summary date ranges could not resolve wheel build requirements because the
+PyPI TLS connection failed. The original empty summaries, logs, snapshots, and
+job records remain intact, bound by [launch evidence](launch-failure-evidence.json).
+These are host launch errors, separate from task setup, verifier errors, and
+model correctness failures.
+
+Three single-task supplemental plans register one new output for each task:
+[Git](launch-recovery-git-multibranch.json),
+[KV](launch-recovery-kv-store-grpc.json), and
+[logs](launch-recovery-log-summary-date-ranges.json).
+They use the exact same frozen main checkout, configurations, successful native
+installation evidence, images, prompt, resources, and budgets. Original jobs
+remain failed; generated outputs and scored results are never replaced.
+
+Start these plans only when the original queue reaches its final pair and one
+of those jobs has ended, leaving one free slot with no later primary jobs to
+start. Execute the three single-task plans serially through the existing
+orchestrator and standard entry point. Total task concurrency stays at two or
+fewer. All primary and supplemental outputs must freeze before new trajectory
+or official-test-content review. Report all original launch failures and the
+supplemental scheduling separately from the original fixed queue.
+
+All primary and supplemental jobs completed. The six relocated OCI archives and 44 cached blobs were restored and hash-verified. Final inspection retained all 34 image identities and found no remaining task containers.

benchmarks/harbor/configs/dev34-flash1017-main-20261019/adaptive-rejection-sampler.json

@@ -0,0 +1,35 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 720
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/adaptive-rejection-sampler",
+      "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de"
+    }
+  ],
+  "environment": {
+    "type": "docker",
+    "extra_docker_compose": [
+      "/home/minix/projects/nanoPyCodeAgent/jobs/dev34-flash1017-main-20261019/benchmarks/harbor/configs/dev34-flash1017-main-20261019/dependency-cache.compose.json"
+    ]
+  },
+  "verifier": {
+    "import_path": "harbor_adapter:RetryingVerifier",
+    "kwargs": {
+      "max_attempts": 3,
+      "retry_dependency_errors": false
+    }
+  }
+}

benchmarks/harbor/configs/dev34-flash1017-main-20261019/apt-cache.conf

@@ -0,0 +1,8 @@
+// Route only public APT HTTP origins through the verified index cache.
+Acquire::http::Proxy::deb.debian.org "http://127.0.0.1:18742/";
+Acquire::http::Proxy::security.debian.org "http://127.0.0.1:18742/";
+Acquire::http::Proxy::archive.debian.org "http://127.0.0.1:18742/";
+Acquire::http::Proxy::archive.ubuntu.com "http://127.0.0.1:18742/";
+Acquire::http::Proxy::security.ubuntu.com "http://127.0.0.1:18742/";
+Acquire::http::Proxy::ports.ubuntu.com "http://127.0.0.1:18742/";
+Acquire::http::Proxy::old-releases.ubuntu.com "http://127.0.0.1:18742/";

benchmarks/harbor/configs/dev34-flash1017-main-20261019/build-cython-ext.json

@@ -0,0 +1,35 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 720
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/build-cython-ext",
+      "ref": "sha256:35dc561790e73f13e59837828ac0863c0893edc6b71d42cf7d58656c843adf12"
+    }
+  ],
+  "environment": {
+    "type": "docker",
+    "extra_docker_compose": [
+      "/home/minix/projects/nanoPyCodeAgent/jobs/dev34-flash1017-main-20261019/benchmarks/harbor/configs/dev34-flash1017-main-20261019/dependency-cache.compose.json"
+    ]
+  },
+  "verifier": {
+    "import_path": "harbor_adapter:RetryingVerifier",
+    "kwargs": {
+      "max_attempts": 3,
+      "retry_dependency_errors": false
+    }
+  }
+}

benchmarks/harbor/configs/dev34-flash1017-main-20261019/build-pov-ray.json

@@ -0,0 +1,35 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 11820
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/build-pov-ray",
+      "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb"
+    }
+  ],
+  "environment": {
+    "type": "docker",
+    "extra_docker_compose": [
+      "/home/minix/projects/nanoPyCodeAgent/jobs/dev34-flash1017-main-20261019/benchmarks/harbor/configs/dev34-flash1017-main-20261019/dependency-cache.compose.json"
+    ]
+  },
+  "verifier": {
+    "import_path": "harbor_adapter:RetryingVerifier",
+    "kwargs": {
+      "max_attempts": 3,
+      "retry_dependency_errors": false
+    }
+  }
+}

benchmarks/harbor/configs/dev34-flash1017-main-20261019/caffe-cifar-10.json

@@ -0,0 +1,35 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 3420
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/caffe-cifar-10",
+      "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf"
+    }
+  ],
+  "environment": {
+    "type": "docker",
+    "extra_docker_compose": [
+      "/home/minix/projects/nanoPyCodeAgent/jobs/dev34-flash1017-main-20261019/benchmarks/harbor/configs/dev34-flash1017-main-20261019/dependency-cache.compose.json"
+    ]
+  },
+  "verifier": {
+    "import_path": "harbor_adapter:RetryingVerifier",
+    "kwargs": {
+      "max_attempts": 3,
+      "retry_dependency_errors": false
+    }
+  }
+}