Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions benchmarks/harbor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,12 @@ preserves all 34 attempts, the supervision amendment, two unscored legacy
timeouts, public-reference exposure, and partial accounting. It does not claim
a complete, uncontaminated 34-task pass rate.

The authorized [three-task Pi rerun](reports/en/pi-deepseek-flash-dev3-rerun-20261102.md)
recorded two official passes and one scored failure after confirmed timeout
cancellation. Both new passes accessed public reference solutions and tests;
neither is an independent solution. Original outcomes remain unchanged. The
rerun also fixed Pi installation checks when Harbor returns absent output streams.

### DeepSeek Harness baselines

`harbor_adapter:DeepSeekHarnessAgent` runs the upstream Python SDK's
Expand Down
5 changes: 5 additions & 0 deletions benchmarks/harbor/README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,11 @@ Docker 空间受限时,可以通过 `batch_images` 计划按批恢复已注册
保留了全部 34 次尝试、进程托管修正、两次不评分的旧超时、公开参考材料接触和
不完整用量统计,不宣称完整且无答案接触的 34 题通过率。

经授权的[三题 Pi 重跑](reports/zh-CN/pi-deepseek-flash-dev3-rerun-20261102.md)
记录了两次官方通过,以及一次超时后确认停止的计分失败。两次新的通过都访问了
公开参考解法和测试,均不属于独立解题。原始结果保持不变。本次重跑还修复了
Harbor 返回缺失输出流时 Pi 安装检查失败的问题。

### DeepSeek Harness baseline

`harbor_adapter:DeepSeekHarnessAgent` 使用上游 Python SDK 的 `sdk-minimal`
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Authorized three-task Pi rerun

Status: completed and audited on November 2, 2026. **Two official passes, both
reference-exposed, and one scored failure after confirmed timeout cancellation;
zero unscored attempts.** WAL and pipeline both read reference solutions and
tests whose reconstructed bytes match the registered task hashes. Sampler failed
all nine official tests after the supervisor stopped 11 processes. These results
provide no independent passing solution from the three reruns.

See the complete [English report](../../reports/en/pi-deepseek-flash-dev3-rerun-20261102.md),
[Simplified Chinese report](../../reports/zh-CN/pi-deepseek-flash-dev3-rerun-20261102.md),
and [structured results](../../results/pi-deepseek-flash-dev3-rerun-20261102.json).
[Preflight evidence](preflight-evidence.json) and [launch evidence](launch-evidence.json)
precede the model jobs. The [freeze manifest](outputs-freeze.json) covers 326
files before trajectory review; all 1,449 original frozen artifacts remain
unchanged. WAL exposed the credential in two private raw logs. All three official
verifiers completed on their first attempt. No containers remain running;
the six initially protected images remain present. See
[validation evidence](validation-evidence.json).

The maintainer requested one fresh Pi attempt on `adaptive-rejection-sampler`,
`db-wal-recovery`, and `torch-pipeline-parallelism`. The first two attempts in the
[original baseline](../../reports/en/pi-deepseek-flash-dev34-20261024.md) were
unscored because their timeout boundaries were not confirmed. The original
pipeline attempt passed after reading public reference tests and a solution.
This is an outcome-selected development supplement, not an independent benchmark.
Keep every original outcome and every new attempt separately; do not select the
better score.

The [v2 registration](experiment-v2.json) preserves Pi 1.4.0 and its complete
distribution digest, the original public model configuration,
`deepseek/deepseek-flash`, the official OpenAI-compatible endpoint, `high`
reasoning, native prompts/tools/compaction/retries, and unset temperature.
The alias is mutable; identical backend weights across dates are not established.
All three tasks retain their original revisions, files, Docker image identities,
resources, 900-second agent limit, and 900-second per-attempt verifier limit.
Setup has its separate 1,800-second limit. Concurrency is two with pair barriers,
in the original relative task order. There are no whole-trial retries.

The registered standard adapter source is
`f3a171fcbc390c6bcb0c941be6b9cdd9dc7c456c`. Each job snapshots and hashes it through
`python -m harbor_adapter run --config ...`. Confirmed cancellation and mandatory
agent quiescence protect the agent/verifier boundary. Only verifier timeouts may
retry, at most three attempts on the same output with the original time limit.
Setup errors and verifier errors remain distinct from scored failures.

The original registration used `46c554c2cf31a2d93bb37493083aab5b8fb82012`.
All three checks failed during Pi model-registration validation because Harbor
returned `None` for empty stderr. [Initial evidence](preflight-v1-evidence.json)
preserves those setup failures and confirms zero model or verifier executions.
The v2 source normalizes absent output streams while retaining rejection of
diagnostics and missing provider/model rows. Three regression cases failed
before the fix; all 304 Harbor tests passed after it. The initial model queue
remains unexecuted; v2 has distinct job names and orchestration records.

Installation checks precede all model work. Pipeline now has the repository's
reviewed `torch-pipeline-parallelism-v1` dependency profile. Sampler and WAL
explicitly remain `not_configured`; successful Pi installation does not certify
their verifier dependencies. Existing verified APT, uv/Python-release, and PyPI
wheel caches preserve original packages and sources. The PyPI transport checks
each wheel's filename, size, SHA256, and original URL against current release
metadata. `UV_HTTP_TIMEOUT=600` permits verification before serving cold wheels;
task-native limits remain unchanged. Cache events have a separate audit log.

Each task starts a fresh container and Pi session. Earlier trajectories, failure
diagnostics, tests, and solutions are not supplied to the agent. No new prompt or
network restriction is introduced. Freeze all new outputs before reviewing new
trajectories or hidden assertions, then audit reference-material access. A rerun
alone does not establish absence of answer exposure. Keep raw logs private and
publish complete English and Simplified Chinese reports with the same evidence.

The following commands document the original launch from its pinned source
revision. Completed job names and registrations are immutable. A future repeat
needs a new registration and fresh job names, with its source checkout explicitly
pinned; do not reuse this completed queue from a newer repository revision.

```bash
export UV_PROJECT_ENVIRONMENT="$PWD/jobs/dsh-harbor-venv"
export UV_CACHE_DIR="$PWD/.cache/uv"
uv run --frozen --project benchmarks/harbor \
python benchmarks/harbor/scripts/run_comparison.py \
--plan benchmarks/harbor/configs/pi-deepseek-flash-dev3-rerun-20261102/experiment-v2.json \
--phase preflight
# Supply the authorized DeepSeek credential through DEEPSEEK_API_KEY.
uv run --frozen --project benchmarks/harbor \
python benchmarks/harbor/scripts/run_comparison.py \
--plan benchmarks/harbor/configs/pi-deepseek-flash-dev3-rerun-20261102/experiment-v2.json \
--phase model
```

Native job outputs are under `jobs/pi-dev3-rerun-*-20261102/`, with immutable
inputs in the corresponding `-input/` directories. Orchestration records are in
`jobs/pi-deepseek-flash-dev3-rerun-20261102-v2-orchestration/`, with initial failed
checks retained under `jobs/pi-deepseek-flash-dev3-rerun-20261102-orchestration/`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"n_concurrent_trials": 1,
"n_attempts": 1,
"retry": {
"max_retries": 0
},
"agents": [
{
"name": "harbor_adapter:PiCodeAgent",
"model_name": "deepseek/deepseek-flash",
"override_setup_timeout_sec": 1800,
"kwargs": {
"distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi",
"distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9",
"model_config_path": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-deepseek-flash-dev3-rerun-20261102/pi-models.json",
"thinking": "high"
}
}
],
"environment": {
"type": "docker",
"extra_docker_compose": [
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-deepseek-flash-dev3-rerun-20261102/dependency-cache.compose.json"
]
},
"tasks": [
{
"name": "terminal-bench/adaptive-rejection-sampler",
"ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de"
}
],
"verifier": {
"import_path": "harbor_adapter:RetryingVerifier",
"kwargs": {
"max_attempts": 3,
"require_agent_quiescence": true
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
// Route only public APT HTTP origins through the verified index cache.
Acquire::http::Proxy::deb.debian.org "http://127.0.0.1:18742/";
Acquire::http::Proxy::security.debian.org "http://127.0.0.1:18742/";
Acquire::http::Proxy::archive.debian.org "http://127.0.0.1:18742/";
Acquire::http::Proxy::archive.ubuntu.com "http://127.0.0.1:18742/";
Acquire::http::Proxy::security.ubuntu.com "http://127.0.0.1:18742/";
Acquire::http::Proxy::ports.ubuntu.com "http://127.0.0.1:18742/";
Acquire::http::Proxy::old-releases.ubuntu.com "http://127.0.0.1:18742/";
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"n_concurrent_trials": 1,
"n_attempts": 1,
"retry": {
"max_retries": 0
},
"agents": [
{
"name": "harbor_adapter:PiCodeAgent",
"model_name": "deepseek/deepseek-flash",
"override_setup_timeout_sec": 1800,
"kwargs": {
"distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi",
"distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9",
"model_config_path": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-deepseek-flash-dev3-rerun-20261102/pi-models.json",
"thinking": "high"
}
}
],
"environment": {
"type": "docker",
"extra_docker_compose": [
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-deepseek-flash-dev3-rerun-20261102/dependency-cache.compose.json"
]
},
"tasks": [
{
"name": "terminal-bench/db-wal-recovery",
"ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4"
}
],
"verifier": {
"import_path": "harbor_adapter:RetryingVerifier",
"kwargs": {
"max_attempts": 3,
"require_agent_quiescence": true
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
{
"services": {
"main": {
"volumes": [
{
"type": "bind",
"source": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-deepseek-flash-dev3-rerun-20261102/apt-cache.conf",
"target": "/etc/apt/apt.conf.d/99nanopy-index-cache",
"read_only": true
}
],
"environment": {
"UV_INSTALLER_GITHUB_BASE_URL": "http://127.0.0.1:18743",
"UV_PYTHON_INSTALL_MIRROR": "http://127.0.0.1:18743/astral-sh/python-build-standalone/releases/download",
"UV_HTTP_TIMEOUT": "600",
"UV_DEFAULT_INDEX": "http://127.0.0.1:18744/simple"
}
},
"apt-index-cache": {
"image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67",
"network_mode": "service:main",
"entrypoint": [
"python3",
"/proxy/apt_index_proxy.py",
"--bind",
"127.0.0.1",
"--port",
"18742",
"--cache",
"/cache",
"--workers",
"16"
],
"user": "1000:1000",
"read_only": true,
"cap_drop": [
"ALL"
],
"security_opt": [
"no-new-privileges:true"
],
"mem_limit": "256m",
"cpus": 0.5,
"volumes": [
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/scripts/apt_index_proxy.py:/proxy/apt_index_proxy.py:ro",
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/scripts/cache_image.py:/proxy/cache_image.py:ro",
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/nanopy-harbor/apt-indexes:/cache"
],
"healthcheck": {
"test": [
"CMD",
"python3",
"-c",
"import socket; socket.create_connection((\"127.0.0.1\",18742), timeout=2).close()"
],
"interval": "1s",
"timeout": "3s",
"retries": 30
}
},
"uv-release-cache": {
"image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67",
"network_mode": "service:main",
"entrypoint": [
"python3",
"/proxy/uv_release_cache.py",
"serve",
"--cache",
"/cache",
"--manifest",
"/manifest.json"
],
"user": "1000:1000",
"read_only": true,
"cap_drop": [
"ALL"
],
"security_opt": [
"no-new-privileges:true"
],
"cpus": 0.5,
"mem_limit": "256m",
"volumes": [
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/scripts/uv_release_cache.py:/proxy/uv_release_cache.py:ro",
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/scripts/cache_image.py:/proxy/cache_image.py:ro",
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/nanopy-harbor/uv-releases:/cache:ro",
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-deepseek-flash-dev3-rerun-20261102/dependency-release-manifest.json:/manifest.json:ro"
],
"healthcheck": {
"test": [
"CMD",
"python3",
"-c",
"import socket; socket.create_connection((\"127.0.0.1\",18743), timeout=2).close()"
],
"interval": "1s",
"timeout": "3s",
"retries": 30
}
},
"pypi-wheel-cache": {
"image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67",
"network_mode": "service:main",
"entrypoint": [
"python3",
"/proxy/pypi_wheel_cache.py",
"--cache",
"/cache",
"--bind",
"127.0.0.1",
"--port",
"18744",
"--workers",
"16",
"--downloads",
"2",
"--audit-log",
"/cache/audit/pi-dev3-rerun-20261102.jsonl"
],
"user": "1000:1000",
"read_only": true,
"cap_drop": [
"ALL"
],
"security_opt": [
"no-new-privileges:true"
],
"mem_limit": "512m",
"cpus": 1.0,
"volumes": [
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/scripts/pypi_wheel_cache.py:/proxy/pypi_wheel_cache.py:ro",
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/scripts/cache_image.py:/proxy/cache_image.py:ro",
"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/nanopy-harbor/pypi-ranges:/cache"
],
"healthcheck": {
"test": [
"CMD",
"python3",
"-c",
"import socket; socket.create_connection((\"127.0.0.1\",18744), timeout=2).close()"
],
"interval": "1s",
"timeout": "3s",
"retries": 30
}
}
}
}
Loading
Loading