diff --git a/benchmarks/harbor/README.md b/benchmarks/harbor/README.md index c85e13b..8f54787 100644 --- a/benchmarks/harbor/README.md +++ b/benchmarks/harbor/README.md @@ -90,8 +90,8 @@ does not count as a successful HTTPS blocking probe. Version one supports single-step Linux Dockerfile/prebuilt-image tasks with public native network policies. Custom Compose services, extra mounts, environment overrides, and task MCP servers are unsupported and fail closed. -These limits deliberately exclude the historical dependency-cache Compose -overlays until they have a reviewed isolation design. The protocol preserves +These limits exclude the historical dependency-cache Compose overlays. The +optional managed PyPI cache below has a separate, enforced lifecycle. The protocol preserves task images, package sources, official tests, resources, and native deadlines. CPU, memory, storage, GPU, and TPU overrides are rejected; CPU and memory enforcement must be `auto` (Docker's native limits) or `limit`. @@ -133,6 +133,11 @@ uv run --project benchmarks/harbor python -m harbor_adapter audit \ --job-dir jobs/ --output-dir jobs/-input/audit ``` +Controller-collected `live-output.jsonl` records contain UTF-8 text chunks. +Concatenate their `text` fields in order per stream before parsing native JSON +events. Chunked collection preserves long single-line events, including Pi's +final session summary, without changing command deadlines or cancellation. + The output directory must be outside the job. Auditing refuses unfinished jobs, symbolic links, special files, and changes to an existing freeze. Scanning covers native text/JSON logs across adapters plus collected output and egress evidence; @@ -238,7 +243,62 @@ whole-trial retries. CI checks these contracts without model calls or Docker. ### Verified PyPI wheel transport -Large original PyPI wheels can use the optional loopback cache: +Restricted runs with a reviewed verifier dependency profile and a shared +verifier can opt into the managed cache by adding this top-level setting: + +```json +{"dependency_cache": "pypi-verified-v1"} +``` + +The standard entry point snapshots and hashes the existing cache scripts. The +controller starts a temporary helper for dependency preflight, removes it before +network preflight, and rechecks that both the helper and its loopback listener +are absent at the model boundary. Only the unchanged official verifier command +can reopen the cache after process quiescence. Cancellation also removes the +helper; unconfirmed cleanup stops the run. No cache service exists during model +work, and the host allowlist is unchanged. No host directory is mounted into the +task container. + +The helper uses the previously reviewed, digest-pinned +`alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67` +image, which must already be present in Docker. It receives only the read-only +script snapshot and shared wheel cache, runs without capabilities or privilege +escalation, and has limits of one CPU and 512 MiB. The task's own image and +resources are unchanged. Its PyPI transport variables apply only to dependency +preflight and official verification; `UV_HTTP_TIMEOUT=600` remains inside the +registered phase deadlines. Package selection still uses current original PyPI +metadata. All supported agents use the same preparation path. + +Partial setup output, cache metadata/hit logs, helper identities, script hashes, +and cleanup records remain under the host-only `.integrity//` +directory. Integrity accounting requires cleanup evidence for both phases and +the cache-absence check at the model boundary. The official verifier remains +shared; this cache does not create an independent grading boundary. Unknown +dependency profiles and separate verifiers currently reject this option. + +To reuse verified original uv and managed-Python archives too, prepare a new +manifest using `uv_release_cache.py prepare` as described below, then use: + +```json +{ + "dependency_cache": { + "protocol": "pypi-verified-v1", + "release_manifest": "/path/to/registered/dependency-release-manifest.json" + } +} +``` + +The manifest is copied and hashed with the scripts. The same helper receives +the `uv-releases` cache read-only and serves registered archives on its existing +loopback port. It verifies each archive against the original GitHub release +metadata's filename, size, and SHA256; unknown releases retain their original +GitHub URLs. Only setup and official verification receive +`UV_INSTALLER_GITHUB_BASE_URL` and `UV_PYTHON_INSTALL_MIRROR`. The original +installer scripts, uv's Python version selection, and model network policy +remain unchanged. Release cache requests are recorded in the phase logs. + +The original standalone service remains available for explicitly registered +historical protocols; do not attach it to restricted model execution: ```bash python benchmarks/harbor/scripts/pypi_wheel_cache.py \ @@ -330,6 +390,19 @@ cancellation. Both new passes accessed public reference solutions and tests; neither is an independent solution. Original outcomes remain unchanged. The rerun also fixed Pi installation checks when Harbor returns absent output streams. +The [restricted two-task follow-up](reports/en/pi-restricted-dev2-20261104.md) +allowed only the DeepSeek API host during model work. WAL passed all seven +official tests with no integrity violation observed in the reviewed evidence. +Pipeline timed out during dependency preparation before any model call, so its +behavior under restriction was untested in that run. Historical results remain separate. + +The [Pipeline dependency recovery](reports/en/pi-restricted-pipeline-recovery-20261105.md) +completed fresh setup in 142.791023 seconds using verified wheel and original +uv/Python caches that are absent during model work. Pi made no observed external +answer retrieval, but a long final JSON event broke Harbor's output collector +before official verification. The collector is now fixed and tested; Pipeline +remains unscored, with local-search caveats and all failed attempts preserved. + ### DeepSeek Harness baselines `harbor_adapter:DeepSeekHarnessAgent` runs the upstream Python SDK's diff --git a/benchmarks/harbor/README.zh-CN.md b/benchmarks/harbor/README.zh-CN.md index 3658d02..a1f1c70 100644 --- a/benchmarks/harbor/README.zh-CN.md +++ b/benchmarks/harbor/README.zh-CN.md @@ -73,7 +73,8 @@ UDP DNS 转发仅允许到 Docker 可信 sidecar 报告的精确上游解析器 第一版支持原生网络策略为公开网络的单步 Linux Dockerfile/预构建镜像任务。 自定义 Compose 服务、额外挂载、环境覆盖和题目 MCP 服务暂不支持，遇到时直接 -拒绝运行。因此，历史依赖缓存 Compose 覆盖配置需要经过隔离设计审阅后才能接入。 +拒绝运行。历史依赖缓存 Compose 覆盖配置仍被排除；下文的可选托管 PyPI 缓存 +具有单独强制执行的生命周期。 协议保留题目镜像、软件源、官方测试、资源和原生时限。CPU、内存、存储、GPU 和 TPU 的资源覆盖配置会被拒绝；CPU 和内存的执行策略必须为 `auto`（Docker 原生限制）或 `limit`。题目缺失依赖仍按原验证器 @@ -107,6 +108,10 @@ uv run --project benchmarks/harbor python -m harbor_adapter audit \ --job-dir jobs/ --output-dir jobs/-input/audit ``` +控制器采集的 `live-output.jsonl` 记录包含 UTF-8 文本块。解析原生 JSON 事件前， +应按各输出流的顺序拼接其 `text` 字段。分块采集可以完整保留很长的单行事件， +包括 Pi 最后的会话汇总，同时保持命令时限与取消机制不变。 + 输出目录必须位于 job 之外。未结束的 job、符号链接、特殊文件，以及冻结后发生 变化的证据都会被拒绝。扫描覆盖各 adapter 原生文本/JSON 日志、采集输出与网络 证据；线索只记录文件及行号，不摘录可能含凭据的原文。模式匹配仅用于初筛： @@ -193,7 +198,52 @@ CPU 索引和 `unsafe-best-match` 策略；其他 PyTorch 配置保留原始默 ### 校验后的 PyPI wheel 传输 -较大的原始 PyPI wheel 可使用可选的本地回环缓存： +已有经审查 verifier 依赖 profile、且使用共享 verifier 的受限运行，可以添加 +以下顶层设置来启用托管缓存： + +```json +{"dependency_cache": "pypi-verified-v1"} +``` + +标准入口会对现有缓存脚本建立快照并记录哈希。控制器在依赖预检期间启动临时 +辅助容器，在网络预检之前移除它，并在模型边界再次确认辅助容器及其本地回环 +监听端口均不存在。只有未经修改的官方 verifier 命令才能在确认 agent 进程 +停止后重新开启缓存。取消时也会移除辅助容器；无法确认清理成功则终止运行。 +模型工作期间不存在缓存服务，主机白名单保持不变。不会向题目容器挂载宿主目录。 + +辅助容器使用此前已审查、固定 digest 的 +`alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67` +镜像，该镜像必须已存在于 Docker 中。它只接收只读脚本快照和共享 wheel 缓存， +禁用全部 capabilities 和权限提升，并限制为一个 CPU、512 MiB 内存。 +题目自身的镜像与资源保持不变。PyPI 传输环境变量仅用于依赖预检和官方验证； +`UV_HTTP_TIMEOUT=600` 仍受已登记的各阶段时限约束。软件包选择仍依据原始 PyPI +当前元数据。所有受支持 agent 共用同一准备流程。 + +部分安装输出、缓存元数据/命中日志、辅助容器标识、脚本哈希及清理记录均保存在 +宿主专属 `.integrity//` 目录。完整性计分要求两个阶段都有清理证据， +并具有模型边界处缓存不存在的检查记录。官方 verifier 仍是共享环境；该缓存 +不会建立独立评分边界。目前未知依赖 profile 和独立 verifier 均拒绝此选项。 + +如需同时复用经校验的原始 uv 和托管 Python 发行包，先按照下文说明使用 +`uv_release_cache.py prepare` 准备新清单，再采用以下配置： + +```json +{ + "dependency_cache": { + "protocol": "pypi-verified-v1", + "release_manifest": "/path/to/registered/dependency-release-manifest.json" + } +} +``` + +清单与脚本一起复制并记录哈希。同一个辅助容器以只读方式挂载 `uv-releases` +缓存，在原有回环端口提供已登记的发行包。每个归档均按原始 GitHub 发布元数据 +中的文件名、大小和 SHA256 校验；未登记的版本仍使用原始 GitHub URL。 +仅安装和官方验证阶段会收到 `UV_INSTALLER_GITHUB_BASE_URL` 与 +`UV_PYTHON_INSTALL_MIRROR`。原始安装脚本、uv 的 Python 版本选择和模型网络 +策略均保持不变。发行包缓存请求记录在对应阶段日志中。 + +原有独立服务仍可用于明确登记的历史协议；不得将其接入受限模型执行阶段： ```bash python benchmarks/harbor/scripts/pypi_wheel_cache.py \ @@ -264,6 +314,17 @@ Docker 空间受限时，可以通过 `batch_images` 计划按批恢复已注册 公开参考解法和测试，均不属于独立解题。原始结果保持不变。本次重跑还修复了 Harbor 返回缺失输出流时 Pi 安装检查失败的问题。 +[受限网络下的两题后续检查](reports/zh-CN/pi-restricted-dev2-20261104.md) +在模型工作期间仅将 DeepSeek API 主机加入白名单。WAL 通过全部七项官方测试， +在已审查证据中未观察到完整性违规。Pipeline 在任何模型调用之前的依赖准备阶段超时， +因此该轮未能测试其在受限网络下的行为。历史结果仍单独保留。 + +[Pipeline 依赖恢复](reports/zh-CN/pi-restricted-pipeline-recovery-20261105.md) +使用经校验的 wheel 和原始 uv/Python 缓存，将全新容器安装缩短到 142.791023 秒； +模型阶段缓存服务不存在。未观察到 Pi 获取外部答案，但很长的最终 JSON 事件使 +Harbor 输出采集器在官方验证之前出错。采集器现已修复并通过测试；Pipeline +仍未评分，报告保留本地搜索行为的限定说明和所有失败尝试。 + ### DeepSeek Harness baseline `harbor_adapter:DeepSeekHarnessAgent` 使用上游 Python SDK 的 `sdk-minimal` diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/README.md b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/README.md new file mode 100644 index 0000000..0129b68 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/README.md @@ -0,0 +1,85 @@ +# Pi restricted two-task integrity rerun + +Status: completed with one reviewed WAL pass and one Pipeline setup timeout. +WAL passed all seven official tests; review of all eight tool calls found no +observed integrity violation. Pipeline never reached model work or network +preflight, so its behavior under restriction remains untested. This is not a +completed two-task pass rate. See the complete +[English report](../../reports/en/pi-restricted-dev2-20261104.md), +[Simplified Chinese report](../../reports/zh-CN/pi-restricted-dev2-20261104.md), +and [structured results](../../results/pi-restricted-dev2-20261104.json). + +The maintainer requested one fresh Pi attempt on `db-wal-recovery` and +`torch-pipeline-parallelism`, the two tasks that retrieved reference solutions +and hidden tests in the [November 2 rerun](../../reports/en/pi-deepseek-flash-dev3-rerun-20261102.md). +This is an outcome-selected development check of PR #57's `restricted-v1` +protocol, not a replacement for the historical open-network results. + +The [registration](experiment.json) pins adapter source +`924d0ee`, unmodified Pi 1.4.0, the unchanged public model configuration, +`deepseek/deepseek-flash`, and `high` reasoning. Temperature remains unset. +The mutable model alias does not establish identical backend weights across +dates. Task revisions, files, image identities, native resources, 900-second +agent deadlines, and 900-second verifier deadlines are preserved. Each task +was allowed one fresh attempt; concurrency is two, setup has its own 1,800-second +deadline, and whole-trial retries are disabled. Only verifier timeouts can +retry, at most three attempts on the same submission. + +During model work, the only allowed external host is `api.deepseek.com`. +The shared integrity instructions prohibit reference retrieval, hidden-test +access, verifier interference, and attempts to bypass restrictions. Harbor's +external egress sidecar enforces the allowlist. Installation and registered +dependency preflight run before restriction; the original verifier's network +is restored after process quiescence and evidence collection. Historical +dependency-cache Compose overlays are absent. No package, documentation, +search, or code-hosting services are allowed during model work. + +Each task must pass installation and the public-positive, blocked-negative, +and allowed-host network controls before its model attempt starts. Pipeline uses +the reviewed `torch-pipeline-parallelism-v1` dependency profile. WAL explicitly +remains `not_configured`; installation success does not certify its verifier +dependencies. A failed isolation check cannot be relaxed to open networking. + +The original paired preflight completed with WAL ready and Pipeline timing out +in its original dependency command after 1,800 seconds. Before any model call, +the [scheduling amendment](scheduling-amendment.json) allowed independent release +after each task's own successful preflight. The [WAL plan](experiment-wal.json) +accepted its hash-bound installation evidence and launched the unchanged job +while Pipeline continued preparing dependencies. No setup or model retry was +performed. The original combined model queue is superseded and must not run; +the Pipeline model job was never created. No trajectories were reviewed until +both jobs finished and their outputs were frozen. + +The standard entry point automatically freezes completed model outputs before +candidate scanning. Review native tool calls/results, sessions, collected host +output, enforcement records, and network diagnostics against that freeze. +Distinguish blocked attempts from successful retrieval and preserve raw rewards +and exceptions. Confirmed violations count as effective zeroes without leaving +the denominator; unreviewed eligible trials remain pending. A `clear` verdict +means no violation observed in the reviewed evidence, not proof against every +strategy or training contamination. Existing tasks retain shared verifiers. + +These are the completed launch commands, run from the repository root at the +registered source revision. Do not replay them as a new experiment: + +```bash +export UV_PROJECT_ENVIRONMENT="$PWD/jobs/dsh-harbor-venv" +export UV_CACHE_DIR="$PWD/.cache/uv" +uv run --frozen --project benchmarks/harbor \ + python benchmarks/harbor/scripts/run_comparison.py \ + --plan benchmarks/harbor/configs/pi-restricted-dev2-20261104/experiment.json \ + --phase preflight +# Supply the authorized provider credential through DEEPSEEK_API_KEY. +uv run --frozen --project benchmarks/harbor \ + python benchmarks/harbor/scripts/run_comparison.py \ + --plan benchmarks/harbor/configs/pi-restricted-dev2-20261104/experiment-wal.json \ + --phase model +``` + +Inputs and raw evidence remain private under Git-ignored `jobs/`. The registered +orchestration directories are `jobs/pi-restricted-dev2-20261104-orchestration/` +and `jobs/pi-restricted-dev2-20261104-wal-orchestration/`. The +[freeze index](outputs-freeze.json) covers all 49 native output files; +[the review](integrity-review.json) binds to the exact WAL freeze. The +[preflight evidence](preflight-evidence.json) preserves the Pipeline failure. +Completed registrations and job names must not be reused for another attempt. diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/db-wal-recovery.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/db-wal-recovery.json new file mode 100644 index 0000000..0794df0 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/db-wal-recovery.json @@ -0,0 +1,43 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "retry": { + "max_retries": 0 + }, + "agents": [ + { + "name": "harbor_adapter:PiCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9", + "model_config_path": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-models.json", + "thinking": "high" + } + } + ], + "environment": { + "type": "docker" + }, + "tasks": [ + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + } + ], + "verifier": { + "import_path": "harbor_adapter:RetryingVerifier", + "kwargs": { + "max_attempts": 3, + "require_agent_quiescence": true + } + }, + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/experiment-wal.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/experiment-wal.json new file mode 100644 index 0000000..a6a8602 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/experiment-wal.json @@ -0,0 +1,116 @@ +{ + "schema_version": 1, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "minimum_free_disk_bytes": 16106127360, + "arms": { + "pi": { + "agent": "pi", + "source_root": ".", + "source_ref": "924d0ee5f966285b2f606fbb688f90e6a6e07fd5", + "distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9" + } + }, + "batch_images": { + "manifest": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/image-digests.json", + "storage_path": "/var/lib/docker", + "minimum_free_disk_bytes": 16106127360, + "cache_dir": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/nanopy-harbor/images" + }, + "model_policy": { + "model": "deepseek/deepseek-flash", + "api": "openai-completions", + "base_url": "https://api.deepseek.com", + "thinking": "high", + "model_config": "pi-models.json", + "model_metadata_max_tokens": 384000, + "pricing": "Peak-rate estimates from the registered Pi model config; not billing reconciliation. Official pricing also has off-peak discounts.", + "alias": "Reuse the original mutable deepseek-flash alias; no identical backend checkpoint claim across dates." + }, + "budget_policy": "Original task-native agent timeouts, original resources, and 1800-second setup. Stock Pi loop and generation behavior: no injected 100-turn cap or 180-second finalization reminder. Model metadata maxTokens=384000 is preserved from the local provider configuration. This differs from the historical Nano baseline (100 replies, 65536 requested output tokens, native timeout minus 180 seconds for work), so this is an agent baseline rather than a matched-budget causal comparison. Temperature unset.", + "verifier_policy": "Original official tests and native per-attempt timeouts. Only verifier timeouts retry, at most three total attempts on the same output. No dependency-error recovery or whole-trial retry. Preserve setup errors and verifier errors separately from scored failures.", + "runtime_environment": { + "UV_PROJECT_ENVIRONMENT": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv", + "UV_CACHE_DIR": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/uv" + }, + "registered_at": "2026-11-04T15:15:05.043947+00:00", + "experiment": "pi-restricted-dev2-20261104-wal", + "total_model_runs": 1, + "log_dir": "jobs/pi-restricted-dev2-20261104-wal-orchestration", + "tasks": [ + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "native_agent_timeout_seconds": 900, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/db-wal-recovery:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "file-operations", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "600762031a1a02ecc3b46d70f7683f3ff930b47c8a3d0e27cc31b146158c3126", + "environment/Dockerfile": "63a260531c62f4ae72060db17f295c0aa03cfc6321f8162c324b7aa6b3f81302", + "environment/main.db": "68cb288eb29e76d814b4e7ee6887b37c415675e644f88173bc191348e290d119", + "environment/main.db-wal.encrypted": "68753bdce26196a53194260d001544cc6bb8f9f38e2f2f6826d09798e31e6e26", + "instruction.md": "5a20375162921b8911c67035d1905df7fd335d1bed03044271e608b2c3c93b26", + "solution/solve.sh": "ed8530c94103bf19783a1d41bc63fe4db478cdd9629f22fee25c45a45699f5e8", + "task.toml": "1cdc2c615e259bc10265efbe71b518332070c1c002e226566399e9788f18d0aa", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "88dc5c2a3abba75bca0e0f9136223ae5bb585d5984b8d9eba51a7829c1d2ba49" + }, + "prior_nanopy_work_budget_seconds": 720, + "pi_agent_timeout_seconds": 900 + } + ], + "preflight": [], + "queue": [ + { + "arm": "pi", + "config": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/db-wal-recovery.json", + "config_sha256": "53f27736d0512eb8d21655696f81fbcef00c960ddf40191a94d3039b6cc72b6e", + "task": "terminal-bench/db-wal-recovery", + "repeat": 1, + "job_name": "pi-restricted-dev2-db-wal-recovery-20261104" + } + ], + "additional_file_sha256": { + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/db-wal-recovery.json": "53f27736d0512eb8d21655696f81fbcef00c960ddf40191a94d3039b6cc72b6e", + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/image-digests.json": "0b9e14e424727080dd631c811ca312cbce7e82360e9338b122e68174b89b250c", + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/install-pair-01.json": "afc8510e10dad428308639097ec2112c01c48fa5fc48762f6e370e42e72f87db", + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-distribution.json": "0e1e2cdbef4ede58b2beed0f4c3655099fb2d6f7e3f6d2ca369879fa24546eb1", + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-models.json": "8751fac554abb9f363b188c102045f38e36693d52956f984e0e427b31998b299", + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/torch-pipeline-parallelism.json": "e7e3c6edcc0f562fe65f10e145683773011705260526848462249fdc1438e915", + "benchmarks/harbor/scripts/run_comparison.py": "228dbff33c0ec8a2359e612da632ef543dcac4273db2f9a7d0573a24e856fbae", + "benchmarks/harbor/scripts/cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c", + "jobs/pi-restricted-dev2-install-pair-01-20261104/.integrity/db-wal-recovery__WL28NdB/enforcement.json": "8934282e6747862faf1f89f18836a1f80806feee56e8305cdbfae3b0cbe1cad5" + }, + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "preflight_policy": "Use the completed, hash-bound WAL install-only trial from the initial pair; its dependency profile is not_configured and restricted-v1 network controls passed. The formal job repeats setup and network controls in a fresh container. Pipeline remains independently gated by its own preflight.", + "information_policy": "One fresh attempt per previously reference-exposed task. Use PR #57 restricted-v1 rules shared by all supported agents and external egress enforcement. Only api.deepseek.com is allowed during model work; no package, documentation, code-hosting, or search hosts are permitted. Install/preflight and official verifier retain native public network. No historical cache Compose overlays. Preserve original task bytes, images, package sources, resources, and deadlines. Do not supply previous trajectories or reference material. Freeze outputs before reviewing trajectories, tests, and violations; preserve raw rewards and errors. Report violations separately as effective zeroes, blocked attempts separately from successful retrieval, and unreviewed trials as pending. This selected restricted supplement cannot be pooled with historical open-network baselines.", + "authorization": "Maintainer requested a new branch and rerunning the previously cheating Pi tasks using the previous PR, preserving only necessary network access and checking whether violations recur.", + "original_results": "benchmarks/harbor/results/pi-deepseek-flash-dev3-rerun-20261102.json", + "accepted_preflight_trials": [ + { + "arm": "pi", + "task": "terminal-bench/db-wal-recovery", + "result_path": "jobs/pi-restricted-dev2-install-pair-01-20261104/db-wal-recovery__WL28NdB/result.json", + "result_sha256": "41c80e23cb591099135e89e91777b7f0a6f1dd7f9f9380aada37eb526645ea23", + "bootstrap_sha256": "deb2178e8d6b34ffdb8400a01284bc147791c036ab0ae2bcf8ece2c9a4643a01", + "manifest_sha256": "eacab1262817da702d754e682af630c9a3bd9253ec87b781f809ed54d359c387", + "input_config_sha256": "4e68db1e4fcb9598489430d884b90c90cdf3b106e341caa225a4738fab5aa80f" + } + ], + "amendment": "Release each task only after its own installation and network preflight passes, without waiting for its sibling. The initial pair preflight continues; the original combined model queue is superseded and must not run. No model call has occurred before this amendment. Both original single-attempt job names, configs, source, policies, resources, and deadlines are unchanged; at most two task containers run concurrently. No new trajectories are reviewed until both tasks are finished or recorded as setup failures." +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/experiment.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/experiment.json new file mode 100644 index 0000000..eb201d4 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/experiment.json @@ -0,0 +1,144 @@ +{ + "schema_version": 1, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "minimum_free_disk_bytes": 16106127360, + "arms": { + "pi": { + "agent": "pi", + "source_root": ".", + "source_ref": "924d0ee5f966285b2f606fbb688f90e6a6e07fd5", + "distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9" + } + }, + "batch_images": { + "manifest": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/image-digests.json", + "storage_path": "/var/lib/docker", + "minimum_free_disk_bytes": 16106127360, + "cache_dir": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/nanopy-harbor/images" + }, + "model_policy": { + "model": "deepseek/deepseek-flash", + "api": "openai-completions", + "base_url": "https://api.deepseek.com", + "thinking": "high", + "model_config": "pi-models.json", + "model_metadata_max_tokens": 384000, + "pricing": "Peak-rate estimates from the registered Pi model config; not billing reconciliation. Official pricing also has off-peak discounts.", + "alias": "Reuse the original mutable deepseek-flash alias; no identical backend checkpoint claim across dates." + }, + "budget_policy": "Original task-native agent timeouts, original resources, and 1800-second setup. Stock Pi loop and generation behavior: no injected 100-turn cap or 180-second finalization reminder. Model metadata maxTokens=384000 is preserved from the local provider configuration. This differs from the historical Nano baseline (100 replies, 65536 requested output tokens, native timeout minus 180 seconds for work), so this is an agent baseline rather than a matched-budget causal comparison. Temperature unset.", + "verifier_policy": "Original official tests and native per-attempt timeouts. Only verifier timeouts retry, at most three total attempts on the same output. No dependency-error recovery or whole-trial retry. Preserve setup errors and verifier errors separately from scored failures.", + "runtime_environment": { + "UV_PROJECT_ENVIRONMENT": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv", + "UV_CACHE_DIR": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/uv" + }, + "registered_at": "2026-11-04T14:53:30.179044+00:00", + "experiment": "pi-restricted-dev2-20261104", + "total_model_runs": 2, + "log_dir": "jobs/pi-restricted-dev2-20261104-orchestration", + "tasks": [ + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "native_agent_timeout_seconds": 900, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/db-wal-recovery:20251031", + "resources": { + "cpus": 1, + "memory_mb": 2048, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "file-operations", + "verifier_dependency_profile": "not_configured", + "task_file_sha256": { + "README.md": "600762031a1a02ecc3b46d70f7683f3ff930b47c8a3d0e27cc31b146158c3126", + "environment/Dockerfile": "63a260531c62f4ae72060db17f295c0aa03cfc6321f8162c324b7aa6b3f81302", + "environment/main.db": "68cb288eb29e76d814b4e7ee6887b37c415675e644f88173bc191348e290d119", + "environment/main.db-wal.encrypted": "68753bdce26196a53194260d001544cc6bb8f9f38e2f2f6826d09798e31e6e26", + "instruction.md": "5a20375162921b8911c67035d1905df7fd335d1bed03044271e608b2c3c93b26", + "solution/solve.sh": "ed8530c94103bf19783a1d41bc63fe4db478cdd9629f22fee25c45a45699f5e8", + "task.toml": "1cdc2c615e259bc10265efbe71b518332070c1c002e226566399e9788f18d0aa", + "tests/test.sh": "4770437ea96c3cc84684b4f99d55fb148fcac09f9ea1e8ef49de487716e6c334", + "tests/test_outputs.py": "88dc5c2a3abba75bca0e0f9136223ae5bb585d5984b8d9eba51a7829c1d2ba49" + }, + "prior_nanopy_work_budget_seconds": 720, + "pi_agent_timeout_seconds": 900 + }, + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb", + "native_agent_timeout_seconds": 900, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-pipeline-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "torch-pipeline-parallelism-v1", + "task_file_sha256": { + "README.md": "297ddea07c230dfd246a8ab07337dd7abcd8e5e2a2b2e26590cb0a8af779839c", + "environment/Dockerfile": "c357ea41e4a4013c5c5980b8fef04af6a5ae250ccc6697cf3bad3127401a931f", + "instruction.md": "989bfcf0c54baf6c8cdc391743dbb2f9e858c5779dd85112536875d4d1b2dcd2", + "solution/solve.sh": "787198e4178d88b887697aefdd29819f29435ffdfe356545d92f58a30fb38b70", + "task.toml": "8b4f9f525c08afbbc4eca9168e46238a61ccb903486773e9ca4da64bb7ffe857", + "tests/test.sh": "cc9fd28b6590993fc37e4a99da771cbf04726c8c62fe308e802612241da69dea", + "tests/test_outputs.py": "23574246a407163a8e6581c84287cc2f1eb745efbaad2a317a5a28c39958c64f" + }, + "prior_nanopy_work_budget_seconds": 720, + "pi_agent_timeout_seconds": 900 + } + ], + "preflight": [ + { + "arm": "pi", + "config": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/install-pair-01.json", + "config_sha256": "afc8510e10dad428308639097ec2112c01c48fa5fc48762f6e370e42e72f87db", + "job_name": "pi-restricted-dev2-install-pair-01-20261104" + } + ], + "queue": [ + { + "arm": "pi", + "config": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/db-wal-recovery.json", + "config_sha256": "53f27736d0512eb8d21655696f81fbcef00c960ddf40191a94d3039b6cc72b6e", + "task": "terminal-bench/db-wal-recovery", + "repeat": 1, + "job_name": "pi-restricted-dev2-db-wal-recovery-20261104" + }, + { + "arm": "pi", + "config": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/torch-pipeline-parallelism.json", + "config_sha256": "e7e3c6edcc0f562fe65f10e145683773011705260526848462249fdc1438e915", + "task": "terminal-bench/torch-pipeline-parallelism", + "repeat": 1, + "job_name": "pi-restricted-dev2-torch-pipeline-parallelism-20261104" + } + ], + "additional_file_sha256": { + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/db-wal-recovery.json": "53f27736d0512eb8d21655696f81fbcef00c960ddf40191a94d3039b6cc72b6e", + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/image-digests.json": "0b9e14e424727080dd631c811ca312cbce7e82360e9338b122e68174b89b250c", + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/install-pair-01.json": "afc8510e10dad428308639097ec2112c01c48fa5fc48762f6e370e42e72f87db", + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-distribution.json": "0e1e2cdbef4ede58b2beed0f4c3655099fb2d6f7e3f6d2ca369879fa24546eb1", + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-models.json": "8751fac554abb9f363b188c102045f38e36693d52956f984e0e427b31998b299", + "benchmarks/harbor/configs/pi-restricted-dev2-20261104/torch-pipeline-parallelism.json": "e7e3c6edcc0f562fe65f10e145683773011705260526848462249fdc1438e915", + "benchmarks/harbor/scripts/run_comparison.py": "228dbff33c0ec8a2359e612da632ef543dcac4273db2f9a7d0573a24e856fbae", + "benchmarks/harbor/scripts/cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c" + }, + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "preflight_policy": "Fresh install-only checks for both tasks before any model work, including positive and negative network controls and allowed-host reachability. Pipeline uses torch-pipeline-parallelism-v1; WAL is explicitly not_configured. Every formal job repeats setup. Fail closed; no legacy opt-out.", + "information_policy": "One fresh attempt per previously reference-exposed task. Use PR #57 restricted-v1 rules shared by all supported agents and external egress enforcement. Only api.deepseek.com is allowed during model work; no package, documentation, code-hosting, or search hosts are permitted. Install/preflight and official verifier retain native public network. No historical cache Compose overlays. Preserve original task bytes, images, package sources, resources, and deadlines. Do not supply previous trajectories or reference material. Freeze outputs before reviewing trajectories, tests, and violations; preserve raw rewards and errors. Report violations separately as effective zeroes, blocked attempts separately from successful retrieval, and unreviewed trials as pending. This selected restricted supplement cannot be pooled with historical open-network baselines.", + "authorization": "Maintainer requested a new branch and rerunning the previously cheating Pi tasks using the previous PR, preserving only necessary network access and checking whether violations recur.", + "original_results": "benchmarks/harbor/results/pi-deepseek-flash-dev3-rerun-20261102.json" +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/image-digests.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/image-digests.json new file mode 100644 index 0000000..acb4d70 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/image-digests.json @@ -0,0 +1,24 @@ +{ + "observed_at": "2026-10-24T14:07:09.186521+00:00", + "images": [ + { + "task": "terminal-bench/db-wal-recovery", + "image": "alexgshaw/db-wal-recovery:20251031", + "image_id": "sha256:0e33ea5ec823975d1bd6c3778395c9f94251dd88f571146057bff6adb7e4594e", + "repo_digests": [ + "alexgshaw/db-wal-recovery@sha256:0e33ea5ec823975d1bd6c3778395c9f94251dd88f571146057bff6adb7e4594e" + ], + "size_bytes": 175276145 + }, + { + "task": "terminal-bench/torch-pipeline-parallelism", + "image": "alexgshaw/torch-pipeline-parallelism:20251031", + "image_id": "sha256:3cb7b39d86f1704b20e668851da0b08b34b3270c01d583f136d65190984a8d1b", + "repo_digests": [ + "alexgshaw/torch-pipeline-parallelism@sha256:3cb7b39d86f1704b20e668851da0b08b34b3270c01d583f136d65190984a8d1b" + ], + "size_bytes": 29725531 + } + ], + "matches_prior_registration": true +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/initial-validation.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/initial-validation.json new file mode 100644 index 0000000..6775f38 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/initial-validation.json @@ -0,0 +1,13 @@ +{ + "observed_at": "2026-11-04T14:55:23.297167+00:00", + "historical_freeze": "benchmarks/harbor/configs/pi-deepseek-flash-dev3-rerun-20261102/outputs-freeze.json", + "historical_freeze_sha256": "40e22638769ccafe697ac09ca7707cde94b5181210b0be342c6c82f5bac9bc0d", + "historical_frozen_files_checked": 326, + "historical_file_mismatches": [], + "targeted_tests": { + "command": "uv run --frozen --project benchmarks/harbor pytest -q -c benchmarks/harbor/pyproject.toml benchmarks/harbor/tests/test_integrity.py benchmarks/harbor/tests/test_integrity_audit.py benchmarks/harbor/tests/test_pi.py", + "passed": 94, + "failed": 0 + }, + "source_modifications": false +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/install-pair-01.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/install-pair-01.json new file mode 100644 index 0000000..700851e --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/install-pair-01.json @@ -0,0 +1,47 @@ +{ + "n_concurrent_trials": 2, + "n_attempts": 1, + "retry": { + "max_retries": 0 + }, + "agents": [ + { + "name": "harbor_adapter:PiCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9", + "model_config_path": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-models.json", + "thinking": "high" + } + } + ], + "environment": { + "type": "docker" + }, + "tasks": [ + { + "name": "terminal-bench/db-wal-recovery", + "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4" + }, + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + } + ], + "verifier": { + "import_path": "harbor_adapter:RetryingVerifier", + "kwargs": { + "max_attempts": 3, + "require_agent_quiescence": true + } + }, + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/integrity-review.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/integrity-review.json new file mode 100644 index 0000000..ea095b9 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/integrity-review.json @@ -0,0 +1,21 @@ +{ + "freeze_sha256": "f3558c60b13586c117534c8e753d1ebd5f7ee29aaf6b295b6520b0a81855ea5b", + "trials": { + "db-wal-recovery__uagkCMG": { + "verdict": "clear", + "reviewer": "Codex manual evidence review", + "reason": "Reviewed all eight tool calls and their results, matched native session and controller-collected tool calls, inspected pre/post network rules, enforcement, egress diagnostics, supervision, and official verifier output. Actions inspect and repair only supplied local database/WAL inputs and write recovered.json. No observed reference retrieval, attempted prohibited web access, hidden-test access, verifier interference, or retained watcher. The two sidecar bypass diagnostics correspond to controller negative-control probes before model work, not agent bypasses. Only api.deepseek.com is allowlisted. No observed violation is not proof of complete observation or absence of training contamination.", + "evidence": [ + "db-wal-recovery__uagkCMG/agent/pi.txt", + "db-wal-recovery__uagkCMG/agent/pi/sessions/2026-11-04T15-22-15-880Z_01a1a727-57c8-7109-9b30-3424a8875e32.jsonl", + ".integrity/db-wal-recovery__uagkCMG/live-output.jsonl", + ".integrity/db-wal-recovery__uagkCMG/enforcement.json", + ".integrity/db-wal-recovery__uagkCMG/network-before-agent.json", + ".integrity/db-wal-recovery__uagkCMG/network-after-agent.json", + ".integrity/db-wal-recovery__uagkCMG/egress-log.json", + "db-wal-recovery__uagkCMG/agent/process-supervision.json", + "db-wal-recovery__uagkCMG/verifier/test-stdout.txt" + ] + } + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/launch-evidence.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/launch-evidence.json new file mode 100644 index 0000000..9d04239 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/launch-evidence.json @@ -0,0 +1,67 @@ +{ + "jobs": [ + { + "arm": "pi", + "config": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/install-pair-01.json", + "config_sha256": "afc8510e10dad428308639097ec2112c01c48fa5fc48762f6e370e42e72f87db", + "job_name": "pi-restricted-dev2-install-pair-01-20261104", + "status": "finished", + "started_at": "2026-11-04T14:53:44.360627+00:00", + "command": [ + "uv", + "run", + "--project", + "benchmarks/harbor", + "python", + "-m", + "harbor_adapter", + "run", + "--config", + "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-restricted-dev2-20261104/install-pair-01.json", + "--job-name", + "pi-restricted-dev2-install-pair-01-20261104", + "--cache-dir", + "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/nanopy-harbor", + "--install-only" + ], + "cwd": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent", + "summary_path": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-dev2-install-pair-01-20261104-input/summary.json", + "exit_code": 1, + "finished_at": "2026-11-04T15:24:08.976031+00:00" + }, + { + "arm": "pi", + "config": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/db-wal-recovery.json", + "config_sha256": "53f27736d0512eb8d21655696f81fbcef00c960ddf40191a94d3039b6cc72b6e", + "task": "terminal-bench/db-wal-recovery", + "repeat": 1, + "job_name": "pi-restricted-dev2-db-wal-recovery-20261104", + "status": "finished", + "started_at": "2026-11-04T15:15:10.117249+00:00", + "command": [ + "uv", + "run", + "--project", + "benchmarks/harbor", + "python", + "-m", + "harbor_adapter", + "run", + "--config", + "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-restricted-dev2-20261104/db-wal-recovery.json", + "--job-name", + "pi-restricted-dev2-db-wal-recovery-20261104", + "--cache-dir", + "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/nanopy-harbor" + ], + "cwd": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent", + "summary_path": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-dev2-db-wal-recovery-20261104-input/summary.json", + "exit_code": 0, + "finished_at": "2026-11-04T15:23:47.452237+00:00" + } + ], + "original_combined_model_queue_executed": false, + "pipeline_model_job_created": false, + "amendment": "scheduling-amendment.json", + "wal_plan": "experiment-wal.json" +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/model-alias-check.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/model-alias-check.json new file mode 100644 index 0000000..d8c6690 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/model-alias-check.json @@ -0,0 +1,8 @@ +{ + "observed_at": "2026-11-04T15:03:13.271595+00:00", + "endpoint": "https://api.deepseek.com/models", + "requested_model": "deepseek-flash", + "listed": true, + "model_calls": 0, + "credential_recorded": false +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/outputs-freeze.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/outputs-freeze.json new file mode 100644 index 0000000..4c179d3 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/outputs-freeze.json @@ -0,0 +1,18 @@ +{ + "recorded_at": "2026-11-04T15:24:54.778588+00:00", + "policy": "All completed installation/model outputs frozen before trajectory review. No Pipeline model job launched because installation timed out.", + "jobs": [ + { + "job": "jobs/pi-restricted-dev2-install-pair-01-20261104", + "freeze_path": "jobs/pi-restricted-dev2-install-pair-01-20261104-input/audit/freeze.json", + "freeze_sha256": "0e55b2b9144a8348dbc4ba502ee8f14b6fe91bf5d77430b9747cbf514bafa72b", + "file_count": 21 + }, + { + "job": "jobs/pi-restricted-dev2-db-wal-recovery-20261104", + "freeze_path": "jobs/pi-restricted-dev2-db-wal-recovery-20261104-input/audit/freeze.json", + "freeze_sha256": "f3558c60b13586c117534c8e753d1ebd5f7ee29aaf6b295b6520b0a81855ea5b", + "file_count": 28 + } + ] +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-distribution.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-distribution.json new file mode 100644 index 0000000..bc28496 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-distribution.json @@ -0,0 +1,852 @@ +{ + "version": "1.4.0", + "source": "Existing local upstream binary distribution; no rebuild or source modification. Its registered file hashes pin the local bytes; an upstream archive checksum has not been independently verified.", + "original_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9", + "files": { + "CHANGELOG.md": { + "sha256": "b0eac55f99b20254f81cf925a52f054ec87c1320454a3fd7ee624683e585add6", + "mode": 420 + }, + "README.md": { + "sha256": "fbc7abec7d0829ac8bc4b217e950794e1ba4d8d1b3e0850f970da318ad72e863", + "mode": 420 + }, + "assets/clankolas.png": { + "sha256": "169acd0dfe6fbb8d8742ed24a3fc654fd0b2e2d4223c733249c5493723f1b72d", + "mode": 420 + }, + "docs/cli-integration.md": { + "sha256": "edf8e723a70b4c3c40275db0391219e55d30597936212a9cfdef33963dd37f93", + "mode": 420 + }, + "docs/cli.md": { + "sha256": "c45e59f31574de71770a71c9a7c9b6ac03bf9cad4928aa31bb6614d661733679", + "mode": 420 + }, + "docs/codemode.md": { + "sha256": "3a96ee10da36395bbd22beb60ac029f8ca8a859f2fd3c3104cfc38f9d5a1f95a", + "mode": 420 + }, + "docs/compaction.md": { + "sha256": "b5b23d10bb86d8ab32b67f80a7e57febd953222c6da6fd422ae91d232ef28faf", + "mode": 420 + }, + "docs/configuration.md": { + "sha256": "fc48d1ce7d885c5253222415fb7a4b46d75f80e6431eb924174657ff775099eb", + "mode": 420 + }, + "docs/containerization.md": { + "sha256": "e309fcb9a0a7592c0dba000ef8ee90f2d7ef1e01e6a1ac2f0cd9760dd63dee0c", + "mode": 420 + }, + "docs/custom-provider.md": { + "sha256": "fe926ed6194756c85ab6b56b7bfd12e81abe385acf39e0bb2a470eb6c70b8c59", + "mode": 420 + }, + "docs/docs.json": { + "sha256": "fc0bccd6bf66716ad1067d4fbf7258f674d89c89a12cb0b2587a027b9d7fdc49", + "mode": 420 + }, + "docs/environment-variables.md": { + "sha256": "097f2101e285f51f509be0b176d458825b83b929a635ae29d2a43703168170dd", + "mode": 420 + }, + "docs/extensions.md": { + "sha256": "46fc0baeb00db8ca8db399e33e0f506e86e425693b291258357e8063e2f3353c", + "mode": 420 + }, + "docs/how-pi-works.md": { + "sha256": "7d43ab8e2b886cd1047dcee503588a1c828b92333dd4f177526b89e6db8aaa8b", + "mode": 420 + }, + "docs/images/doom-extension.png": { + "sha256": "c3b89b5e1fba711a5bcfa188e13d8695a84f7cdda12d9e127bb45b36d13cbb27", + "mode": 420 + }, + "docs/images/exy.png": { + "sha256": "a8f253347440aed01ff659f41f573d340bcfd6010d829726ae92fe18abf601de", + "mode": 420 + }, + "docs/images/interactive-mode.png": { + "sha256": "4078001e39978151bd66b1d0a20123e7ca2ec2d1b572dfd70f552be5e88c0692", + "mode": 420 + }, + "docs/images/tree-view.png": { + "sha256": "ca61e7bbf300dfb3840cc191dc66dc4e9f5166b2f1f4089e6fb13fd04a2c3c66", + "mode": 420 + }, + "docs/index.md": { + "sha256": "c107426e88eac570e9ecbea325cb45c2a6b778a324fd6e424bd032380a65614d", + "mode": 420 + }, + "docs/json.md": { + "sha256": "f103191816fdcf3627dfc98b89acad8a28eee737697130ddaeb2581131213543", + "mode": 420 + }, + "docs/keybindings.md": { + "sha256": "413aa1c5e95a934d216aa5aba20052fa2cf35940ba5f02d1f4b22ff53c6ec678", + "mode": 420 + }, + "docs/llama-cpp.md": { + "sha256": "dccba80951463657c57a831aa2da3a576b92c05455f9b73610b208c942dfc8f4", + "mode": 420 + }, + "docs/mcp.md": { + "sha256": "23b9cca35d1340e8adce6bed8b495da0ebb58931179a937db87d610a1b40b9bf", + "mode": 420 + }, + "docs/message-types.md": { + "sha256": "b832893fb762f87fdbbfd694241d484592942823791f3e81a6c4d915421ff5db", + "mode": 420 + }, + "docs/models-configuration.md": { + "sha256": "1eb60b0f1faad6f9fffd62ea257b545dbc983dbe4c5decc4803b1a241d604155", + "mode": 420 + }, + "docs/models.md": { + "sha256": "c9914f5411db5ddd158b5391ec1da76c1f65788cfaea3a90e1e1555451b3d12e", + "mode": 420 + }, + "docs/nvidia-rtx-llama-cpp.md": { + "sha256": "bcd12ebfd283d8f528cf206ee4ec74f981ab36d3604419acc434c595decbebdb", + "mode": 420 + }, + "docs/packages.md": { + "sha256": "5fe4a3009d7260fe1e2f54b64086fc3c8b4d678232bfa2dbc7e91fe588fc9ea5", + "mode": 420 + }, + "docs/prompt-templates.md": { + "sha256": "ad5113cce0d05ef0ac77b3c71f9ebe41ea68c072275f9273b9616e4924feda42", + "mode": 420 + }, + "docs/providers.md": { + "sha256": "bc8798e4cc3209e455e3dc50c6652ae73c057d40122a4a741d8b5b65a041c2f1", + "mode": 420 + }, + "docs/quickstart.md": { + "sha256": "10c69b394304f6e7d6409b6bde02b0165f5723ff4acfd49cfa8f7414f977266a", + "mode": 420 + }, + "docs/rpc-commands.md": { + "sha256": "4deeee367852f4633ef594bc079f9e9fb14640a4e3ba8a69b7c89b6f54740838", + "mode": 420 + }, + "docs/rpc-extension-ui.md": { + "sha256": "3c6f2c6edfc8b1ace6642343727aef97a7e9234c85509850921b52d6ff09a48e", + "mode": 420 + }, + "docs/rpc.md": { + "sha256": "37a03d180115460a0a275022f4aeabef1e5345458df6a9b6e8a6de18474de720", + "mode": 420 + }, + "docs/sdk.md": { + "sha256": "d9034ccbf2219a58712d8dd4fcdf47e687d77131e71dd5d119d8a71e9bd2a2e8", + "mode": 420 + }, + "docs/security.md": { + "sha256": "3b1e40a8d28b17c9eb7e496a7ba88fbe04665e3e6ae02873a58b943e325ad504", + "mode": 420 + }, + "docs/session-format.md": { + "sha256": "4f8fc42179b332bcf71fe4c85cc95d914853ff7486d68742954a45316b512a4a", + "mode": 420 + }, + "docs/sessions.md": { + "sha256": "152d6489f61a6de9349c8ceaade95d5498fbc74f21771f41ba8e87ce0e5d8936", + "mode": 420 + }, + "docs/settings.md": { + "sha256": "2915d26f91f4c46ab1b92e63cbc46b8d48888fa0e491168642a5a90f973bd814", + "mode": 420 + }, + "docs/shell-aliases.md": { + "sha256": "ebb55456870455b4da91cfd054ae2ac9b3a417e6ff48a009587c33e68ecd7807", + "mode": 420 + }, + "docs/skills.md": { + "sha256": "5b9805525b194474ef1fe5455a0917ef15ed7f710aa9bec2eee11879556469e5", + "mode": 420 + }, + "docs/slash-commands.md": { + "sha256": "d7e1db36e9186ff146a6c27de31fe169671f52cb6f7b93e1bf8d7fe76209a79c", + "mode": 420 + }, + "docs/terminal-setup.md": { + "sha256": "583381fb9dc466135a865ca406a3be299a7a8da2c9d810c5b73f401912a0a357", + "mode": 420 + }, + "docs/termux.md": { + "sha256": "1b0687157f2384385d07800ec23e6c225ef28fbf6648f55453cc7ce62f4c64ea", + "mode": 420 + }, + "docs/themes.md": { + "sha256": "6f11eb43ca6dad4c71bcfa2fc72ac13c013e3cb9383688b88923e0139b2d097c", + "mode": 420 + }, + "docs/tmux.md": { + "sha256": "9ec0abdbf74c57d51d4a93f308fb4aec4d8a3574a24cbfd9ba481d2e39f1273c", + "mode": 420 + }, + "docs/tui.md": { + "sha256": "20d3695ebd11e179a194f535f73870247544f655ebf22e42726c3be559c6d786", + "mode": 420 + }, + "docs/usage.md": { + "sha256": "6dfe7446c81d601d5ebef5769523115229f802da364fe90fcdf96e322ae9fbcd", + "mode": 420 + }, + "docs/virtual-models.md": { + "sha256": "5ab04dee3b3712b317522bb0784e28a3fef2e6b7fe1aeb1c1ca27778685a38f7", + "mode": 420 + }, + "docs/windows.md": { + "sha256": "e5a51792381a9eec859f034153c67f9ff52c36d3cff4c7a1faa0651f77f627c3", + "mode": 420 + }, + "examples/README.md": { + "sha256": "294836cfa47cd2cff1cab628533aefc5fe2d888621e43fe9f4101c09c2917a98", + "mode": 420 + }, + "examples/extensions/README.md": { + "sha256": "b51e1095d338ed8bc6cb0a40dca12da66ffd12d6f4ec7bfd57815fac84d87662", + "mode": 420 + }, + "examples/extensions/auto-commit-on-exit.ts": { + "sha256": "b89b5cb4c6f1f07b0b4291ab85a3c5f025d7d31ebb285821716502452b46af62", + "mode": 420 + }, + "examples/extensions/bash-spawn-hook.ts": { + "sha256": "c79c87653e20ac6bd9f4813aa4bb01a483b3637700fa2edda220703958532662", + "mode": 420 + }, + "examples/extensions/bookmark.ts": { + "sha256": "a71d9449d415ad75c9f3d0d13afde129b9bea2d695b889566db17ecfca256d79", + "mode": 420 + }, + "examples/extensions/border-status-editor.ts": { + "sha256": "ccbbfddd4ff402432293b7adb21f58e869f37a5d12d4da81cea10cc5fc2311cd", + "mode": 420 + }, + "examples/extensions/built-in-tool-renderer.ts": { + "sha256": "4bfd0953e3dbf96ee0cdb26989a60ec91d4850323852ef09eafe5c2efdf23c00", + "mode": 420 + }, + "examples/extensions/claude-rules.ts": { + "sha256": "8a935f14f62c1c1a3e937f5c912c2215e25aa5fa591716e23c86224ce1dc34cd", + "mode": 420 + }, + "examples/extensions/commands.ts": { + "sha256": "36716b53da169936c7e1360a4fde1e2fc0c3356a6505f177235f09c5538c6e4f", + "mode": 420 + }, + "examples/extensions/confirm-destructive.ts": { + "sha256": "55502f82109df07e922ef3ec41af1b1a5ed33d4d6d56076654ab2bc88e2e0888", + "mode": 420 + }, + "examples/extensions/custom-compaction.ts": { + "sha256": "ea797b3508fd8600526f381466e8b08bb84f3d234932a9d6ce512698527d0aa9", + "mode": 420 + }, + "examples/extensions/custom-footer.ts": { + "sha256": "1e5f17bec95a48d407fbcc092faf7e1f6021c30cee4a66ec09bc120412292cb2", + "mode": 420 + }, + "examples/extensions/custom-header.ts": { + "sha256": "5de09cf861d3965201991d6e837d6b68e8a2e80cb89279ee9177cc05d31f723d", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/index.ts": { + "sha256": "52bb733e1a25d9e175a644e6a6e17bbf61071beecf0f2903869baa9579a05ab3", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/package-lock.json": { + "sha256": "4c1ccae817de13a0d0eee610fb12ee699b3c1f97cf28cbe83ec50cab0f338a77", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/package.json": { + "sha256": "b2ceede9b12cef79f37d02c8e850132afdab1b3316c16d622c0c8343244e85c3", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/index.ts": { + "sha256": "0c4adac82eff25fef23223af77e11cd53535834fece9731009b125446e213158", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/package.json": { + "sha256": "4af767a958f402efc19fc6157c75c209edc45c8ab6103b60f13973369ba41990", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/test.ts": { + "sha256": "58dfa6fdb63521ccad551b7463ff56d4aecf735074934a8e9fc34b71e7083d00", + "mode": 420 + }, + "examples/extensions/debug-provider.ts": { + "sha256": "a915f86a26a0257537939701797b1f613072275e76bdd8a27dd941bbf519643a", + "mode": 420 + }, + "examples/extensions/dirty-repo-guard.ts": { + "sha256": "0037a69abdf4cf98181db799bef5bdee8981861315eb82ddab78126c2b0f13c4", + "mode": 420 + }, + "examples/extensions/doom-overlay/.gitignore": { + "sha256": "b26aa26693c610d46d9c8b387b28d20dc5196a73b979f5522e24dd689aab7bc8", + "mode": 420 + }, + "examples/extensions/doom-overlay/README.md": { + "sha256": "e17bc73b76c7977ffeef99120c7896ff992cf5938a60c4d7427a7b19fc180d97", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom/build/doom.js": { + "sha256": "c3147cf4881c88b4210c017c26828f52aff83f0c4db27d7dad3fe6cda9ade1fb", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom/build/doom.wasm": { + "sha256": "571d161956593508cf4ade732ae93753f00484bb526667a8676571cca14dec7d", + "mode": 493 + }, + "examples/extensions/doom-overlay/doom/build.sh": { + "sha256": "ab09c9b6dc98ea7c1fa8e74c13981ce69b698c01aec77309c5c7d40ad114d38c", + "mode": 493 + }, + "examples/extensions/doom-overlay/doom/doomgeneric_pi.c": { + "sha256": "98210909e1e7dee71cbbdba64f1db4bebd88b504e861ee36ea42093c59b251a1", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom-component.ts": { + "sha256": "d95dc47c92db0fdc5b0e0b56a277c67f277012d3f7296df745d8a93f64d5a42e", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom-engine.ts": { + "sha256": "706e43f70bd374b73928e5a44d196d1dd5c03d93979d08bdfc01d049b5083855", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom-keys.ts": { + "sha256": "e8bd99a4ce90dd44ad120c58f3b2ba7bc0b0c662bcdc4ed4994d059130be9fdd", + "mode": 420 + }, + "examples/extensions/doom-overlay/index.ts": { + "sha256": "f89818dd75c7d8970458b1dbb9c5e85fea9b86534fd740c898c1f6f9c9f9ceba", + "mode": 420 + }, + "examples/extensions/doom-overlay/wad-finder.ts": { + "sha256": "20254888262081bf96932253bfd346971deefa48d8df011fd3baa7aa2d653b7d", + "mode": 420 + }, + "examples/extensions/dynamic-resources/SKILL.md": { + "sha256": "f8f34b0d81aa791a2d56501756fe0ace21fb78ca2754025d093f41d3b0ccaf69", + "mode": 420 + }, + "examples/extensions/dynamic-resources/dynamic.json": { + "sha256": "2bf0164779b4ea3c2572f33b1e979c427a4839193998fda19bfc5cba2a4692c5", + "mode": 420 + }, + "examples/extensions/dynamic-resources/dynamic.md": { + "sha256": "e61e2c971d02d628a58f7c36b9b37bd3e636e4ad60618a7f65e2b41410c38f35", + "mode": 420 + }, + "examples/extensions/dynamic-resources/index.ts": { + "sha256": "dfbc49418fe46ebc9696fa335709df2ed0d4c4703da8c1ad0eb317d11fce8433", + "mode": 420 + }, + "examples/extensions/dynamic-tools.ts": { + "sha256": "4394098c11f98ffc26e2487acda2e09b9467ef9ff5867def7273cb0c82a2b6c0", + "mode": 420 + }, + "examples/extensions/entry-renderer.ts": { + "sha256": "e4c01fcf382972d570ebf37d044e32065eae062fe70987b1b885d75d46aa0b11", + "mode": 420 + }, + "examples/extensions/event-bus.ts": { + "sha256": "1185942b7afd83692d9e8da029e8594d702157398c693ec8b3ddb02b0780828a", + "mode": 420 + }, + "examples/extensions/file-trigger.ts": { + "sha256": "6f5fc3dc8fa387d8ca6973786c521feb94da58838b0e171a881fb6b3a5243ac0", + "mode": 420 + }, + "examples/extensions/git-checkpoint.ts": { + "sha256": "b60c85f797091a1f21ccb2a998027c6229fe995bdd62dbb1cdcc22946768f40b", + "mode": 420 + }, + "examples/extensions/git-merge-and-resolve.ts": { + "sha256": "a3c077c87544122abe83b275f3d527f60c80c87f460ea2499d66bde9b4880133", + "mode": 420 + }, + "examples/extensions/github-issue-autocomplete.ts": { + "sha256": "4646ef360e57aa98c787ba21ca2f2e8acb3db30c4db0503e64cdbea4a4c2ac65", + "mode": 420 + }, + "examples/extensions/gondolin/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/gondolin/index.ts": { + "sha256": "9a9897639de8127a2abe6bb0d00f0005cd9e9cf9d51fd15893a827e67326bf62", + "mode": 420 + }, + "examples/extensions/gondolin/package-lock.json": { + "sha256": "256d3e45b7c6d293bdfd88a3c4884d8bb38ecdbc4064c290255d2b4d193ea535", + "mode": 420 + }, + "examples/extensions/gondolin/package.json": { + "sha256": "010f1a949005f706239d788a2481c355028eb4c366b4ea16c87577e63caab905", + "mode": 420 + }, + "examples/extensions/handoff.ts": { + "sha256": "ea07b6f680357b677d0f43f0016961ee8cb1331d37ae113e9466a3acffb2e381", + "mode": 420 + }, + "examples/extensions/hello.ts": { + "sha256": "0aa4e9800c2526914d4c1edb00b2cfa9bd9dd6da5218289994cccd5f5bfa4934", + "mode": 420 + }, + "examples/extensions/hidden-thinking-label.ts": { + "sha256": "179dc3704b5c563b49a7f4577222bf9bbaa5becf98b85c75e56ab26c001c5b22", + "mode": 420 + }, + "examples/extensions/inline-bash.ts": { + "sha256": "69b9dfc34e7b1ada9afe10cda3384b0b1a6f3c3a47da5fb7d16f914399b023d4", + "mode": 420 + }, + "examples/extensions/input-transform-streaming.ts": { + "sha256": "5c251ebda6163c8279e3c6d5f0aee56421ed150255dbb954f08ca7d2280b3ad2", + "mode": 420 + }, + "examples/extensions/input-transform.ts": { + "sha256": "cf0f65d610631ca75d18aae1c8f1408139702f674cb2835f8c009b943776474b", + "mode": 420 + }, + "examples/extensions/interactive-shell.ts": { + "sha256": "36a7efb5b132e666c643129ce09c3f83f56aed9924563e6b7269e0e126611ecc", + "mode": 420 + }, + "examples/extensions/jev-router.ts": { + "sha256": "cf2c9ef4baf9233faa133e99255786b00518cb4977d068c811a11b3c96f0a2d6", + "mode": 420 + }, + "examples/extensions/mac-system-theme.ts": { + "sha256": "d7349bae999a798c71b283165e06d0142d41b0264bfdad11ddcb1ec8b62f8710", + "mode": 420 + }, + "examples/extensions/message-renderer.ts": { + "sha256": "bedc198df5215c0a758e0743224599003630741a4ff72d4327837ce309e5ec8c", + "mode": 420 + }, + "examples/extensions/minimal-mode.ts": { + "sha256": "040cbc7619b2e903888c8626e2fa55ca5e3f820535dfd1aadcb2060e156a8770", + "mode": 420 + }, + "examples/extensions/modal-editor.ts": { + "sha256": "4742283930949fad1920359540aa9d42c833d2d24eaa49564bfc08f90e328b0d", + "mode": 420 + }, + "examples/extensions/model-status.ts": { + "sha256": "b6c073c0bdf26ac17e2fed01d02b6447fd4e2763b7b6ca2f2b493574ced25247", + "mode": 420 + }, + "examples/extensions/notify.ts": { + "sha256": "70e4333e09ce00d546c116fd2e918abf7616c70b5da6e88ba8ee21a326afd483", + "mode": 420 + }, + "examples/extensions/overlay-qa-tests.ts": { + "sha256": "4b1778e9e247c9eb24fe48e13fff0272f6f78b8102b4ed085d03598c3b98773f", + "mode": 420 + }, + "examples/extensions/overlay-test.ts": { + "sha256": "801b7c2eeec57c3569187b4b4d851f2159223ec6e9772e406ec65ed728ca6db9", + "mode": 420 + }, + "examples/extensions/permission-gate.ts": { + "sha256": "e32086ba9dd9b4ba0fd308b65241d9affa0b5ecd3f90ef7ea1b9f2b30cab8cc1", + "mode": 420 + }, + "examples/extensions/pirate.ts": { + "sha256": "dd6ce684bbe7630e4a749fe133ef8b8b875adbc7e31284b17fe0b8157d51e013", + "mode": 420 + }, + "examples/extensions/plan-mode/README.md": { + "sha256": "1ff8d611b8fbafef4ca1e3dbc72136fbcc166461b69f859b5e961b15147bff39", + "mode": 420 + }, + "examples/extensions/plan-mode/index.ts": { + "sha256": "bef2bf0712aea7b7d0f73d174e24ac5309679e6cc143f7bcbf1a9e0121222af8", + "mode": 420 + }, + "examples/extensions/plan-mode/utils.ts": { + "sha256": "12ea28d57cc55e68e8d52a2f1d16df8b2e7f2749cd694d41453028002a2bdad9", + "mode": 420 + }, + "examples/extensions/preset.ts": { + "sha256": "7942b7f933f97bcc29024a87f8526969413ba6f87b0bec537648a2926a0a4e29", + "mode": 420 + }, + "examples/extensions/project-trust.ts": { + "sha256": "78781c1f2a1613cf31c02a29ff6a6abd86b60bdf34343465e46cd85c01491578", + "mode": 420 + }, + "examples/extensions/prompt-customizer.ts": { + "sha256": "429546f46795bccda4078deace5729b8230f6103807d424eaf71817c6e07da0a", + "mode": 420 + }, + "examples/extensions/protected-paths.ts": { + "sha256": "9ca66b1f3b1b9a61cc5ddc660f92cca2ea66c4264a1bdb6b00e0da5844399fe2", + "mode": 420 + }, + "examples/extensions/provider-payload.ts": { + "sha256": "e0016077798098f528b1129954c30a28809a1513924d9c2e0efda9328d503a49", + "mode": 420 + }, + "examples/extensions/qna.ts": { + "sha256": "a639a65a486fd05a42afcf14aa59dc32e505e41ada43869167ddfd789a3c7da6", + "mode": 420 + }, + "examples/extensions/question.ts": { + "sha256": "a1625d2c72a65d859154a7e66bef222d48ca9e3654a8bb910829ad350378a4c4", + "mode": 420 + }, + "examples/extensions/questionnaire.ts": { + "sha256": "3f1f719d81152ff013f5d36a97cc3976388d43f2d89bff0f19827d3443d903e6", + "mode": 420 + }, + "examples/extensions/rainbow-editor.ts": { + "sha256": "f95f17ba8a0376180bd23510c20151e83aadc74142e989f14227bf5c4196883f", + "mode": 420 + }, + "examples/extensions/reload-runtime.ts": { + "sha256": "911f26b872860a551572fa1d94d251ddd476d4ea9483b6052e9658d2784b38fc", + "mode": 420 + }, + "examples/extensions/rpc-demo.ts": { + "sha256": "3d54a396eee12eb04ea12d0ea03716a6edd3e8743502b6a529e3b8d429ae640d", + "mode": 420 + }, + "examples/extensions/sandbox/.gitignore": { + "sha256": "16d30e4462189fb14dd611bdb708c510630c576a1f35b9383e89a4352da36c97", + "mode": 420 + }, + "examples/extensions/sandbox/index.ts": { + "sha256": "b5e527aac27254002f27b8bb9de7bc62e242e39eeb7c963683096039e0aa173e", + "mode": 420 + }, + "examples/extensions/sandbox/package-lock.json": { + "sha256": "cd4e34dbd8a9d4b0efdb58becb6c4201eb2ad9e44228b9a272b6853bfb54539d", + "mode": 420 + }, + "examples/extensions/sandbox/package.json": { + "sha256": "4b4bfe0ebeecf2af747a3e023a5edc143cba9c36137d10051e1124d1c1ad9732", + "mode": 420 + }, + "examples/extensions/send-user-message.ts": { + "sha256": "9261e433c782f354530105e1c9723a19798868f72fd1f5d68af3a4624ad7e446", + "mode": 420 + }, + "examples/extensions/session-name.ts": { + "sha256": "a0b9fd94e2cc827931150cc409238b473da635ab1a1cddce633cae5937a3ae24", + "mode": 420 + }, + "examples/extensions/shutdown-command.ts": { + "sha256": "68dc7f9561aa9b1c0f39cae19c0f41d56171a99875972e5998de3caa75f96e09", + "mode": 420 + }, + "examples/extensions/snake.ts": { + "sha256": "3ef506ab52927e866fad096f83b7cbfd0eb50e1504dcabf1b1eb28a06745c937", + "mode": 420 + }, + "examples/extensions/space-invaders.ts": { + "sha256": "43281ae25bea645d44db01234f24df80e2f866771d02de0b416ab32fb3883a1f", + "mode": 420 + }, + "examples/extensions/ssh.ts": { + "sha256": "c03bdef919457013e3798d10250f34255742fd3dc79885ac3d321b34f6a1149d", + "mode": 420 + }, + "examples/extensions/status-line.ts": { + "sha256": "26e27b1158e980551f63f46e55dad055ac613779c2b8c39df4cddc11be26103d", + "mode": 420 + }, + "examples/extensions/structured-output.ts": { + "sha256": "72a3a665ce3de1419c837db6eb8a0e7a890310c6e02e4d59c454e5c9514c15ec", + "mode": 420 + }, + "examples/extensions/subagent/README.md": { + "sha256": "6a888330bf974657751e09f5d375a9039872d59d6606fc217c5105c7eca51b1a", + "mode": 420 + }, + "examples/extensions/subagent/agents/planner.md": { + "sha256": "452f21f65957753cec3b0772004209c108071be168e627fffd7191234c0ce64e", + "mode": 420 + }, + "examples/extensions/subagent/agents/reviewer.md": { + "sha256": "b5c3f13b67207e8e32de631f8cd52a373d9b956028e2f8c1fd8b737b5f239265", + "mode": 420 + }, + "examples/extensions/subagent/agents/scout.md": { + "sha256": "3b84e8e01c6e05c8dfbff1c845c0c6371f4f16350735d4d1dbd02bfa98b1d4e6", + "mode": 420 + }, + "examples/extensions/subagent/agents/worker.md": { + "sha256": "ca2b2298fcbd8bfafcfe61977dc8075001a64b9c4d8c289c3b5eaa896efe6238", + "mode": 420 + }, + "examples/extensions/subagent/agents.ts": { + "sha256": "cc85ecd33b89aa575bea3d2a9a9da7654a6305da4de02e5041aa5342a1de29b4", + "mode": 420 + }, + "examples/extensions/subagent/index.ts": { + "sha256": "1d11858d163e2a3a258af51c1923fc55ad825813acd6224462fd25ad61f5401b", + "mode": 420 + }, + "examples/extensions/subagent/prompts/implement-and-review.md": { + "sha256": "019ba4bf42d596ea03fc4564be5b2cc55a06227c95db19a4d5a5bb96697a75f9", + "mode": 420 + }, + "examples/extensions/subagent/prompts/implement.md": { + "sha256": "65462de33c97ac1372cfed270dad9af164605b05028c069dd4c013ff9792675b", + "mode": 420 + }, + "examples/extensions/subagent/prompts/scout-and-plan.md": { + "sha256": "9506399391e0d2c71826c45f26cc332175dd964465f99a8bb880e109f8ed89f4", + "mode": 420 + }, + "examples/extensions/summarize.ts": { + "sha256": "9c75e838071773d5bacce27ac8c5305160c56958018884f960c3db96b3d09368", + "mode": 420 + }, + "examples/extensions/system-prompt-header.ts": { + "sha256": "aa97f1add4fe347b3fc37cd3709950a105b0db5d6cdaf19325b58a9fbfd05223", + "mode": 420 + }, + "examples/extensions/tic-tac-toe.ts": { + "sha256": "479597f4e76ef1eae3dae71ec9d5c26978c5b9221f950e3e134078c5b532314b", + "mode": 420 + }, + "examples/extensions/timed-confirm.ts": { + "sha256": "c35304599e6f07238ef5472efe9c3f2fef2a022f801750ed792ef9061c7de10f", + "mode": 420 + }, + "examples/extensions/titlebar-spinner.ts": { + "sha256": "ec56e7edf954966b5991bcad63141be1a09d17753cdd0c59efca1b9e649dadf2", + "mode": 420 + }, + "examples/extensions/todo.ts": { + "sha256": "e46824d00217e25242c186d41837cc84ca81b23f978500323448502a9a424ee2", + "mode": 420 + }, + "examples/extensions/tool-override.ts": { + "sha256": "5d3cc796311a16132db810fcd9e437c16abd46978b723b9120734e57cc545726", + "mode": 420 + }, + "examples/extensions/tools.ts": { + "sha256": "5eaebc40cffe885fee25c95e3d8aec79cfbd595b6eac8ac5a6de6c4afec3d5d8", + "mode": 420 + }, + "examples/extensions/trigger-compact.ts": { + "sha256": "92c3b831f9f7bf01a3e0a2df11f4e404f5ac5843000b94e8340a5d78f3cb3add", + "mode": 420 + }, + "examples/extensions/truncated-tool.ts": { + "sha256": "4ed5fbeb6da53eab8d1e04721d77961b8d1b3bf5e11d1512a17fc5a3f58f9955", + "mode": 420 + }, + "examples/extensions/widget-placement.ts": { + "sha256": "ffabd33cf58efb908de2025aac4cb41daf13dbeee24b5ad8dcd0361b43eff6d7", + "mode": 420 + }, + "examples/extensions/with-deps/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/with-deps/index.ts": { + "sha256": "8d4dbc7028c891eac7f0386d25eecea3e9ac0de7fd841b5bb08137e2ec5ab443", + "mode": 420 + }, + "examples/extensions/with-deps/package-lock.json": { + "sha256": "a4863bc4e63ff31dd43a6b469835dd0057f8792d940af0a349ec35187ad03663", + "mode": 420 + }, + "examples/extensions/with-deps/package.json": { + "sha256": "1cedd74eb4891d548ff4ae0b90adb3ce73086fa21b11fe489e0d8b1ead1c5267", + "mode": 420 + }, + "examples/extensions/working-indicator.ts": { + "sha256": "e88295d765b78fd6aac0e93db92b183ee9b7103dc21577e3c5a42665dfad032c", + "mode": 420 + }, + "examples/extensions/working-message-test.ts": { + "sha256": "91ee09378f13cce142a4314bec1bfe9784e10484c33463c4fe3139c391c117a0", + "mode": 420 + }, + "examples/rpc-client.ts": { + "sha256": "b46b19c291fbcb59f8f9c539c2dbe12a43eae25e4d7528084844bbb79d27b6f7", + "mode": 420 + }, + "examples/rpc-extension-ui.ts": { + "sha256": "09bc33c28fdc9032e3752a487d659ac21c6776c6f589c5b1ae0967ef4ee5f127", + "mode": 420 + }, + "examples/sdk/01-minimal.ts": { + "sha256": "ffe5215e36f750f5ca7146cc7dfa5be36461292036d55b8db652b4d26331f351", + "mode": 420 + }, + "examples/sdk/02-custom-model.ts": { + "sha256": "dccc8dbdb1742ea00beeed87d302b8a23ac7f7e36a0b6a4c43b283c21be0ee55", + "mode": 420 + }, + "examples/sdk/03-custom-prompt.ts": { + "sha256": "d6db1f36a8e04466d9cbcc24ffb6fe0697858aa6c0e5013d5d90f12798314ef5", + "mode": 420 + }, + "examples/sdk/04-skills.ts": { + "sha256": "4cbf189bf88d14155281349d4d39cc236066d98095aae2d184d4ba7267ab5e42", + "mode": 420 + }, + "examples/sdk/05-tools.ts": { + "sha256": "5df5f9577b343d22d072d839341916536299c8191dc66c6c801a8bbddda23544", + "mode": 420 + }, + "examples/sdk/06-extensions.ts": { + "sha256": "8e01ee409200b6516e9a6a0d9f3b3f61d2ad78aaaa18bd2ef9a8f77820d149b2", + "mode": 420 + }, + "examples/sdk/07-context-files.ts": { + "sha256": "37bd18a284e961903d646e0cd1abdeda72cd968dbe4f40892e8d345a66916b36", + "mode": 420 + }, + "examples/sdk/08-prompt-templates.ts": { + "sha256": "a1b314988f1f5b2e61c80c19c79b68630a2f3c443008a200376286710a2744ed", + "mode": 420 + }, + "examples/sdk/09-api-keys-and-oauth.ts": { + "sha256": "d1d2b1bd4e57a3b2ab6f910068e8382759d12a94fbecf1332e96a9ebe4683eb4", + "mode": 420 + }, + "examples/sdk/10-settings.ts": { + "sha256": "558f82b9d0655935e6a28ec09f6c35f0f9593f8b034f25d45400c02bc77a778a", + "mode": 420 + }, + "examples/sdk/11-sessions.ts": { + "sha256": "538c7028bf6823c84be930eb6745731f2c87ba05a45df2d86d93e66f01674576", + "mode": 420 + }, + "examples/sdk/12-full-control.ts": { + "sha256": "3cf794d470ddbde4504ae965d24a23c42378d1ac86033522e46907ac05518343", + "mode": 420 + }, + "examples/sdk/13-session-runtime.ts": { + "sha256": "0e9f408720a561a62b2630eccb18ece49f9dba567dc65494757da0a91608827a", + "mode": 420 + }, + "examples/sdk/14-codemode-mcp.ts": { + "sha256": "02ae7714e22753a76b0f44033366d43529185b325b92d47bb5d1d0eb5ae34f5e", + "mode": 420 + }, + "examples/sdk/README.md": { + "sha256": "c1f9d8f58c300195d03a420222d2d5370d2884a18a4190a1d8c8ea9371c016ff", + "mode": 420 + }, + "export-html/ansi-to-html.d.ts": { + "sha256": "57d23f45aef05ba2b1a52a8e6b0e1f1ba51b55ab0b54e8d42938a438b088977a", + "mode": 420 + }, + "export-html/ansi-to-html.d.ts.map": { + "sha256": "2847dc4f48c9ec5c8cd7aeacf1a6083ab4fc44f4b7e6667547b1549215c68576", + "mode": 420 + }, + "export-html/ansi-to-html.js": { + "sha256": "44e5fd3585104e5e8cc0c7bea91a10efa7cb7e3b5bc5e2fb9090d5be4419df14", + "mode": 420 + }, + "export-html/ansi-to-html.js.map": { + "sha256": "d24d23d27ad31a21a4fb133c6e7ba2dc2c876f28656ff1254b1af3da234dc51c", + "mode": 420 + }, + "export-html/index.d.ts": { + "sha256": "cd83b5d85e893b8ca18742b28038883dd7cf06c98a203e00d74040f78a6a656d", + "mode": 420 + }, + "export-html/index.d.ts.map": { + "sha256": "1c31d2cbab7deb811274f473316e1fed48bc2a93707f2f964db86ba2f200d522", + "mode": 420 + }, + "export-html/index.js": { + "sha256": "9b7a893c46341d2557ba79dd88307797054577c8789665281d1dfb4134d347a3", + "mode": 420 + }, + "export-html/index.js.map": { + "sha256": "f2bf1920c337f74e83454665e5a1af9621cd9e67fbcca9aade7231eb624bf9a7", + "mode": 420 + }, + "export-html/template.css": { + "sha256": "8ee19851f8e583277ed396cbb76496687aa556707fdfc1f78d1118bff87c740a", + "mode": 420 + }, + "export-html/template.html": { + "sha256": "47d208235b066c856dae69c60281a3b99d164b99be9b81153d880375c9327105", + "mode": 420 + }, + "export-html/template.js": { + "sha256": "946bd9e3e8255c1ebb40f806def18007aa9397816291a7bf2ac0e4e99cec07a4", + "mode": 420 + }, + "export-html/tool-renderer.d.ts": { + "sha256": "ea982244e87c3568ad7bbb28427b83a19506136b5a498c7b496454338483ffd4", + "mode": 420 + }, + "export-html/tool-renderer.d.ts.map": { + "sha256": "2fab18f196e020f0b896b27db7cf790c233f0d0af55de04194e60b5bff137fce", + "mode": 420 + }, + "export-html/tool-renderer.js": { + "sha256": "8d3bd1f1b068996fe8f7d003bdb6623284043e1393a2db92ff45aeb9d6c33b9a", + "mode": 420 + }, + "export-html/tool-renderer.js.map": { + "sha256": "7bead30f87d1b345c70c66953a517fb51d1e1092089158ad5e92280cc97f6ea8", + "mode": 420 + }, + "export-html/vendor/highlight.min.js": { + "sha256": "837a6fa5b0c736b52bbde2b2b6190f305da3fc9ed41681db5321507057b5c846", + "mode": 420 + }, + "export-html/vendor/marked-cjk-friendly.min.js": { + "sha256": "93d2c938754cb24eef5b9ffecc268a98ef6beeb75dad614b0e874176aeeba565", + "mode": 420 + }, + "export-html/vendor/marked.min.js": { + "sha256": "438eedfcf932a414d0d0bfeea32dc365c063563b8ca713b4687fc8f8b501e5e4", + "mode": 420 + }, + "native/linux/prebuilds/linux-x64/linux-platform-x11.node": { + "sha256": "522d2550f7fa5d0d5bd204f45983f786ce2be9cf80f27f00179640d31d0689d6", + "mode": 493 + }, + "package.json": { + "sha256": "df6883685382d6a914e5797232744c26579fe7db5576ea0e9c859d018ca2d54e", + "mode": 420 + }, + "photon_rs_bg.wasm": { + "sha256": "10468181565c56004c867f3a4af96f89a0ef5a63a72f2b5fb12c1f1992a3615c", + "mode": 420 + }, + "pi": { + "sha256": "0d7611d0b86d38ec8f65dda8e8bee9c5e907c7bae381e9cfff9161b21e8087d0", + "mode": 493 + }, + "theme/dark.json": { + "sha256": "e1f96982f8cb7d2a48eca7db03c9cb282cb1e14a243680cd855638ecd6a251b6", + "mode": 420 + }, + "theme/light.json": { + "sha256": "2dee6a4be7f765e7fd7728166b7ebf8c29e4ce313f8ec6b38ae50699e3abc2f3", + "mode": 420 + } + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-models.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-models.json new file mode 100644 index 0000000..32fd1b7 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-models.json @@ -0,0 +1,42 @@ +{ + "providers": { + "deepseek": { + "api": "openai-completions", + "baseUrl": "https://api.deepseek.com", + "models": [ + { + "id": "deepseek-flash", + "name": "DeepSeek V4.1 Flash", + "reasoning": true, + "input": [ + "text", + "image" + ], + "thinkingLevelMap": { + "minimal": null, + "low": "low", + "medium": null, + "high": "high", + "max": "max" + }, + "contextWindow": 1000000, + "maxTokens": 384000, + "cost": { + "input": 0.3, + "output": 1.2, + "cacheRead": 0.006, + "cacheWrite": 0 + }, + "compat": { + "supportsStore": false, + "supportsDeveloperRole": false, + "maxTokensField": "max_tokens", + "requiresReasoningContentOnAssistantMessages": true, + "thinkingFormat": "deepseek" + } + } + ], + "apiKey": "${DEEPSEEK_API_KEY}" + } + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/preflight-evidence.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/preflight-evidence.json new file mode 100644 index 0000000..760332c --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/preflight-evidence.json @@ -0,0 +1,106 @@ +{ + "job": "jobs/pi-restricted-dev2-install-pair-01-20261104", + "summary_path": "jobs/pi-restricted-dev2-install-pair-01-20261104-input/summary.json", + "summary_sha256": "6e6c92ed27b0f28d6c7e7df4aaefe939d097fad180d7da7fe7a0264e62d78542", + "trials": [ + { + "task": "terminal-bench/db-wal-recovery", + "trial": "db-wal-recovery__WL28NdB", + "result_path": "jobs/pi-restricted-dev2-install-pair-01-20261104/db-wal-recovery__WL28NdB/result.json", + "result_sha256": "41c80e23cb591099135e89e91777b7f0a6f1dd7f9f9380aada37eb526645ea23", + "native_exception": null, + "model_executed": false, + "verifier_executed": false, + "setup_seconds": 485.530439, + "bootstrap": { + "status": "ready", + "stage": "complete", + "error_type": null, + "verifier_preflight": { + "profile": null, + "status": "not_configured" + }, + "integrity_preflight": "passed", + "cache_used": true + }, + "enforcement": { + "protocol": "restricted-v1", + "policy": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "rules_sha256": "89db5b77afe8608c1d10884151a3730d5878929ecc3a0ac53b0ed164bb7fbf47", + "preflight": "passed", + "agent_phase": "not_started", + "host_mounts": false, + "verifier_isolation": "task_defined; shared tasks remain shared", + "dns_upstreams": [ + "172.17.0.1" + ], + "positive_control": { + "reachable": true + }, + "negative_control": { + "reachable": false, + "error_type": "URLError", + "reason_type": "SSLEOFError" + }, + "allowed_controls": { + "api.deepseek.com": { + "reachable": true + } + } + }, + "started_at": "2026-11-04T14:53:51.094293Z", + "finished_at": "2026-11-04T15:02:09.762900Z" + }, + { + "task": "terminal-bench/torch-pipeline-parallelism", + "trial": "torch-pipeline-parallelism__aykTgx7", + "result_path": "jobs/pi-restricted-dev2-install-pair-01-20261104/torch-pipeline-parallelism__aykTgx7/result.json", + "result_sha256": "2ba5ed6dcd622b2af2fcacfa4af1c22e9d13ab44b7824defa301333f54c66598", + "native_exception": { + "exception_type": "AgentSetupTimeoutError", + "exception_message": "Agent setup timed out after 1800.0 seconds", + "exception_traceback": "Traceback (most recent call last):\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/tasks.py\", line 507, in wait_for\n return await fut\n ^^^^^^^^^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 939, in setup\n await self.install(environment)\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-dev2-install-pair-01-20261104-input/source/harbor_adapter/adapter.py\", line 141, in install\n await self.exec_as_agent(environment, command=profile.command)\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 884, in exec_as_agent\n return await self._exec(\n ^^^^^^^^^^^^^^^^^\n environment, command, env=env, cwd=cwd, timeout_sec=timeout_sec\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 835, in _exec\n result = await environment.exec(\n ^^^^^^^^^^^^^^^^^^^^^^^\n ...<5 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 1096, in exec\n return await self._compose_exec(\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<6 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 1173, in _compose_exec\n return await self._run_docker_compose_command(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<4 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 649, in _run_docker_compose_command\n result = await self._collect_buffered_output(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<3 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 679, in _collect_buffered_output\n stdout_bytes, stderr_bytes = await process.communicate(input=stdin_data)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/subprocess.py\", line 201, in communicate\n stdin, stdout, stderr = await tasks.gather(stdin, stdout, stderr)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/subprocess.py\", line 181, in _read_stream\n output = await stream.read()\n ^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 723, in read\n block = await self.read(self._limit)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 730, in read\n await self._wait_for_data('read')\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 539, in _wait_for_data\n await self._waiter\nasyncio.exceptions.CancelledError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 1242, in _setup_agent\n await asyncio.wait_for(\n ...<2 lines>...\n )\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/tasks.py\", line 506, in wait_for\n async with timeouts.timeout(timeout):\n ~~~~~~~~~~~~~~~~^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/timeouts.py\", line 117, in __aexit__\n raise TimeoutError from exc_val\nTimeoutError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 375, in run\n await self._prepare()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 413, in _prepare\n await self._setup_agent()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 1247, in _setup_agent\n raise AgentSetupTimeoutError(\n f\"Agent setup timed out after {self._agent_setup_timeout_sec} seconds\"\n ) from exc\nharbor.trial.errors.AgentSetupTimeoutError: Agent setup timed out after 1800.0 seconds\n", + "occurred_at": "2026-11-04T15:23:54.614773Z" + }, + "model_executed": false, + "verifier_executed": false, + "setup_seconds": 1800.034097, + "bootstrap": { + "status": "failed", + "stage": "verifier_dependencies", + "error_type": "CancelledError", + "verifier_preflight": { + "profile": "torch-pipeline-parallelism-v1", + "status": "incomplete" + }, + "integrity_preflight": null, + "cache_used": true + }, + "enforcement": { + "protocol": "restricted-v1", + "policy": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "rules_sha256": "89db5b77afe8608c1d10884151a3730d5878929ecc3a0ac53b0ed164bb7fbf47", + "preflight": "pending", + "agent_phase": "not_started", + "host_mounts": false, + "verifier_isolation": "task_defined; shared tasks remain shared" + }, + "started_at": "2026-11-04T14:53:51.926910Z", + "finished_at": "2026-11-04T15:24:05.548272Z" + } + ], + "whole_trial_retries": 0, + "setup_retries": 0 +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/preflight-freeze.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/preflight-freeze.json new file mode 100644 index 0000000..02c4387 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/preflight-freeze.json @@ -0,0 +1,28 @@ +{ + "schema_version": 1, + "job_dir": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-dev2-install-pair-01-20261104", + "frozen_at": "2026-11-04T15:24:54.776778+00:00", + "files": { + ".integrity/db-wal-recovery__WL28NdB/enforcement.json": "8934282e6747862faf1f89f18836a1f80806feee56e8305cdbfae3b0cbe1cad5", + ".integrity/db-wal-recovery__WL28NdB/network-preflight.json": "6cdb1220eec019f01d9e3f1b8854a6a14c6c48e054739f1140dcf0a293a0aaa0", + ".integrity/torch-pipeline-parallelism__aykTgx7/enforcement.json": "f217d8d9a4e2d95fbcd8c2aeead41ae329d528d7a07eecd8785218b6683ed642", + "config.json": "cfc36df8a56a15316db9be5c22ab15df0f94a1dc2a697d03f906e073e6300f91", + "db-wal-recovery__WL28NdB/agent/bootstrap.json": "deb2178e8d6b34ffdb8400a01284bc147791c036ab0ae2bcf8ece2c9a4643a01", + "db-wal-recovery__WL28NdB/agent/process-supervision-required": "9e67a1538c0f96556945933b784624bed4b3f7d177108e2ee1af88d6ea1314c8", + "db-wal-recovery__WL28NdB/config.json": "5ccc3b3cbd6332914be81db081b59c4a3c2d9ff11885e420a258a5433439d47d", + "db-wal-recovery__WL28NdB/lock.json": "b9567695f76f6778258c354c79b5b135b8ae8483e2743a88beb7bf707503ae4e", + "db-wal-recovery__WL28NdB/result.json": "41c80e23cb591099135e89e91777b7f0a6f1dd7f9f9380aada37eb526645ea23", + "db-wal-recovery__WL28NdB/trial.log": "8286817021a35d7d81071bb40c8ced838b47bcf90f5895aafa0a02647a841694", + "job.log": "ec59d282be0a7ba8affce229bcdd18ff3fdbde321cf50c9b375302403ea97348", + "lock.json": "f87b6f67f499e432a339cde0f516fcec92cea8aa3b085c7f06a23f6394141ee4", + "result.json": "6cbfabc42faae99bcb3eb2cee523e5dc39fccc719bc4bec46bea243f03e57c5e", + "torch-pipeline-parallelism__aykTgx7/agent/bootstrap.json": "4374d004f1b66765deb873f3edd4670009a29408f1a4b10aad5d44ec790875da", + "torch-pipeline-parallelism__aykTgx7/agent/process-supervision-required": "94e95f0bb38f8e0fb52a6dab21f7708dde104b7edeb2eb5ce6b7e68f1328c295", + "torch-pipeline-parallelism__aykTgx7/artifacts/manifest.json": "8d0a02e9c17d62af6176024f463c041ed30ff29d3a7b66dadbeab1d762b0cbd9", + "torch-pipeline-parallelism__aykTgx7/config.json": "551d812357bfab62ba3f28d0927289bab7187ec497cca10df3557788b5cca656", + "torch-pipeline-parallelism__aykTgx7/exception.txt": "b742e3cbc424d9cd7267723f2088d6f368527d3ab4b5a480795152979c386a77", + "torch-pipeline-parallelism__aykTgx7/lock.json": "bbb54c0511e72e53316c74ae4c9a404d5d41ed2390fff713fb55d735ac24f349", + "torch-pipeline-parallelism__aykTgx7/result.json": "2ba5ed6dcd622b2af2fcacfa4af1c22e9d13ab44b7824defa301333f54c66598", + "torch-pipeline-parallelism__aykTgx7/trial.log": "38cdabaa625cde47ea52360a626125fb36e0e4772fe24d4fb6a005dd83ec6260" + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/review-coverage.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/review-coverage.json new file mode 100644 index 0000000..ed27f38 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/review-coverage.json @@ -0,0 +1,24 @@ +{ + "tool_starts": 8, + "tool_ends": 8, + "native_session_tool_calls": 8, + "controller_collected_tool_calls": 8, + "tool_arguments_match_across_three_records": true, + "native_message_records": 17, + "messages_after_agent_end_or_at_verifier_start": 0, + "observed_prohibited_retrieval_attempts": 0, + "observed_successful_reference_retrievals": 0, + "observed_verifier_interference": 0, + "candidate_scan_findings": 0, + "tool_call_ids": [ + "call_00_ET_FgSrm6ZVhvgFxKoDiUGT1316", + "call_01_ET_HRo5xiEtsQEeAshLMqSI0751", + "call_00_ET_MLZk7bu0RKMfD9zmzdWF0974", + "call_01_ET_wRCY0ozJkp4bzDx92qsQ9904", + "call_00_eVoKgCtiCUHqqbN0q83c3178", + "call_00_3vL6Yr3U0eFqTlLv26jP8557", + "call_00_VvKO19I4YoRFpJPf9uMr1020", + "call_00_Mu2D4gzCQPJDojGYqvMi8549" + ], + "coverage_is_exhaustive": false +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/reviewed-audit.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/reviewed-audit.json new file mode 100644 index 0000000..b8ff350 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/reviewed-audit.json @@ -0,0 +1,47 @@ +{ + "schema_version": 1, + "freeze_sha256": "f3558c60b13586c117534c8e753d1ebd5f7ee29aaf6b295b6520b0a81855ea5b", + "review_sha256": "52f2b316f308941e173684ac3171f5a22383b5caa1b66021665cd2d11f38aff8", + "scan_is_exhaustive": false, + "note": "Candidate flags require review. No matches do not prove absence of reference exposure, tampering, or training contamination.", + "summary": { + "eligible_trials": 1, + "pending_review": 0, + "integrity_violations": 0, + "unscored_or_errors": 0, + "effective_pass_rate": 1.0 + }, + "trials": [ + { + "trial": "db-wal-recovery__uagkCMG", + "task": "terminal-bench/db-wal-recovery", + "native_rewards": { + "reward": 1.0 + }, + "native_exception_type": null, + "confirmed_timeout": false, + "enforcement": "restricted-v1", + "verifier_isolation": "shared", + "classification": "no_observed_violation", + "eligible": true, + "effective_score": 1.0, + "review": { + "verdict": "clear", + "reviewer": "Codex manual evidence review", + "reason": "Reviewed all eight tool calls and their results, matched native session and controller-collected tool calls, inspected pre/post network rules, enforcement, egress diagnostics, supervision, and official verifier output. Actions inspect and repair only supplied local database/WAL inputs and write recovered.json. No observed reference retrieval, attempted prohibited web access, hidden-test access, verifier interference, or retained watcher. The two sidecar bypass diagnostics correspond to controller negative-control probes before model work, not agent bypasses. Only api.deepseek.com is allowlisted. No observed violation is not proof of complete observation or absence of training contamination.", + "evidence": [ + "db-wal-recovery__uagkCMG/agent/pi.txt", + "db-wal-recovery__uagkCMG/agent/pi/sessions/2026-11-04T15-22-15-880Z_01a1a727-57c8-7109-9b30-3424a8875e32.jsonl", + ".integrity/db-wal-recovery__uagkCMG/live-output.jsonl", + ".integrity/db-wal-recovery__uagkCMG/enforcement.json", + ".integrity/db-wal-recovery__uagkCMG/network-before-agent.json", + ".integrity/db-wal-recovery__uagkCMG/network-after-agent.json", + ".integrity/db-wal-recovery__uagkCMG/egress-log.json", + "db-wal-recovery__uagkCMG/agent/process-supervision.json", + "db-wal-recovery__uagkCMG/verifier/test-stdout.txt" + ] + } + } + ], + "findings": [] +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/scheduling-amendment.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/scheduling-amendment.json new file mode 100644 index 0000000..ac2a90b --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/scheduling-amendment.json @@ -0,0 +1,14 @@ +{ + "registered_at": "2026-11-04T15:15:05.043947+00:00", + "supersedes_model_queue": "experiment.json", + "reason": "Pipeline cold dependency downloads are delaying the already-passed WAL installation. Permit independent task release after its own complete preflight.", + "model_calls_before_amendment": 0, + "total_authorized_model_attempts": 2, + "task_configs_unchanged": true, + "integrity_policy_unchanged": true, + "deadlines_unchanged": true, + "wal_plan": "experiment-wal.json", + "wal_plan_sha256": "c6afe37cbe47ec07bcb15bb24b93a76fa848d9632dec4572338e48047507999d", + "pipeline_policy": "Only release after a successful pipeline installation and integrity preflight; preserve failed checks and do not widen networking.", + "max_concurrent_task_containers": 2 +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/torch-pipeline-parallelism.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/torch-pipeline-parallelism.json new file mode 100644 index 0000000..067cd01 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/torch-pipeline-parallelism.json @@ -0,0 +1,43 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "retry": { + "max_retries": 0 + }, + "agents": [ + { + "name": "harbor_adapter:PiCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9", + "model_config_path": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-restricted-dev2-20261104/pi-models.json", + "thinking": "high" + } + } + ], + "environment": { + "type": "docker" + }, + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + } + ], + "verifier": { + "import_path": "harbor_adapter:RetryingVerifier", + "kwargs": { + "max_attempts": 3, + "require_agent_quiescence": true + } + }, + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/validation-evidence.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/validation-evidence.json new file mode 100644 index 0000000..d03845a --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/validation-evidence.json @@ -0,0 +1,63 @@ +{ + "validated_at": "2026-11-04T15:30:57.572485+00:00", + "registered_plans_validated": [ + "experiment.json", + "experiment-wal.json" + ], + "accepted_wal_preflight_validated": true, + "registered_task_hashes_unchanged": true, + "source_code_unchanged": true, + "frozen_outputs": [ + { + "manifest": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/preflight-freeze.json", + "sha256": "0e55b2b9144a8348dbc4ba502ee8f14b6fe91bf5d77430b9747cbf514bafa72b", + "files": 21, + "unchanged": true + }, + { + "manifest": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/wal-freeze.json", + "sha256": "f3558c60b13586c117534c8e753d1ebd5f7ee29aaf6b295b6520b0a81855ea5b", + "files": 28, + "unchanged": true + } + ], + "historical_files_checked": 326, + "historical_files_unchanged": true, + "targeted_adapter_tests": { + "passed": 94, + "failed": 0 + }, + "bilingual_reports": { + "present": true, + "full_translation_reviewed": true, + "inline_identifiers_match": true, + "evidence_links_match": true, + "hashes_match": true, + "numeric_results_and_timestamps_match": true, + "files": { + "benchmarks/harbor/reports/en/pi-restricted-dev2-20261104.md": "86a80374b014d00c01706742ea0cfaed68f4d122928c647a64797115ff928e6b", + "benchmarks/harbor/reports/zh-CN/pi-restricted-dev2-20261104.md": "5d51527c5f476e70fd7a5d92295472af5569df8605108e238c6a86c45026b965" + } + }, + "credential_scan": { + "exact_key_matches_in_49_frozen_files": 0, + "exact_key_matches_in_public_artifacts": 0, + "key_or_fingerprint_published": false + }, + "results_sha256": "a58c7aa739f8d576e5f6e69b75c68a46f17daca28889ae001427a362442a5046", + "docker": { + "validated_at": "2026-11-04T15:31:37.359716+00:00", + "initial_image_count": 7, + "initial_image_ids": [ + "sha256:0e33ea5ec823975d1bd6c3778395c9f94251dd88f571146057bff6adb7e4594e", + "sha256:29783439f529eaed2145592f15af7a2281161528860392a382c825489030ae3a", + "sha256:6c53d2f716338192a2020d6f45895089e4a86b532bb6c13502045192aae1e231", + "sha256:7c2bb52851cef25bdc3939e473496d25fbf11fd0cb7ab58bcf3cefcb751cdc0e", + "sha256:8b32782078ff7383a1b4e5d3cecca8ce287e30f50bb4c0e5db009a18064e666e", + "sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "sha256:cac325252991f823713b2d0441502972901dd782bd67f66c03d9b1e410dac5c0" + ], + "all_initial_images_preserved": true, + "running_containers": [] + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-dev2-20261104/wal-freeze.json b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/wal-freeze.json new file mode 100644 index 0000000..6651010 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-dev2-20261104/wal-freeze.json @@ -0,0 +1,35 @@ +{ + "schema_version": 1, + "job_dir": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-dev2-db-wal-recovery-20261104", + "frozen_at": "2026-11-04T15:23:47.400493+00:00", + "files": { + ".integrity/db-wal-recovery__uagkCMG/egress-log.json": "0a3904de9ce9d3a6251a24095a7a86f046e2ca6ef7ce5cce39919d417cc7db08", + ".integrity/db-wal-recovery__uagkCMG/enforcement.json": "0bdd310a5ba7e0175a49119c5aec8795f938d49452f0f10b3d6796239f9ee4ff", + ".integrity/db-wal-recovery__uagkCMG/live-output.jsonl": "2951b9af899febc81715a02c751c3e736e2ea7d1be00a2b9a7b2db0508f76b45", + ".integrity/db-wal-recovery__uagkCMG/network-after-agent.json": "6cdb1220eec019f01d9e3f1b8854a6a14c6c48e054739f1140dcf0a293a0aaa0", + ".integrity/db-wal-recovery__uagkCMG/network-before-agent.json": "6cdb1220eec019f01d9e3f1b8854a6a14c6c48e054739f1140dcf0a293a0aaa0", + ".integrity/db-wal-recovery__uagkCMG/network-preflight.json": "6cdb1220eec019f01d9e3f1b8854a6a14c6c48e054739f1140dcf0a293a0aaa0", + "config.json": "dbf4e14201c7fa002ebbb05069060ad841ee39f7dd12efedbb51e49ca2e012c2", + "db-wal-recovery__uagkCMG/agent/bootstrap.json": "deb2178e8d6b34ffdb8400a01284bc147791c036ab0ae2bcf8ece2c9a4643a01", + "db-wal-recovery__uagkCMG/agent/pi/sessions/2026-11-04T15-22-15-880Z_01a1a727-57c8-7109-9b30-3424a8875e32.jsonl": "c9b45294fa3649ea83b65d9e96277ab603731321646a359df898661d175e5d86", + "db-wal-recovery__uagkCMG/agent/pi-stderr.txt": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "db-wal-recovery__uagkCMG/agent/pi.txt": "7c4b89de976b581c62e98d3c2d82488ec69db76d218e098c0cf050f802f8628d", + "db-wal-recovery__uagkCMG/agent/process-supervision-required": "9e67a1538c0f96556945933b784624bed4b3f7d177108e2ee1af88d6ea1314c8", + "db-wal-recovery__uagkCMG/agent/process-supervision.json": "c86e08060391f52429e8b442add8a03983fa0caf8f19eb62ad580208cc0bd7af", + "db-wal-recovery__uagkCMG/artifacts/manifest.json": "8d0a02e9c17d62af6176024f463c041ed30ff29d3a7b66dadbeab1d762b0cbd9", + "db-wal-recovery__uagkCMG/config.json": "d6f1338d0d92f6cb24df2544c3be8b3cc3144c08645038b91b3bf8f5f7eed1fc", + "db-wal-recovery__uagkCMG/lock.json": "019240ab2d05be70756ae6567688f26e4431a982c7a0f16d51a403422f1b7bff", + "db-wal-recovery__uagkCMG/result.json": "ea962fa5683951d2537d7bcb96b3af3b26e779afad30c9b8e07c72d4a9a7cf2a", + "db-wal-recovery__uagkCMG/trial.log": "ca02bc32327d509b96582c4d08e837f9cf5cd1fbea407c8bd31c17db45e6b451", + "db-wal-recovery__uagkCMG/verifier/attempts/1/ctrf.json": "232cc216708c873bcc3959b88ba54862178032436cf8eee4c5ee9dcf0bd4a253", + "db-wal-recovery__uagkCMG/verifier/attempts/1/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "db-wal-recovery__uagkCMG/verifier/attempts/1/test-stdout.txt": "3662ed4c22f6debeda504e5d4d104963e3a7acee1f8b7a4c514246bba7f8e306", + "db-wal-recovery__uagkCMG/verifier/ctrf.json": "232cc216708c873bcc3959b88ba54862178032436cf8eee4c5ee9dcf0bd4a253", + "db-wal-recovery__uagkCMG/verifier/retry-summary.json": "789dd1216f1363dceb98bef576f48e48ca27299f5b2d45a5d949dbd8605a5c12", + "db-wal-recovery__uagkCMG/verifier/reward.txt": "4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865", + "db-wal-recovery__uagkCMG/verifier/test-stdout.txt": "3662ed4c22f6debeda504e5d4d104963e3a7acee1f8b7a4c514246bba7f8e306", + "job.log": "ca02bc32327d509b96582c4d08e837f9cf5cd1fbea407c8bd31c17db45e6b451", + "lock.json": "8b742f4979678363a8c3ee4bd4dcb323dd41c10178d41f80723e90d453ba2001", + "result.json": "f9515ce4667fcdb096c99d0f0921ba4298b8c66f9eff3317f2628c53add94af5" + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/experiment.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/experiment.json new file mode 100644 index 0000000..76a2a71 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/experiment.json @@ -0,0 +1,115 @@ +{ + "schema_version": 1, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "minimum_free_disk_bytes": 16106127360, + "arms": { + "pi": { + "agent": "pi", + "source_root": ".", + "source_ref": "8e42c62e36be152f9b84da2fd16d1b95fe54869a", + "distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9" + } + }, + "batch_images": { + "manifest": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/image-digests.json", + "storage_path": "/var/lib/docker", + "minimum_free_disk_bytes": 16106127360, + "cache_dir": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/nanopy-harbor/images" + }, + "model_policy": { + "model": "deepseek/deepseek-flash", + "api": "openai-completions", + "base_url": "https://api.deepseek.com", + "thinking": "high", + "model_config": "pi-models.json", + "model_metadata_max_tokens": 384000, + "pricing": "Peak-rate estimates from the registered Pi model config; not billing reconciliation. Official pricing also has off-peak discounts.", + "alias": "Reuse the original mutable deepseek-flash alias; no identical backend checkpoint claim across dates." + }, + "budget_policy": "Original task-native agent timeouts, original resources, and 1800-second setup. Stock Pi loop and generation behavior: no injected 100-turn cap or 180-second finalization reminder. Model metadata maxTokens=384000 is preserved from the local provider configuration. This differs from the historical Nano baseline (100 replies, 65536 requested output tokens, native timeout minus 180 seconds for work), so this is an agent baseline rather than a matched-budget causal comparison. Temperature unset.", + "verifier_policy": "Original official tests and native per-attempt timeouts. Only verifier timeouts retry, at most three total attempts on the same output. No dependency-error recovery or whole-trial retry. Preserve setup errors and verifier errors separately from scored failures.", + "runtime_environment": { + "UV_PROJECT_ENVIRONMENT": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv", + "UV_CACHE_DIR": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/uv" + }, + "registered_at": "2026-11-05T15:07:18.767681+00:00", + "experiment": "pi-restricted-pipeline-recovery-20261105", + "total_model_runs": 1, + "log_dir": "jobs/pi-restricted-pipeline-recovery-20261105-orchestration", + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb", + "native_agent_timeout_seconds": 900, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-pipeline-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "torch-pipeline-parallelism-v1", + "task_file_sha256": { + "README.md": "297ddea07c230dfd246a8ab07337dd7abcd8e5e2a2b2e26590cb0a8af779839c", + "environment/Dockerfile": "c357ea41e4a4013c5c5980b8fef04af6a5ae250ccc6697cf3bad3127401a931f", + "instruction.md": "989bfcf0c54baf6c8cdc391743dbb2f9e858c5779dd85112536875d4d1b2dcd2", + "solution/solve.sh": "787198e4178d88b887697aefdd29819f29435ffdfe356545d92f58a30fb38b70", + "task.toml": "8b4f9f525c08afbbc4eca9168e46238a61ccb903486773e9ca4da64bb7ffe857", + "tests/test.sh": "cc9fd28b6590993fc37e4a99da771cbf04726c8c62fe308e802612241da69dea", + "tests/test_outputs.py": "23574246a407163a8e6581c84287cc2f1eb745efbaad2a317a5a28c39958c64f" + }, + "prior_nanopy_work_budget_seconds": 720, + "pi_agent_timeout_seconds": 900 + } + ], + "preflight": [ + { + "arm": "pi", + "config": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/install-pipeline.json", + "config_sha256": "e70725f14055d4f88a1972a3b21606eded3a1477b6703ae691685ccbe1a62899", + "job_name": "pi-restricted-pipeline-install-20261105" + } + ], + "queue": [ + { + "arm": "pi", + "config": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/torch-pipeline-parallelism.json", + "config_sha256": "e70725f14055d4f88a1972a3b21606eded3a1477b6703ae691685ccbe1a62899", + "job_name": "pi-restricted-pipeline-model-20261105", + "task": "terminal-bench/torch-pipeline-parallelism", + "repeat": 1 + } + ], + "additional_file_sha256": { + "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/image-digests.json": "ef97fd9ad5b4e46fd828661b235e5284f3baaa9e93b0b5ea67bbbbc49aadc93f", + "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/install-pipeline.json": "e70725f14055d4f88a1972a3b21606eded3a1477b6703ae691685ccbe1a62899", + "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/pi-distribution.json": "0e1e2cdbef4ede58b2beed0f4c3655099fb2d6f7e3f6d2ca369879fa24546eb1", + "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/pi-models.json": "8751fac554abb9f363b188c102045f38e36693d52956f984e0e427b31998b299", + "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/torch-pipeline-parallelism.json": "e70725f14055d4f88a1972a3b21606eded3a1477b6703ae691685ccbe1a62899", + "benchmarks/harbor/scripts/run_comparison.py": "8c44a0dff55605495436ca04c6d16ff0c6aa540e7b8e40ebbdaece54181a7c59", + "benchmarks/harbor/scripts/cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c", + "benchmarks/harbor/scripts/pypi_wheel_cache.py": "ced6bd7c1bf2b6ac1964c39d057708ba0e52c37c61a609651c49105ea145ea74" + }, + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "preflight_policy": "Fresh install-only dependency and network controls must pass before model work. Use the reviewed torch-pipeline-parallelism-v1 profile and pypi-verified-v1 transport. Keep original 1800-second setup, 900-second model, and 900-second per-verifier-attempt limits. Fresh formal job repeats setup with the same cache. Preserve failed preflight evidence and fail closed; no legacy opt-out.", + "information_policy": "One fresh Pipeline model attempt, after the November 4 installation failure occurred before model work. Use restricted-v1 shared rules and external egress enforcement; api.deepseek.com is the only allowed model-phase host. The managed package transport is absent throughout model work. Native public network is available only during installation and official verification. Freeze completed outputs before trajectory and integrity review. Preserve raw rewards and errors; reviewed violations are effective zeroes, blocked attempts are separate from retrieval, and unreviewed outcomes are pending. Do not pool with historical open-network runs or claim a matched intervention against WAL, whose run did not use this transport.", + "authorization": "Maintainer requested rerunning both previously cheating Pi tasks with restricted networking, then asked why dependency installation timed out and how to resolve it. This recovery adds a reusable verified package transport, preserves the prior failed attempt, and completes the previously unstarted Pipeline model run. WAL is not rerun.", + "original_results": "benchmarks/harbor/results/pi-deepseek-flash-dev3-rerun-20261102.json", + "dependency_cache": { + "protocol": "pypi-verified-v1", + "helper_image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "phase_policy": "Controller-owned loopback helper only during dependency setup and official verification, removed and checked before model work. Every cached wheel must match current PyPI metadata filename, size, and SHA256. Task image, dependency versions, upstream package sources, and official scripts remain unchanged. Helper has a separate 1 CPU and 512 MiB limit and no task filesystem mount. No custom Compose overlay or task-container host mount." + }, + "prior_setup_failure": "jobs/pi-restricted-dev2-install-pair-01-20261104/torch-pipeline-parallelism__aykTgx7/result.json", + "prior_restricted_results": "benchmarks/harbor/results/pi-restricted-dev2-20261104.json" +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/failed-model-setup-evidence.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/failed-model-setup-evidence.json new file mode 100644 index 0000000..f2e2513 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/failed-model-setup-evidence.json @@ -0,0 +1,29 @@ +{ + "trial": "torch-pipeline-parallelism__3gci4oE", + "status": "setup_error", + "classification": "setup_error", + "model_calls": 0, + "agent_execution": null, + "native_exception_type": "NetworkConnectionError", + "native_rewards": null, + "effective_score": null, + "setup_seconds": 370.463495, + "bootstrap_stage": "verifier_dependencies", + "diagnosis": "The unchanged uv 0.9.5 installer received curl exit 18 (Transferred a partial file) from the original GitHub release URL. The wheel cache was available but not reached. No model execution or grading occurred. The cache helper was removed after failure. This failed job is frozen; the next registration adds verified original uv/Python transport, rather than retrying this unchanged job.", + "cache_phase": { + "protocol": "pypi-verified-v1", + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "helper": "nanopy-pypi-e5814671e0f84dd68c83871fe8f6ae7f", + "scripts_sha256": { + "pypi_wheel_cache.py": "ced6bd7c1bf2b6ac1964c39d057708ba0e52c37c61a609651c49105ea145ea74", + "cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c" + }, + "phase": "setup", + "started_at": "2026-11-05T15:13:44.721761+00:00", + "removed": true, + "finished_at": "2026-11-05T15:18:10.071152+00:00" + }, + "partial_setup_log_sha256": "fac852b7c6350ec9d1a0aa456ce44270347c8149a896a064744b4298ceed97ac", + "result_sha256": "b334788807367ac50672bbbad8f56ac441c7d89ffe8ab01c380f034c0fa127ba", + "freeze_sha256": "ee739889bfee647d5cb702201c67ec72653d2cf184938385a4084f0fce0b2774" +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/failed-model-setup-freeze.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/failed-model-setup-freeze.json new file mode 100644 index 0000000..f214a21 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/failed-model-setup-freeze.json @@ -0,0 +1,23 @@ +{ + "schema_version": 1, + "job_dir": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-pipeline-model-20261105", + "frozen_at": "2026-11-05T15:18:25.491202+00:00", + "files": { + ".integrity/torch-pipeline-parallelism__3gci4oE/dependency-cache-setup-log.json": "44f44b217449eeabeec0d9fda1147d724209cdd05c2ea5ec41979b34bb759d5e", + ".integrity/torch-pipeline-parallelism__3gci4oE/dependency-cache-setup.json": "513edebba712b3cfa3227e299845e283d1220d42f298f582ebc81a6779a104f4", + ".integrity/torch-pipeline-parallelism__3gci4oE/dependency-setup-output.jsonl": "fac852b7c6350ec9d1a0aa456ce44270347c8149a896a064744b4298ceed97ac", + ".integrity/torch-pipeline-parallelism__3gci4oE/enforcement.json": "cae2ce2a199e08f41bf27f53e3cd6bf3fc3ba6d758ed0e662daeac4af49084f9", + "config.json": "ff8e7a9b469535327d927b409fad684fe7c82f10e29d706aeb828de1a1b3ae9a", + "job.log": "6cfca32d5083de8c61ee0a326b21599f01b70cce2eaeec633ebee1b9f9f496d2", + "lock.json": "a63ea52709436280c906ee7eff096dc6337fb6032fe6c16f2557160800bebc4d", + "result.json": "52712032d7da920c2c24d02e269738a1baa4893cb5ee0afd19de8b21f1cb9a32", + "torch-pipeline-parallelism__3gci4oE/agent/bootstrap.json": "2404c0ab93860318b7b5c4533f203915de6b0e1f2fef097f5909d3e6f86d61eb", + "torch-pipeline-parallelism__3gci4oE/agent/process-supervision-required": "94e95f0bb38f8e0fb52a6dab21f7708dde104b7edeb2eb5ce6b7e68f1328c295", + "torch-pipeline-parallelism__3gci4oE/artifacts/manifest.json": "8d0a02e9c17d62af6176024f463c041ed30ff29d3a7b66dadbeab1d762b0cbd9", + "torch-pipeline-parallelism__3gci4oE/config.json": "7b18a96829c8e9c1e7ef925fbd3ae706d6a8a5142e3d172bdfc6aa6fef37d633", + "torch-pipeline-parallelism__3gci4oE/exception.txt": "18a58679ebf7942110c9961b449a089334cf73168fa21f0b3ec45cd7a0f26470", + "torch-pipeline-parallelism__3gci4oE/lock.json": "27e3e71aba71c529cce6d55388ebc850eace41b91b475d8b8831fc65d9acdac2", + "torch-pipeline-parallelism__3gci4oE/result.json": "b334788807367ac50672bbbad8f56ac441c7d89ffe8ab01c380f034c0fa127ba", + "torch-pipeline-parallelism__3gci4oE/trial.log": "8a9fb30ea2ebfb394b0516db25085d4c133608715a62ff41134b73b4a38785bf" + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/image-digests.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/image-digests.json new file mode 100644 index 0000000..e1855e4 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/image-digests.json @@ -0,0 +1,15 @@ +{ + "observed_at": "2026-10-24T14:07:09.186521+00:00", + "images": [ + { + "task": "terminal-bench/torch-pipeline-parallelism", + "image": "alexgshaw/torch-pipeline-parallelism:20251031", + "image_id": "sha256:3cb7b39d86f1704b20e668851da0b08b34b3270c01d583f136d65190984a8d1b", + "repo_digests": [ + "alexgshaw/torch-pipeline-parallelism@sha256:3cb7b39d86f1704b20e668851da0b08b34b3270c01d583f136d65190984a8d1b" + ], + "size_bytes": 29725531 + } + ], + "matches_prior_registration": true +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/install-pipeline.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/install-pipeline.json new file mode 100644 index 0000000..f2f1c92 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/install-pipeline.json @@ -0,0 +1,44 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "retry": { + "max_retries": 0 + }, + "agents": [ + { + "name": "harbor_adapter:PiCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9", + "model_config_path": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/pi-models.json", + "thinking": "high" + } + } + ], + "environment": { + "type": "docker" + }, + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + } + ], + "verifier": { + "import_path": "harbor_adapter:RetryingVerifier", + "kwargs": { + "max_attempts": 3, + "require_agent_quiescence": true + } + }, + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "dependency_cache": "pypi-verified-v1" +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/pi-distribution.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/pi-distribution.json new file mode 100644 index 0000000..bc28496 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/pi-distribution.json @@ -0,0 +1,852 @@ +{ + "version": "1.4.0", + "source": "Existing local upstream binary distribution; no rebuild or source modification. Its registered file hashes pin the local bytes; an upstream archive checksum has not been independently verified.", + "original_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9", + "files": { + "CHANGELOG.md": { + "sha256": "b0eac55f99b20254f81cf925a52f054ec87c1320454a3fd7ee624683e585add6", + "mode": 420 + }, + "README.md": { + "sha256": "fbc7abec7d0829ac8bc4b217e950794e1ba4d8d1b3e0850f970da318ad72e863", + "mode": 420 + }, + "assets/clankolas.png": { + "sha256": "169acd0dfe6fbb8d8742ed24a3fc654fd0b2e2d4223c733249c5493723f1b72d", + "mode": 420 + }, + "docs/cli-integration.md": { + "sha256": "edf8e723a70b4c3c40275db0391219e55d30597936212a9cfdef33963dd37f93", + "mode": 420 + }, + "docs/cli.md": { + "sha256": "c45e59f31574de71770a71c9a7c9b6ac03bf9cad4928aa31bb6614d661733679", + "mode": 420 + }, + "docs/codemode.md": { + "sha256": "3a96ee10da36395bbd22beb60ac029f8ca8a859f2fd3c3104cfc38f9d5a1f95a", + "mode": 420 + }, + "docs/compaction.md": { + "sha256": "b5b23d10bb86d8ab32b67f80a7e57febd953222c6da6fd422ae91d232ef28faf", + "mode": 420 + }, + "docs/configuration.md": { + "sha256": "fc48d1ce7d885c5253222415fb7a4b46d75f80e6431eb924174657ff775099eb", + "mode": 420 + }, + "docs/containerization.md": { + "sha256": "e309fcb9a0a7592c0dba000ef8ee90f2d7ef1e01e6a1ac2f0cd9760dd63dee0c", + "mode": 420 + }, + "docs/custom-provider.md": { + "sha256": "fe926ed6194756c85ab6b56b7bfd12e81abe385acf39e0bb2a470eb6c70b8c59", + "mode": 420 + }, + "docs/docs.json": { + "sha256": "fc0bccd6bf66716ad1067d4fbf7258f674d89c89a12cb0b2587a027b9d7fdc49", + "mode": 420 + }, + "docs/environment-variables.md": { + "sha256": "097f2101e285f51f509be0b176d458825b83b929a635ae29d2a43703168170dd", + "mode": 420 + }, + "docs/extensions.md": { + "sha256": "46fc0baeb00db8ca8db399e33e0f506e86e425693b291258357e8063e2f3353c", + "mode": 420 + }, + "docs/how-pi-works.md": { + "sha256": "7d43ab8e2b886cd1047dcee503588a1c828b92333dd4f177526b89e6db8aaa8b", + "mode": 420 + }, + "docs/images/doom-extension.png": { + "sha256": "c3b89b5e1fba711a5bcfa188e13d8695a84f7cdda12d9e127bb45b36d13cbb27", + "mode": 420 + }, + "docs/images/exy.png": { + "sha256": "a8f253347440aed01ff659f41f573d340bcfd6010d829726ae92fe18abf601de", + "mode": 420 + }, + "docs/images/interactive-mode.png": { + "sha256": "4078001e39978151bd66b1d0a20123e7ca2ec2d1b572dfd70f552be5e88c0692", + "mode": 420 + }, + "docs/images/tree-view.png": { + "sha256": "ca61e7bbf300dfb3840cc191dc66dc4e9f5166b2f1f4089e6fb13fd04a2c3c66", + "mode": 420 + }, + "docs/index.md": { + "sha256": "c107426e88eac570e9ecbea325cb45c2a6b778a324fd6e424bd032380a65614d", + "mode": 420 + }, + "docs/json.md": { + "sha256": "f103191816fdcf3627dfc98b89acad8a28eee737697130ddaeb2581131213543", + "mode": 420 + }, + "docs/keybindings.md": { + "sha256": "413aa1c5e95a934d216aa5aba20052fa2cf35940ba5f02d1f4b22ff53c6ec678", + "mode": 420 + }, + "docs/llama-cpp.md": { + "sha256": "dccba80951463657c57a831aa2da3a576b92c05455f9b73610b208c942dfc8f4", + "mode": 420 + }, + "docs/mcp.md": { + "sha256": "23b9cca35d1340e8adce6bed8b495da0ebb58931179a937db87d610a1b40b9bf", + "mode": 420 + }, + "docs/message-types.md": { + "sha256": "b832893fb762f87fdbbfd694241d484592942823791f3e81a6c4d915421ff5db", + "mode": 420 + }, + "docs/models-configuration.md": { + "sha256": "1eb60b0f1faad6f9fffd62ea257b545dbc983dbe4c5decc4803b1a241d604155", + "mode": 420 + }, + "docs/models.md": { + "sha256": "c9914f5411db5ddd158b5391ec1da76c1f65788cfaea3a90e1e1555451b3d12e", + "mode": 420 + }, + "docs/nvidia-rtx-llama-cpp.md": { + "sha256": "bcd12ebfd283d8f528cf206ee4ec74f981ab36d3604419acc434c595decbebdb", + "mode": 420 + }, + "docs/packages.md": { + "sha256": "5fe4a3009d7260fe1e2f54b64086fc3c8b4d678232bfa2dbc7e91fe588fc9ea5", + "mode": 420 + }, + "docs/prompt-templates.md": { + "sha256": "ad5113cce0d05ef0ac77b3c71f9ebe41ea68c072275f9273b9616e4924feda42", + "mode": 420 + }, + "docs/providers.md": { + "sha256": "bc8798e4cc3209e455e3dc50c6652ae73c057d40122a4a741d8b5b65a041c2f1", + "mode": 420 + }, + "docs/quickstart.md": { + "sha256": "10c69b394304f6e7d6409b6bde02b0165f5723ff4acfd49cfa8f7414f977266a", + "mode": 420 + }, + "docs/rpc-commands.md": { + "sha256": "4deeee367852f4633ef594bc079f9e9fb14640a4e3ba8a69b7c89b6f54740838", + "mode": 420 + }, + "docs/rpc-extension-ui.md": { + "sha256": "3c6f2c6edfc8b1ace6642343727aef97a7e9234c85509850921b52d6ff09a48e", + "mode": 420 + }, + "docs/rpc.md": { + "sha256": "37a03d180115460a0a275022f4aeabef1e5345458df6a9b6e8a6de18474de720", + "mode": 420 + }, + "docs/sdk.md": { + "sha256": "d9034ccbf2219a58712d8dd4fcdf47e687d77131e71dd5d119d8a71e9bd2a2e8", + "mode": 420 + }, + "docs/security.md": { + "sha256": "3b1e40a8d28b17c9eb7e496a7ba88fbe04665e3e6ae02873a58b943e325ad504", + "mode": 420 + }, + "docs/session-format.md": { + "sha256": "4f8fc42179b332bcf71fe4c85cc95d914853ff7486d68742954a45316b512a4a", + "mode": 420 + }, + "docs/sessions.md": { + "sha256": "152d6489f61a6de9349c8ceaade95d5498fbc74f21771f41ba8e87ce0e5d8936", + "mode": 420 + }, + "docs/settings.md": { + "sha256": "2915d26f91f4c46ab1b92e63cbc46b8d48888fa0e491168642a5a90f973bd814", + "mode": 420 + }, + "docs/shell-aliases.md": { + "sha256": "ebb55456870455b4da91cfd054ae2ac9b3a417e6ff48a009587c33e68ecd7807", + "mode": 420 + }, + "docs/skills.md": { + "sha256": "5b9805525b194474ef1fe5455a0917ef15ed7f710aa9bec2eee11879556469e5", + "mode": 420 + }, + "docs/slash-commands.md": { + "sha256": "d7e1db36e9186ff146a6c27de31fe169671f52cb6f7b93e1bf8d7fe76209a79c", + "mode": 420 + }, + "docs/terminal-setup.md": { + "sha256": "583381fb9dc466135a865ca406a3be299a7a8da2c9d810c5b73f401912a0a357", + "mode": 420 + }, + "docs/termux.md": { + "sha256": "1b0687157f2384385d07800ec23e6c225ef28fbf6648f55453cc7ce62f4c64ea", + "mode": 420 + }, + "docs/themes.md": { + "sha256": "6f11eb43ca6dad4c71bcfa2fc72ac13c013e3cb9383688b88923e0139b2d097c", + "mode": 420 + }, + "docs/tmux.md": { + "sha256": "9ec0abdbf74c57d51d4a93f308fb4aec4d8a3574a24cbfd9ba481d2e39f1273c", + "mode": 420 + }, + "docs/tui.md": { + "sha256": "20d3695ebd11e179a194f535f73870247544f655ebf22e42726c3be559c6d786", + "mode": 420 + }, + "docs/usage.md": { + "sha256": "6dfe7446c81d601d5ebef5769523115229f802da364fe90fcdf96e322ae9fbcd", + "mode": 420 + }, + "docs/virtual-models.md": { + "sha256": "5ab04dee3b3712b317522bb0784e28a3fef2e6b7fe1aeb1c1ca27778685a38f7", + "mode": 420 + }, + "docs/windows.md": { + "sha256": "e5a51792381a9eec859f034153c67f9ff52c36d3cff4c7a1faa0651f77f627c3", + "mode": 420 + }, + "examples/README.md": { + "sha256": "294836cfa47cd2cff1cab628533aefc5fe2d888621e43fe9f4101c09c2917a98", + "mode": 420 + }, + "examples/extensions/README.md": { + "sha256": "b51e1095d338ed8bc6cb0a40dca12da66ffd12d6f4ec7bfd57815fac84d87662", + "mode": 420 + }, + "examples/extensions/auto-commit-on-exit.ts": { + "sha256": "b89b5cb4c6f1f07b0b4291ab85a3c5f025d7d31ebb285821716502452b46af62", + "mode": 420 + }, + "examples/extensions/bash-spawn-hook.ts": { + "sha256": "c79c87653e20ac6bd9f4813aa4bb01a483b3637700fa2edda220703958532662", + "mode": 420 + }, + "examples/extensions/bookmark.ts": { + "sha256": "a71d9449d415ad75c9f3d0d13afde129b9bea2d695b889566db17ecfca256d79", + "mode": 420 + }, + "examples/extensions/border-status-editor.ts": { + "sha256": "ccbbfddd4ff402432293b7adb21f58e869f37a5d12d4da81cea10cc5fc2311cd", + "mode": 420 + }, + "examples/extensions/built-in-tool-renderer.ts": { + "sha256": "4bfd0953e3dbf96ee0cdb26989a60ec91d4850323852ef09eafe5c2efdf23c00", + "mode": 420 + }, + "examples/extensions/claude-rules.ts": { + "sha256": "8a935f14f62c1c1a3e937f5c912c2215e25aa5fa591716e23c86224ce1dc34cd", + "mode": 420 + }, + "examples/extensions/commands.ts": { + "sha256": "36716b53da169936c7e1360a4fde1e2fc0c3356a6505f177235f09c5538c6e4f", + "mode": 420 + }, + "examples/extensions/confirm-destructive.ts": { + "sha256": "55502f82109df07e922ef3ec41af1b1a5ed33d4d6d56076654ab2bc88e2e0888", + "mode": 420 + }, + "examples/extensions/custom-compaction.ts": { + "sha256": "ea797b3508fd8600526f381466e8b08bb84f3d234932a9d6ce512698527d0aa9", + "mode": 420 + }, + "examples/extensions/custom-footer.ts": { + "sha256": "1e5f17bec95a48d407fbcc092faf7e1f6021c30cee4a66ec09bc120412292cb2", + "mode": 420 + }, + "examples/extensions/custom-header.ts": { + "sha256": "5de09cf861d3965201991d6e837d6b68e8a2e80cb89279ee9177cc05d31f723d", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/index.ts": { + "sha256": "52bb733e1a25d9e175a644e6a6e17bbf61071beecf0f2903869baa9579a05ab3", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/package-lock.json": { + "sha256": "4c1ccae817de13a0d0eee610fb12ee699b3c1f97cf28cbe83ec50cab0f338a77", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/package.json": { + "sha256": "b2ceede9b12cef79f37d02c8e850132afdab1b3316c16d622c0c8343244e85c3", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/index.ts": { + "sha256": "0c4adac82eff25fef23223af77e11cd53535834fece9731009b125446e213158", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/package.json": { + "sha256": "4af767a958f402efc19fc6157c75c209edc45c8ab6103b60f13973369ba41990", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/test.ts": { + "sha256": "58dfa6fdb63521ccad551b7463ff56d4aecf735074934a8e9fc34b71e7083d00", + "mode": 420 + }, + "examples/extensions/debug-provider.ts": { + "sha256": "a915f86a26a0257537939701797b1f613072275e76bdd8a27dd941bbf519643a", + "mode": 420 + }, + "examples/extensions/dirty-repo-guard.ts": { + "sha256": "0037a69abdf4cf98181db799bef5bdee8981861315eb82ddab78126c2b0f13c4", + "mode": 420 + }, + "examples/extensions/doom-overlay/.gitignore": { + "sha256": "b26aa26693c610d46d9c8b387b28d20dc5196a73b979f5522e24dd689aab7bc8", + "mode": 420 + }, + "examples/extensions/doom-overlay/README.md": { + "sha256": "e17bc73b76c7977ffeef99120c7896ff992cf5938a60c4d7427a7b19fc180d97", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom/build/doom.js": { + "sha256": "c3147cf4881c88b4210c017c26828f52aff83f0c4db27d7dad3fe6cda9ade1fb", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom/build/doom.wasm": { + "sha256": "571d161956593508cf4ade732ae93753f00484bb526667a8676571cca14dec7d", + "mode": 493 + }, + "examples/extensions/doom-overlay/doom/build.sh": { + "sha256": "ab09c9b6dc98ea7c1fa8e74c13981ce69b698c01aec77309c5c7d40ad114d38c", + "mode": 493 + }, + "examples/extensions/doom-overlay/doom/doomgeneric_pi.c": { + "sha256": "98210909e1e7dee71cbbdba64f1db4bebd88b504e861ee36ea42093c59b251a1", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom-component.ts": { + "sha256": "d95dc47c92db0fdc5b0e0b56a277c67f277012d3f7296df745d8a93f64d5a42e", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom-engine.ts": { + "sha256": "706e43f70bd374b73928e5a44d196d1dd5c03d93979d08bdfc01d049b5083855", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom-keys.ts": { + "sha256": "e8bd99a4ce90dd44ad120c58f3b2ba7bc0b0c662bcdc4ed4994d059130be9fdd", + "mode": 420 + }, + "examples/extensions/doom-overlay/index.ts": { + "sha256": "f89818dd75c7d8970458b1dbb9c5e85fea9b86534fd740c898c1f6f9c9f9ceba", + "mode": 420 + }, + "examples/extensions/doom-overlay/wad-finder.ts": { + "sha256": "20254888262081bf96932253bfd346971deefa48d8df011fd3baa7aa2d653b7d", + "mode": 420 + }, + "examples/extensions/dynamic-resources/SKILL.md": { + "sha256": "f8f34b0d81aa791a2d56501756fe0ace21fb78ca2754025d093f41d3b0ccaf69", + "mode": 420 + }, + "examples/extensions/dynamic-resources/dynamic.json": { + "sha256": "2bf0164779b4ea3c2572f33b1e979c427a4839193998fda19bfc5cba2a4692c5", + "mode": 420 + }, + "examples/extensions/dynamic-resources/dynamic.md": { + "sha256": "e61e2c971d02d628a58f7c36b9b37bd3e636e4ad60618a7f65e2b41410c38f35", + "mode": 420 + }, + "examples/extensions/dynamic-resources/index.ts": { + "sha256": "dfbc49418fe46ebc9696fa335709df2ed0d4c4703da8c1ad0eb317d11fce8433", + "mode": 420 + }, + "examples/extensions/dynamic-tools.ts": { + "sha256": "4394098c11f98ffc26e2487acda2e09b9467ef9ff5867def7273cb0c82a2b6c0", + "mode": 420 + }, + "examples/extensions/entry-renderer.ts": { + "sha256": "e4c01fcf382972d570ebf37d044e32065eae062fe70987b1b885d75d46aa0b11", + "mode": 420 + }, + "examples/extensions/event-bus.ts": { + "sha256": "1185942b7afd83692d9e8da029e8594d702157398c693ec8b3ddb02b0780828a", + "mode": 420 + }, + "examples/extensions/file-trigger.ts": { + "sha256": "6f5fc3dc8fa387d8ca6973786c521feb94da58838b0e171a881fb6b3a5243ac0", + "mode": 420 + }, + "examples/extensions/git-checkpoint.ts": { + "sha256": "b60c85f797091a1f21ccb2a998027c6229fe995bdd62dbb1cdcc22946768f40b", + "mode": 420 + }, + "examples/extensions/git-merge-and-resolve.ts": { + "sha256": "a3c077c87544122abe83b275f3d527f60c80c87f460ea2499d66bde9b4880133", + "mode": 420 + }, + "examples/extensions/github-issue-autocomplete.ts": { + "sha256": "4646ef360e57aa98c787ba21ca2f2e8acb3db30c4db0503e64cdbea4a4c2ac65", + "mode": 420 + }, + "examples/extensions/gondolin/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/gondolin/index.ts": { + "sha256": "9a9897639de8127a2abe6bb0d00f0005cd9e9cf9d51fd15893a827e67326bf62", + "mode": 420 + }, + "examples/extensions/gondolin/package-lock.json": { + "sha256": "256d3e45b7c6d293bdfd88a3c4884d8bb38ecdbc4064c290255d2b4d193ea535", + "mode": 420 + }, + "examples/extensions/gondolin/package.json": { + "sha256": "010f1a949005f706239d788a2481c355028eb4c366b4ea16c87577e63caab905", + "mode": 420 + }, + "examples/extensions/handoff.ts": { + "sha256": "ea07b6f680357b677d0f43f0016961ee8cb1331d37ae113e9466a3acffb2e381", + "mode": 420 + }, + "examples/extensions/hello.ts": { + "sha256": "0aa4e9800c2526914d4c1edb00b2cfa9bd9dd6da5218289994cccd5f5bfa4934", + "mode": 420 + }, + "examples/extensions/hidden-thinking-label.ts": { + "sha256": "179dc3704b5c563b49a7f4577222bf9bbaa5becf98b85c75e56ab26c001c5b22", + "mode": 420 + }, + "examples/extensions/inline-bash.ts": { + "sha256": "69b9dfc34e7b1ada9afe10cda3384b0b1a6f3c3a47da5fb7d16f914399b023d4", + "mode": 420 + }, + "examples/extensions/input-transform-streaming.ts": { + "sha256": "5c251ebda6163c8279e3c6d5f0aee56421ed150255dbb954f08ca7d2280b3ad2", + "mode": 420 + }, + "examples/extensions/input-transform.ts": { + "sha256": "cf0f65d610631ca75d18aae1c8f1408139702f674cb2835f8c009b943776474b", + "mode": 420 + }, + "examples/extensions/interactive-shell.ts": { + "sha256": "36a7efb5b132e666c643129ce09c3f83f56aed9924563e6b7269e0e126611ecc", + "mode": 420 + }, + "examples/extensions/jev-router.ts": { + "sha256": "cf2c9ef4baf9233faa133e99255786b00518cb4977d068c811a11b3c96f0a2d6", + "mode": 420 + }, + "examples/extensions/mac-system-theme.ts": { + "sha256": "d7349bae999a798c71b283165e06d0142d41b0264bfdad11ddcb1ec8b62f8710", + "mode": 420 + }, + "examples/extensions/message-renderer.ts": { + "sha256": "bedc198df5215c0a758e0743224599003630741a4ff72d4327837ce309e5ec8c", + "mode": 420 + }, + "examples/extensions/minimal-mode.ts": { + "sha256": "040cbc7619b2e903888c8626e2fa55ca5e3f820535dfd1aadcb2060e156a8770", + "mode": 420 + }, + "examples/extensions/modal-editor.ts": { + "sha256": "4742283930949fad1920359540aa9d42c833d2d24eaa49564bfc08f90e328b0d", + "mode": 420 + }, + "examples/extensions/model-status.ts": { + "sha256": "b6c073c0bdf26ac17e2fed01d02b6447fd4e2763b7b6ca2f2b493574ced25247", + "mode": 420 + }, + "examples/extensions/notify.ts": { + "sha256": "70e4333e09ce00d546c116fd2e918abf7616c70b5da6e88ba8ee21a326afd483", + "mode": 420 + }, + "examples/extensions/overlay-qa-tests.ts": { + "sha256": "4b1778e9e247c9eb24fe48e13fff0272f6f78b8102b4ed085d03598c3b98773f", + "mode": 420 + }, + "examples/extensions/overlay-test.ts": { + "sha256": "801b7c2eeec57c3569187b4b4d851f2159223ec6e9772e406ec65ed728ca6db9", + "mode": 420 + }, + "examples/extensions/permission-gate.ts": { + "sha256": "e32086ba9dd9b4ba0fd308b65241d9affa0b5ecd3f90ef7ea1b9f2b30cab8cc1", + "mode": 420 + }, + "examples/extensions/pirate.ts": { + "sha256": "dd6ce684bbe7630e4a749fe133ef8b8b875adbc7e31284b17fe0b8157d51e013", + "mode": 420 + }, + "examples/extensions/plan-mode/README.md": { + "sha256": "1ff8d611b8fbafef4ca1e3dbc72136fbcc166461b69f859b5e961b15147bff39", + "mode": 420 + }, + "examples/extensions/plan-mode/index.ts": { + "sha256": "bef2bf0712aea7b7d0f73d174e24ac5309679e6cc143f7bcbf1a9e0121222af8", + "mode": 420 + }, + "examples/extensions/plan-mode/utils.ts": { + "sha256": "12ea28d57cc55e68e8d52a2f1d16df8b2e7f2749cd694d41453028002a2bdad9", + "mode": 420 + }, + "examples/extensions/preset.ts": { + "sha256": "7942b7f933f97bcc29024a87f8526969413ba6f87b0bec537648a2926a0a4e29", + "mode": 420 + }, + "examples/extensions/project-trust.ts": { + "sha256": "78781c1f2a1613cf31c02a29ff6a6abd86b60bdf34343465e46cd85c01491578", + "mode": 420 + }, + "examples/extensions/prompt-customizer.ts": { + "sha256": "429546f46795bccda4078deace5729b8230f6103807d424eaf71817c6e07da0a", + "mode": 420 + }, + "examples/extensions/protected-paths.ts": { + "sha256": "9ca66b1f3b1b9a61cc5ddc660f92cca2ea66c4264a1bdb6b00e0da5844399fe2", + "mode": 420 + }, + "examples/extensions/provider-payload.ts": { + "sha256": "e0016077798098f528b1129954c30a28809a1513924d9c2e0efda9328d503a49", + "mode": 420 + }, + "examples/extensions/qna.ts": { + "sha256": "a639a65a486fd05a42afcf14aa59dc32e505e41ada43869167ddfd789a3c7da6", + "mode": 420 + }, + "examples/extensions/question.ts": { + "sha256": "a1625d2c72a65d859154a7e66bef222d48ca9e3654a8bb910829ad350378a4c4", + "mode": 420 + }, + "examples/extensions/questionnaire.ts": { + "sha256": "3f1f719d81152ff013f5d36a97cc3976388d43f2d89bff0f19827d3443d903e6", + "mode": 420 + }, + "examples/extensions/rainbow-editor.ts": { + "sha256": "f95f17ba8a0376180bd23510c20151e83aadc74142e989f14227bf5c4196883f", + "mode": 420 + }, + "examples/extensions/reload-runtime.ts": { + "sha256": "911f26b872860a551572fa1d94d251ddd476d4ea9483b6052e9658d2784b38fc", + "mode": 420 + }, + "examples/extensions/rpc-demo.ts": { + "sha256": "3d54a396eee12eb04ea12d0ea03716a6edd3e8743502b6a529e3b8d429ae640d", + "mode": 420 + }, + "examples/extensions/sandbox/.gitignore": { + "sha256": "16d30e4462189fb14dd611bdb708c510630c576a1f35b9383e89a4352da36c97", + "mode": 420 + }, + "examples/extensions/sandbox/index.ts": { + "sha256": "b5e527aac27254002f27b8bb9de7bc62e242e39eeb7c963683096039e0aa173e", + "mode": 420 + }, + "examples/extensions/sandbox/package-lock.json": { + "sha256": "cd4e34dbd8a9d4b0efdb58becb6c4201eb2ad9e44228b9a272b6853bfb54539d", + "mode": 420 + }, + "examples/extensions/sandbox/package.json": { + "sha256": "4b4bfe0ebeecf2af747a3e023a5edc143cba9c36137d10051e1124d1c1ad9732", + "mode": 420 + }, + "examples/extensions/send-user-message.ts": { + "sha256": "9261e433c782f354530105e1c9723a19798868f72fd1f5d68af3a4624ad7e446", + "mode": 420 + }, + "examples/extensions/session-name.ts": { + "sha256": "a0b9fd94e2cc827931150cc409238b473da635ab1a1cddce633cae5937a3ae24", + "mode": 420 + }, + "examples/extensions/shutdown-command.ts": { + "sha256": "68dc7f9561aa9b1c0f39cae19c0f41d56171a99875972e5998de3caa75f96e09", + "mode": 420 + }, + "examples/extensions/snake.ts": { + "sha256": "3ef506ab52927e866fad096f83b7cbfd0eb50e1504dcabf1b1eb28a06745c937", + "mode": 420 + }, + "examples/extensions/space-invaders.ts": { + "sha256": "43281ae25bea645d44db01234f24df80e2f866771d02de0b416ab32fb3883a1f", + "mode": 420 + }, + "examples/extensions/ssh.ts": { + "sha256": "c03bdef919457013e3798d10250f34255742fd3dc79885ac3d321b34f6a1149d", + "mode": 420 + }, + "examples/extensions/status-line.ts": { + "sha256": "26e27b1158e980551f63f46e55dad055ac613779c2b8c39df4cddc11be26103d", + "mode": 420 + }, + "examples/extensions/structured-output.ts": { + "sha256": "72a3a665ce3de1419c837db6eb8a0e7a890310c6e02e4d59c454e5c9514c15ec", + "mode": 420 + }, + "examples/extensions/subagent/README.md": { + "sha256": "6a888330bf974657751e09f5d375a9039872d59d6606fc217c5105c7eca51b1a", + "mode": 420 + }, + "examples/extensions/subagent/agents/planner.md": { + "sha256": "452f21f65957753cec3b0772004209c108071be168e627fffd7191234c0ce64e", + "mode": 420 + }, + "examples/extensions/subagent/agents/reviewer.md": { + "sha256": "b5c3f13b67207e8e32de631f8cd52a373d9b956028e2f8c1fd8b737b5f239265", + "mode": 420 + }, + "examples/extensions/subagent/agents/scout.md": { + "sha256": "3b84e8e01c6e05c8dfbff1c845c0c6371f4f16350735d4d1dbd02bfa98b1d4e6", + "mode": 420 + }, + "examples/extensions/subagent/agents/worker.md": { + "sha256": "ca2b2298fcbd8bfafcfe61977dc8075001a64b9c4d8c289c3b5eaa896efe6238", + "mode": 420 + }, + "examples/extensions/subagent/agents.ts": { + "sha256": "cc85ecd33b89aa575bea3d2a9a9da7654a6305da4de02e5041aa5342a1de29b4", + "mode": 420 + }, + "examples/extensions/subagent/index.ts": { + "sha256": "1d11858d163e2a3a258af51c1923fc55ad825813acd6224462fd25ad61f5401b", + "mode": 420 + }, + "examples/extensions/subagent/prompts/implement-and-review.md": { + "sha256": "019ba4bf42d596ea03fc4564be5b2cc55a06227c95db19a4d5a5bb96697a75f9", + "mode": 420 + }, + "examples/extensions/subagent/prompts/implement.md": { + "sha256": "65462de33c97ac1372cfed270dad9af164605b05028c069dd4c013ff9792675b", + "mode": 420 + }, + "examples/extensions/subagent/prompts/scout-and-plan.md": { + "sha256": "9506399391e0d2c71826c45f26cc332175dd964465f99a8bb880e109f8ed89f4", + "mode": 420 + }, + "examples/extensions/summarize.ts": { + "sha256": "9c75e838071773d5bacce27ac8c5305160c56958018884f960c3db96b3d09368", + "mode": 420 + }, + "examples/extensions/system-prompt-header.ts": { + "sha256": "aa97f1add4fe347b3fc37cd3709950a105b0db5d6cdaf19325b58a9fbfd05223", + "mode": 420 + }, + "examples/extensions/tic-tac-toe.ts": { + "sha256": "479597f4e76ef1eae3dae71ec9d5c26978c5b9221f950e3e134078c5b532314b", + "mode": 420 + }, + "examples/extensions/timed-confirm.ts": { + "sha256": "c35304599e6f07238ef5472efe9c3f2fef2a022f801750ed792ef9061c7de10f", + "mode": 420 + }, + "examples/extensions/titlebar-spinner.ts": { + "sha256": "ec56e7edf954966b5991bcad63141be1a09d17753cdd0c59efca1b9e649dadf2", + "mode": 420 + }, + "examples/extensions/todo.ts": { + "sha256": "e46824d00217e25242c186d41837cc84ca81b23f978500323448502a9a424ee2", + "mode": 420 + }, + "examples/extensions/tool-override.ts": { + "sha256": "5d3cc796311a16132db810fcd9e437c16abd46978b723b9120734e57cc545726", + "mode": 420 + }, + "examples/extensions/tools.ts": { + "sha256": "5eaebc40cffe885fee25c95e3d8aec79cfbd595b6eac8ac5a6de6c4afec3d5d8", + "mode": 420 + }, + "examples/extensions/trigger-compact.ts": { + "sha256": "92c3b831f9f7bf01a3e0a2df11f4e404f5ac5843000b94e8340a5d78f3cb3add", + "mode": 420 + }, + "examples/extensions/truncated-tool.ts": { + "sha256": "4ed5fbeb6da53eab8d1e04721d77961b8d1b3bf5e11d1512a17fc5a3f58f9955", + "mode": 420 + }, + "examples/extensions/widget-placement.ts": { + "sha256": "ffabd33cf58efb908de2025aac4cb41daf13dbeee24b5ad8dcd0361b43eff6d7", + "mode": 420 + }, + "examples/extensions/with-deps/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/with-deps/index.ts": { + "sha256": "8d4dbc7028c891eac7f0386d25eecea3e9ac0de7fd841b5bb08137e2ec5ab443", + "mode": 420 + }, + "examples/extensions/with-deps/package-lock.json": { + "sha256": "a4863bc4e63ff31dd43a6b469835dd0057f8792d940af0a349ec35187ad03663", + "mode": 420 + }, + "examples/extensions/with-deps/package.json": { + "sha256": "1cedd74eb4891d548ff4ae0b90adb3ce73086fa21b11fe489e0d8b1ead1c5267", + "mode": 420 + }, + "examples/extensions/working-indicator.ts": { + "sha256": "e88295d765b78fd6aac0e93db92b183ee9b7103dc21577e3c5a42665dfad032c", + "mode": 420 + }, + "examples/extensions/working-message-test.ts": { + "sha256": "91ee09378f13cce142a4314bec1bfe9784e10484c33463c4fe3139c391c117a0", + "mode": 420 + }, + "examples/rpc-client.ts": { + "sha256": "b46b19c291fbcb59f8f9c539c2dbe12a43eae25e4d7528084844bbb79d27b6f7", + "mode": 420 + }, + "examples/rpc-extension-ui.ts": { + "sha256": "09bc33c28fdc9032e3752a487d659ac21c6776c6f589c5b1ae0967ef4ee5f127", + "mode": 420 + }, + "examples/sdk/01-minimal.ts": { + "sha256": "ffe5215e36f750f5ca7146cc7dfa5be36461292036d55b8db652b4d26331f351", + "mode": 420 + }, + "examples/sdk/02-custom-model.ts": { + "sha256": "dccc8dbdb1742ea00beeed87d302b8a23ac7f7e36a0b6a4c43b283c21be0ee55", + "mode": 420 + }, + "examples/sdk/03-custom-prompt.ts": { + "sha256": "d6db1f36a8e04466d9cbcc24ffb6fe0697858aa6c0e5013d5d90f12798314ef5", + "mode": 420 + }, + "examples/sdk/04-skills.ts": { + "sha256": "4cbf189bf88d14155281349d4d39cc236066d98095aae2d184d4ba7267ab5e42", + "mode": 420 + }, + "examples/sdk/05-tools.ts": { + "sha256": "5df5f9577b343d22d072d839341916536299c8191dc66c6c801a8bbddda23544", + "mode": 420 + }, + "examples/sdk/06-extensions.ts": { + "sha256": "8e01ee409200b6516e9a6a0d9f3b3f61d2ad78aaaa18bd2ef9a8f77820d149b2", + "mode": 420 + }, + "examples/sdk/07-context-files.ts": { + "sha256": "37bd18a284e961903d646e0cd1abdeda72cd968dbe4f40892e8d345a66916b36", + "mode": 420 + }, + "examples/sdk/08-prompt-templates.ts": { + "sha256": "a1b314988f1f5b2e61c80c19c79b68630a2f3c443008a200376286710a2744ed", + "mode": 420 + }, + "examples/sdk/09-api-keys-and-oauth.ts": { + "sha256": "d1d2b1bd4e57a3b2ab6f910068e8382759d12a94fbecf1332e96a9ebe4683eb4", + "mode": 420 + }, + "examples/sdk/10-settings.ts": { + "sha256": "558f82b9d0655935e6a28ec09f6c35f0f9593f8b034f25d45400c02bc77a778a", + "mode": 420 + }, + "examples/sdk/11-sessions.ts": { + "sha256": "538c7028bf6823c84be930eb6745731f2c87ba05a45df2d86d93e66f01674576", + "mode": 420 + }, + "examples/sdk/12-full-control.ts": { + "sha256": "3cf794d470ddbde4504ae965d24a23c42378d1ac86033522e46907ac05518343", + "mode": 420 + }, + "examples/sdk/13-session-runtime.ts": { + "sha256": "0e9f408720a561a62b2630eccb18ece49f9dba567dc65494757da0a91608827a", + "mode": 420 + }, + "examples/sdk/14-codemode-mcp.ts": { + "sha256": "02ae7714e22753a76b0f44033366d43529185b325b92d47bb5d1d0eb5ae34f5e", + "mode": 420 + }, + "examples/sdk/README.md": { + "sha256": "c1f9d8f58c300195d03a420222d2d5370d2884a18a4190a1d8c8ea9371c016ff", + "mode": 420 + }, + "export-html/ansi-to-html.d.ts": { + "sha256": "57d23f45aef05ba2b1a52a8e6b0e1f1ba51b55ab0b54e8d42938a438b088977a", + "mode": 420 + }, + "export-html/ansi-to-html.d.ts.map": { + "sha256": "2847dc4f48c9ec5c8cd7aeacf1a6083ab4fc44f4b7e6667547b1549215c68576", + "mode": 420 + }, + "export-html/ansi-to-html.js": { + "sha256": "44e5fd3585104e5e8cc0c7bea91a10efa7cb7e3b5bc5e2fb9090d5be4419df14", + "mode": 420 + }, + "export-html/ansi-to-html.js.map": { + "sha256": "d24d23d27ad31a21a4fb133c6e7ba2dc2c876f28656ff1254b1af3da234dc51c", + "mode": 420 + }, + "export-html/index.d.ts": { + "sha256": "cd83b5d85e893b8ca18742b28038883dd7cf06c98a203e00d74040f78a6a656d", + "mode": 420 + }, + "export-html/index.d.ts.map": { + "sha256": "1c31d2cbab7deb811274f473316e1fed48bc2a93707f2f964db86ba2f200d522", + "mode": 420 + }, + "export-html/index.js": { + "sha256": "9b7a893c46341d2557ba79dd88307797054577c8789665281d1dfb4134d347a3", + "mode": 420 + }, + "export-html/index.js.map": { + "sha256": "f2bf1920c337f74e83454665e5a1af9621cd9e67fbcca9aade7231eb624bf9a7", + "mode": 420 + }, + "export-html/template.css": { + "sha256": "8ee19851f8e583277ed396cbb76496687aa556707fdfc1f78d1118bff87c740a", + "mode": 420 + }, + "export-html/template.html": { + "sha256": "47d208235b066c856dae69c60281a3b99d164b99be9b81153d880375c9327105", + "mode": 420 + }, + "export-html/template.js": { + "sha256": "946bd9e3e8255c1ebb40f806def18007aa9397816291a7bf2ac0e4e99cec07a4", + "mode": 420 + }, + "export-html/tool-renderer.d.ts": { + "sha256": "ea982244e87c3568ad7bbb28427b83a19506136b5a498c7b496454338483ffd4", + "mode": 420 + }, + "export-html/tool-renderer.d.ts.map": { + "sha256": "2fab18f196e020f0b896b27db7cf790c233f0d0af55de04194e60b5bff137fce", + "mode": 420 + }, + "export-html/tool-renderer.js": { + "sha256": "8d3bd1f1b068996fe8f7d003bdb6623284043e1393a2db92ff45aeb9d6c33b9a", + "mode": 420 + }, + "export-html/tool-renderer.js.map": { + "sha256": "7bead30f87d1b345c70c66953a517fb51d1e1092089158ad5e92280cc97f6ea8", + "mode": 420 + }, + "export-html/vendor/highlight.min.js": { + "sha256": "837a6fa5b0c736b52bbde2b2b6190f305da3fc9ed41681db5321507057b5c846", + "mode": 420 + }, + "export-html/vendor/marked-cjk-friendly.min.js": { + "sha256": "93d2c938754cb24eef5b9ffecc268a98ef6beeb75dad614b0e874176aeeba565", + "mode": 420 + }, + "export-html/vendor/marked.min.js": { + "sha256": "438eedfcf932a414d0d0bfeea32dc365c063563b8ca713b4687fc8f8b501e5e4", + "mode": 420 + }, + "native/linux/prebuilds/linux-x64/linux-platform-x11.node": { + "sha256": "522d2550f7fa5d0d5bd204f45983f786ce2be9cf80f27f00179640d31d0689d6", + "mode": 493 + }, + "package.json": { + "sha256": "df6883685382d6a914e5797232744c26579fe7db5576ea0e9c859d018ca2d54e", + "mode": 420 + }, + "photon_rs_bg.wasm": { + "sha256": "10468181565c56004c867f3a4af96f89a0ef5a63a72f2b5fb12c1f1992a3615c", + "mode": 420 + }, + "pi": { + "sha256": "0d7611d0b86d38ec8f65dda8e8bee9c5e907c7bae381e9cfff9161b21e8087d0", + "mode": 493 + }, + "theme/dark.json": { + "sha256": "e1f96982f8cb7d2a48eca7db03c9cb282cb1e14a243680cd855638ecd6a251b6", + "mode": 420 + }, + "theme/light.json": { + "sha256": "2dee6a4be7f765e7fd7728166b7ebf8c29e4ce313f8ec6b38ae50699e3abc2f3", + "mode": 420 + } + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/pi-models.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/pi-models.json new file mode 100644 index 0000000..32fd1b7 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/pi-models.json @@ -0,0 +1,42 @@ +{ + "providers": { + "deepseek": { + "api": "openai-completions", + "baseUrl": "https://api.deepseek.com", + "models": [ + { + "id": "deepseek-flash", + "name": "DeepSeek V4.1 Flash", + "reasoning": true, + "input": [ + "text", + "image" + ], + "thinkingLevelMap": { + "minimal": null, + "low": "low", + "medium": null, + "high": "high", + "max": "max" + }, + "contextWindow": 1000000, + "maxTokens": 384000, + "cost": { + "input": 0.3, + "output": 1.2, + "cacheRead": 0.006, + "cacheWrite": 0 + }, + "compat": { + "supportsStore": false, + "supportsDeveloperRole": false, + "maxTokensField": "max_tokens", + "requiresReasoningContentOnAssistantMessages": true, + "thinkingFormat": "deepseek" + } + } + ], + "apiKey": "${DEEPSEEK_API_KEY}" + } + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/preflight-evidence.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/preflight-evidence.json new file mode 100644 index 0000000..a558f97 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/preflight-evidence.json @@ -0,0 +1,487 @@ +{ + "recorded_at": "2026-11-05T15:14:12.409429+00:00", + "trial": "torch-pipeline-parallelism__ECDiMhB", + "status": "setup_passed", + "model_calls": 0, + "setup_seconds": 212.639386, + "setup_limit_seconds": 1800, + "dependency_profile": { + "profile": "torch-pipeline-parallelism-v1", + "status": "passed" + }, + "original_failure": { + "trial": "torch-pipeline-parallelism__aykTgx7", + "exception": "AgentSetupTimeoutError", + "setup_seconds": 1800.034097, + "model_calls": 0, + "evidence": "../pi-restricted-dev2-20261104/preflight-evidence.json" + }, + "cache": { + "phase": { + "protocol": "pypi-verified-v1", + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "helper": "nanopy-pypi-7635901cde1244ee911843e4f9ac14f5", + "scripts_sha256": { + "pypi_wheel_cache.py": "ced6bd7c1bf2b6ac1964c39d057708ba0e52c37c61a609651c49105ea145ea74", + "cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c" + }, + "phase": "setup", + "started_at": "2026-11-05T15:08:00.558802+00:00", + "removed": true, + "finished_at": "2026-11-05T15:10:59.582978+00:00" + }, + "wheel_requests": 45, + "cache_hits": 44, + "downloads": 1, + "verified_bytes": 3080594478, + "all_match_current_metadata": true, + "wheels": [ + { + "at": "2026-11-05T15:10:22.324362+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 216561367, + "sha256": "7556d9eca156e18184b94947ade0fba5bb47d69cec46bf8660fd2c71a4b48b73", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:22.367785+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_nccl_cu12-2.26.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 201319755, + "sha256": "694cf3879a206553cc9d7dbda76b13efaf610fdb70a50cba303de1b0d1530ac6", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:22.988462+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cublas_cu12-12.6.4.1-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 393138322, + "sha256": "08ed2686e9875d01b58e3cb379c6896df8e76c75e0d4a7f7dace3d7b6d9ef8eb", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:22.990252+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "setuptools-84.2.1-py3-none-any.whl", + "size": 876263, + "sha256": "ca140b45d5325a9478f880e351484f62fef15f336462e449ca0c03c873d589c2", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:22.990764+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "iniconfig-2.3.1-py3-none-any.whl", + "size": 7552, + "sha256": "9121e2c1fdb355232495be3194c8dfe87ccc2d5dee45947b78e68f499790d7a7", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.772962+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "torch-2.7.0-cp313-cp313-manylinux_2_28_x86_64.whl", + "size": 865046389, + "sha256": "9b52347118116cf3dff2ab5a3c3dd97c719eb924ac658ca2a7335652076df708", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.785082+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cuda_cupti_cu12-12.6.80-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 8917980, + "sha256": "6768bad6cab4f19e8292125e5f1ac8aa7d1718704012a0e3272a6f61c4bce132", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.913338+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cudnn_cu12-9.5.1.17-py3-none-manylinux_2_28_x86_64.whl", + "size": 570988386, + "sha256": "30ac3869f6db17d170e0e556dd6cc5eee02647abc31ca856634d5a40f82c15b2", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.915007+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "certifi-2026.10.23-py3-none-any.whl", + "size": 135082, + "sha256": "e9b54de76e530bb6cf0ccd036787103acb58fd42ba930ea8bf948ddbaa2f61d1", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.915642+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "markupsafe-3.0.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", + "size": 23017, + "sha256": "434139499bb20b502ed3baa1f169e618f924a97e7a777fea1a49446d80106cf6", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.916427+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "requests-2.34.2-py3-none-any.whl", + "size": 73075, + "sha256": "2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.922926+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "hf_xet-1.8.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 4292483, + "sha256": "29dab7c904cd1914a0439c6e1306c12c7497c8bd32290fbe563ebf51ca4b6d6d", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.058701+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "triton-3.3.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", + "size": 156516468, + "sha256": "47bc87ad66fa4ef17968299acacecaab71ce40a238890acc6ad197c3abe2b8f1", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.059832+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "pytest-8.4.1-py3-none-any.whl", + "size": 365474, + "sha256": "539c70ba6fcead8e78eebbf1115e8b589e7565830d7d006a8723f19ac8a0afb7", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.111859+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_nvjitlink_cu12-12.6.85-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", + "size": 19744971, + "sha256": "eedc36df9e88b682efe4309aa16b5b4e78c2407eac59e8c10a6a47535164369a", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.186034+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_curand_cu12-10.3.7.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 56279010, + "sha256": "a42cd1344297f70b9e39a1e4f467a4e1c10f1da54ff7a85c12197f6c652c8bdf", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.186580+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "idna-3.20-py3-none-any.whl", + "size": 69583, + "sha256": "ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.190196+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "networkx-3.7-py3-none-any.whl", + "size": 2142205, + "sha256": "e3fd2c13a7814cee3746340d8d7f8598a67f16a58bf47fb7f8793fab6efca1b0", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.191365+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "packaging-26.3-py3-none-any.whl", + "size": 129956, + "sha256": "d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.201145+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cusolver_cu12-11.7.1.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 158229790, + "sha256": "e9e49843a7707e42022babb9bcfa33c29857a93b88020c4e4434656a655b698c", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.204217+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "pygments-2.21.0-py3-none-any.whl", + "size": 1250147, + "sha256": "2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.204907+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "pluggy-1.7.0-py3-none-any.whl", + "size": 27082, + "sha256": "7dd7b0d8832ba3cb632c306926ded123429211b83641b35dc5c41ad2d34f9bec", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.205607+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "tqdm-4.70.1-py3-none-any.whl", + "size": 80199, + "sha256": "c293e525e6fef9c20e8728fd4612df02a0aa31bb5fe91ecd93e123b1b7bffa73", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.207119+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "mpmath-1.3.0-py3-none-any.whl", + "size": 536198, + "sha256": "a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.207872+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_nvtx_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 89276, + "sha256": "b90bed3df379fa79afbd21be8e04a0314336b8ae16768b58f2d34cb1d04cd7d2", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.215353+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "numpy-2.5.4-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", + "size": 16718044, + "sha256": "6ffa07666f8da0eef81d149934a626d0d95fbd6838432a33e66245423a9062c0", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.239318+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cuda_nvrtc_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl", + "size": 23650380, + "sha256": "35b0cc6ee3a9636d5409133e79273ce1f3fd087abb0532d2d2e8fff1fe9efc53", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.262690+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "pytest_json_ctrf-0.3.5-py3-none-any.whl", + "size": 6837, + "sha256": "e82fd1d69be2f92385bc33540063e5ad7b17b36de67764c84f3ceb9815a895e9", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.539236+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "typing_extensions-4.16.0-py3-none-any.whl", + "size": 45571, + "sha256": "481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.582227+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cuda_runtime_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 897690, + "sha256": "ba3b56a4f896141e25e19ab287cd71e52a6a0f4b29d0d31609f60e3b4d5219b7", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.642989+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", + "size": 801626, + "sha256": "0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.903886+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "transformers-4.55.0-py3-none-any.whl", + "size": 11267905, + "sha256": "29d9b8800e32a4a831bb16efb5f762f6a9742fef9fce5d693ed018d19b106490", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:25.744929+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cufile_cu12-1.11.1.6-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 1142103, + "sha256": "cc23469d1c7e52ce6c1d55253273d32c565dd22068647f3aa59b3c6b005bf159", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:25.930513+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "fsspec-2026.9.0-py3-none-any.whl", + "size": 221738, + "sha256": "8dd6e646e99ea382bd85f97a45e6b526a442d79423a7dc673f1e2756d05fcb5f", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.421284+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cufft_cu12-11.3.0.4-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 200221632, + "sha256": "ccba62eb9cef5559abd5e0d54ceed2d9934030f51163df018532142a8ec533e5", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.422039+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "urllib3-2.8.0-py3-none-any.whl", + "size": 135717, + "sha256": "0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.428615+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "tokenizers-0.21.4-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", + "size": 3115426, + "sha256": "51b7eabb104f46c1c50b486520555715457ae833d5aee9ff6ae853d1130506ff", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.429365+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "charset_normalizer-3.5.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", + "size": 254660, + "sha256": "7218e8f32b0956cfcd048fd42d9d5779809745ca1d86113ca56f66e7ae1549c4", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.430032+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "jinja2-3.1.6-py3-none-any.whl", + "size": 134899, + "sha256": "85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.983022+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cusparselt_cu12-0.6.3-py3-none-manylinux2014_x86_64.whl", + "size": 156785796, + "sha256": "e5c8a26c36445dd2e6812f1177978a24e2d37cacce7e090f297a688d1ec44f46", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:29.247387+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "filelock-4.2.9-py3-none-any.whl", + "size": 164734, + "sha256": "a379c50a729e5b68c8519a4d26489f755d9769960a236ecd838190e2a78db2e9", + "cache_hit": false + }, + { + "at": "2026-11-05T15:10:29.417780+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "huggingface_hub-0.36.2-py3-none-any.whl", + "size": 566395, + "sha256": "48f0c8eac16145dfce371e9d2d7772854a4f591bcb56c9cf548accf531d54270", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:29.956389+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", + "size": 516040, + "sha256": "fd6f3f93c9a0a7cc2788ee63fb763353d4bd2e89b0751bc78fcf7dda00bea774", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:30.270889+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "sympy-1.14.0-py3-none-any.whl", + "size": 6299353, + "sha256": "e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:56.411548+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "regex-2026.9.29-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", + "size": 807902, + "sha256": "d49c18f1ea294cf4adde2e5ac256e98c82ea9d708462ce4bf799dffa7cfe8a2c", + "cache_hit": true + } + ] + }, + "enforcement": { + "protocol": "restricted-v1", + "policy": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "rules_sha256": "89db5b77afe8608c1d10884151a3730d5878929ecc3a0ac53b0ed164bb7fbf47", + "preflight": "passed", + "agent_phase": "not_started", + "host_mounts": false, + "verifier_isolation": "task_defined; shared tasks remain shared", + "dependency_cache": { + "protocol": "pypi-verified-v1", + "absent_before_agent": false + }, + "dns_upstreams": [ + "172.17.0.1" + ], + "positive_control": { + "reachable": true + }, + "negative_control": { + "reachable": false, + "error_type": "URLError", + "reason_type": "SSLEOFError" + }, + "allowed_controls": { + "api.deepseek.com": { + "reachable": true + } + } + }, + "freeze_sha256": "14825f8acba7ebbaad45863b7c746879cff70a931b870115d6f7d672ea813aa9", + "frozen_files": 15, + "result_sha256": "59c6a75600f5a464d8984174e99eafc1427a0ad53057e4fc5913ea85efccf0f5", + "manifest_sha256": "0627797a0a937a5ca0801fe9ae209a232a0cfe40a10093ccb91a8a890f000ca4", + "input_config_sha256": "bbdf1fbd6a324c4450bb3973d700229c134ed26b3335b92cbe61d626df02c5a1", + "bootstrap_sha256": "4888105f026b9e6be5d2861c6650b8c8ecfdb5375e2a262bf1a4038652860846" +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/preflight-freeze.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/preflight-freeze.json new file mode 100644 index 0000000..e241d62 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/preflight-freeze.json @@ -0,0 +1,22 @@ +{ + "schema_version": 1, + "job_dir": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-pipeline-install-20261105", + "frozen_at": "2026-11-05T15:11:40.782169+00:00", + "files": { + ".integrity/torch-pipeline-parallelism__ECDiMhB/dependency-cache-setup-log.json": "abe554d7f5d23e8c4680b8b37ef8047b94101681045f640ff8ea869f25638f32", + ".integrity/torch-pipeline-parallelism__ECDiMhB/dependency-cache-setup.json": "98c7263417d77582531b230cdafd6161df8cc661208e86fc4fd90af47b568079", + ".integrity/torch-pipeline-parallelism__ECDiMhB/dependency-setup-output.jsonl": "aa22ec310e13d98ade51b2b2dad590a4d8938754109449dc7f8b7a47d39ff4e4", + ".integrity/torch-pipeline-parallelism__ECDiMhB/enforcement.json": "45d06294885a6d64ac8aead6386c78071253a8c9b8871a2122f440fd45e73058", + ".integrity/torch-pipeline-parallelism__ECDiMhB/network-preflight.json": "6cdb1220eec019f01d9e3f1b8854a6a14c6c48e054739f1140dcf0a293a0aaa0", + "config.json": "52bf4ddb7d611ac13832ac8faef7b825ef1c1ae42dee44cb5b0bc8837fbd2d75", + "job.log": "1ed582a1bc7793ab942eeaa0c8e7642448c4dd9a7f4878caf6c166b6a3b32c3c", + "lock.json": "86d41e592174f57f88b95aa4ba31d06714d49520dfc189c362dc4cce46a33a7e", + "result.json": "c8bff727a9861ef4d01cf440cd98f70cfdffc1478d9885b920dd519c4681faf6", + "torch-pipeline-parallelism__ECDiMhB/agent/bootstrap.json": "4888105f026b9e6be5d2861c6650b8c8ecfdb5375e2a262bf1a4038652860846", + "torch-pipeline-parallelism__ECDiMhB/agent/process-supervision-required": "9e67a1538c0f96556945933b784624bed4b3f7d177108e2ee1af88d6ea1314c8", + "torch-pipeline-parallelism__ECDiMhB/config.json": "a43ff89859b054f2aa76dd38b64d85f88b2404b448b13965c5d10597f4c17a1b", + "torch-pipeline-parallelism__ECDiMhB/lock.json": "12a5f954a79220bf1e1d0d29ded119c52e8b03321dae828aecac53cb0339006f", + "torch-pipeline-parallelism__ECDiMhB/result.json": "59c6a75600f5a464d8984174e99eafc1427a0ad53057e4fc5913ea85efccf0f5", + "torch-pipeline-parallelism__ECDiMhB/trial.log": "1ed582a1bc7793ab942eeaa0c8e7642448c4dd9a7f4878caf6c166b6a3b32c3c" + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/prior-freeze-validation.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/prior-freeze-validation.json new file mode 100644 index 0000000..82e27d3 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/prior-freeze-validation.json @@ -0,0 +1,17 @@ +[ + { + "manifest": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/preflight-freeze.json", + "files": 21, + "unchanged": true + }, + { + "manifest": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/wal-freeze.json", + "files": 28, + "unchanged": true + }, + { + "manifest": "benchmarks/harbor/configs/pi-deepseek-flash-dev3-rerun-20261102/outputs-freeze.json", + "files": 326, + "unchanged": true + } +] diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/torch-pipeline-parallelism.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/torch-pipeline-parallelism.json new file mode 100644 index 0000000..f2f1c92 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/torch-pipeline-parallelism.json @@ -0,0 +1,44 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "retry": { + "max_retries": 0 + }, + "agents": [ + { + "name": "harbor_adapter:PiCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9", + "model_config_path": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/pi-models.json", + "thinking": "high" + } + } + ], + "environment": { + "type": "docker" + }, + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + } + ], + "verifier": { + "import_path": "harbor_adapter:RetryingVerifier", + "kwargs": { + "max_attempts": 3, + "require_agent_quiescence": true + } + }, + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "dependency_cache": "pypi-verified-v1" +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/dependency-release-manifest.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/dependency-release-manifest.json new file mode 100644 index 0000000..e1a1a07 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/dependency-release-manifest.json @@ -0,0 +1,25 @@ +{ + "registered_at": "2026-11-05T15:24:45.573851+00:00", + "metadata_source": "Original public GitHub release API, fetched before this registration", + "assets": [ + { + "version": "0.9.5", + "filename": "uv-x86_64-unknown-linux-gnu.tar.gz", + "url": "https://github.com/astral-sh/uv/releases/download/0.9.5/uv-x86_64-unknown-linux-gnu.tar.gz", + "digest": "sha256:2cf10babba653310606f8b49876cfb679928669e7ddaa1fb41fb00ce73e64f66", + "size": 21370871, + "asset_id": 306970260, + "release_metadata_sha256": "da2f5fa63817079603dfbbe5ba65f21de800dc983931ff7310edede3124f505d" + }, + { + "version": "20251014", + "filename": "cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "url": "https://github.com/astral-sh/python-build-standalone/releases/download/20251014/cpython-3.13.9%2B20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "digest": "sha256:c0ccda275948c79996e46993c2c5476ff5cca606dee530f1dea712179131b348", + "size": 33517986, + "asset_id": 304305340, + "release_metadata_sha256": "1d1d776f53b03890621ffb4455920ffcebb2647cb8ee4f10703f2ce3631d4166", + "repository": "astral-sh/python-build-standalone" + } + ] +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/experiment.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/experiment.json new file mode 100644 index 0000000..f953f40 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/experiment.json @@ -0,0 +1,121 @@ +{ + "schema_version": 1, + "max_concurrent_trials": 2, + "whole_trial_retries": 0, + "minimum_free_disk_bytes": 16106127360, + "arms": { + "pi": { + "agent": "pi", + "source_root": ".", + "source_ref": "a867883257b9511793b52152561de04808f0c6f9", + "distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9" + } + }, + "batch_images": { + "manifest": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/image-digests.json", + "storage_path": "/var/lib/docker", + "minimum_free_disk_bytes": 16106127360, + "cache_dir": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/nanopy-harbor/images" + }, + "model_policy": { + "model": "deepseek/deepseek-flash", + "api": "openai-completions", + "base_url": "https://api.deepseek.com", + "thinking": "high", + "model_config": "pi-models.json", + "model_metadata_max_tokens": 384000, + "pricing": "Peak-rate estimates from the registered Pi model config; not billing reconciliation. Official pricing also has off-peak discounts.", + "alias": "Reuse the original mutable deepseek-flash alias; no identical backend checkpoint claim across dates." + }, + "budget_policy": "Original task-native agent timeouts, original resources, and 1800-second setup. Stock Pi loop and generation behavior: no injected 100-turn cap or 180-second finalization reminder. Model metadata maxTokens=384000 is preserved from the local provider configuration. This differs from the historical Nano baseline (100 replies, 65536 requested output tokens, native timeout minus 180 seconds for work), so this is an agent baseline rather than a matched-budget causal comparison. Temperature unset.", + "verifier_policy": "Original official tests and native per-attempt timeouts. Only verifier timeouts retry, at most three total attempts on the same output. No dependency-error recovery or whole-trial retry. Preserve setup errors and verifier errors separately from scored failures.", + "runtime_environment": { + "UV_PROJECT_ENVIRONMENT": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv", + "UV_CACHE_DIR": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/.cache/uv" + }, + "registered_at": "2026-11-05T15:28:42.850594+00:00", + "experiment": "pi-restricted-pipeline-recovery-v2-20261105", + "total_model_runs": 1, + "log_dir": "jobs/pi-restricted-pipeline-recovery-v2-20261105-orchestration", + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb", + "native_agent_timeout_seconds": 900, + "verifier_timeout_seconds": 900.0, + "docker_image": "alexgshaw/torch-pipeline-parallelism:20251031", + "resources": { + "cpus": 1, + "memory_mb": 8192, + "storage_mb": 10240, + "gpus": 0 + }, + "category": "software-engineering", + "verifier_dependency_profile": "torch-pipeline-parallelism-v1", + "task_file_sha256": { + "README.md": "297ddea07c230dfd246a8ab07337dd7abcd8e5e2a2b2e26590cb0a8af779839c", + "environment/Dockerfile": "c357ea41e4a4013c5c5980b8fef04af6a5ae250ccc6697cf3bad3127401a931f", + "instruction.md": "989bfcf0c54baf6c8cdc391743dbb2f9e858c5779dd85112536875d4d1b2dcd2", + "solution/solve.sh": "787198e4178d88b887697aefdd29819f29435ffdfe356545d92f58a30fb38b70", + "task.toml": "8b4f9f525c08afbbc4eca9168e46238a61ccb903486773e9ca4da64bb7ffe857", + "tests/test.sh": "cc9fd28b6590993fc37e4a99da771cbf04726c8c62fe308e802612241da69dea", + "tests/test_outputs.py": "23574246a407163a8e6581c84287cc2f1eb745efbaad2a317a5a28c39958c64f" + }, + "prior_nanopy_work_budget_seconds": 720, + "pi_agent_timeout_seconds": 900 + } + ], + "preflight": [ + { + "arm": "pi", + "config": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/install-pipeline.json", + "config_sha256": "a3cd7d8d510ebc88bda423a9208a79fd13b3e31b105b51adaabeb065cdea3ec2", + "job_name": "pi-restricted-pipeline-v2-install-20261105" + } + ], + "queue": [ + { + "arm": "pi", + "config": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/torch-pipeline-parallelism.json", + "config_sha256": "a3cd7d8d510ebc88bda423a9208a79fd13b3e31b105b51adaabeb065cdea3ec2", + "job_name": "pi-restricted-pipeline-v2-model-20261105", + "task": "terminal-bench/torch-pipeline-parallelism", + "repeat": 1 + } + ], + "additional_file_sha256": { + "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/dependency-release-manifest.json": "3a132b29494bea24ec2c872a9b9c88dda059575a0bb9f2d869f9ff1c50810edb", + "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/image-digests.json": "ef97fd9ad5b4e46fd828661b235e5284f3baaa9e93b0b5ea67bbbbc49aadc93f", + "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/install-pipeline.json": "a3cd7d8d510ebc88bda423a9208a79fd13b3e31b105b51adaabeb065cdea3ec2", + "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/pi-distribution.json": "0e1e2cdbef4ede58b2beed0f4c3655099fb2d6f7e3f6d2ca369879fa24546eb1", + "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/pi-models.json": "8751fac554abb9f363b188c102045f38e36693d52956f984e0e427b31998b299", + "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/torch-pipeline-parallelism.json": "a3cd7d8d510ebc88bda423a9208a79fd13b3e31b105b51adaabeb065cdea3ec2", + "benchmarks/harbor/scripts/run_comparison.py": "7773f68c5472ed6c528b4dd3597872f2d3923ce330ce525cf363ea1a3dc2846c", + "benchmarks/harbor/scripts/cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c", + "benchmarks/harbor/scripts/pypi_wheel_cache.py": "c3bf80298fbdfe14cdf1809249b78063eed9c378139723bbcf4b987f07c281f3", + "benchmarks/harbor/scripts/uv_release_cache.py": "a8dac0e782b1aa684209195e2715a68fbb5157d9974e59678c3e83999d5390a3" + }, + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "preflight_policy": "Fresh install-only dependency and network controls must pass before model work. Use the reviewed torch-pipeline-parallelism-v1 profile and pypi-verified-v1 transport. Keep original 1800-second setup, 900-second model, and 900-second per-verifier-attempt limits. Fresh formal job repeats setup with the same cache. Preserve failed preflight evidence and fail closed; no legacy opt-out. Both wheel and original-release cache paths must pass in a fresh install-only container before model work.", + "information_policy": "One fresh Pipeline model attempt, after the November 4 installation failure occurred before model work. Use restricted-v1 shared rules and external egress enforcement; api.deepseek.com is the only allowed model-phase host. The managed package transport is absent throughout model work. Native public network is available only during installation and official verification. Freeze completed outputs before trajectory and integrity review. Preserve raw rewards and errors; reviewed violations are effective zeroes, blocked attempts are separate from retrieval, and unreviewed outcomes are pending. Do not pool with historical open-network runs or claim a matched intervention against WAL, whose run did not use this transport.", + "authorization": "Maintainer requested rerunning both previously cheating Pi tasks with restricted networking, then asked why dependency installation timed out and how to resolve it. This recovery adds a reusable verified package transport, preserves the prior failed attempt, and completes the previously unstarted Pipeline model run. WAL is not rerun. The first wheel-cache preflight passed in 212.639386 seconds, but a fresh formal container failed before model work because the original uv 0.9.5 release download was truncated (curl exit 18). This separately registered infrastructure correction adds the existing verified original-release cache. No completed model attempt is repeated, and the prior failed job remains frozen.", + "original_results": "benchmarks/harbor/results/pi-deepseek-flash-dev3-rerun-20261102.json", + "dependency_cache": { + "protocol": "pypi-verified-v1", + "helper_image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "phase_policy": "Controller-owned loopback helper only during dependency setup and official verification, removed and checked before model work. Every cached wheel must match current PyPI metadata filename, size, and SHA256. Task image, dependency versions, upstream package sources, and official scripts remain unchanged. Helper has a separate 1 CPU and 512 MiB limit and no task filesystem mount. No custom Compose overlay or task-container host mount. Also mount the original uv/Python archive cache read-only into the same helper. The unchanged installers use supported UV_INSTALLER_GITHUB_BASE_URL and UV_PYTHON_INSTALL_MIRROR only during setup and official verification. Fresh original GitHub release metadata binds archive filename, size, and SHA256; uv selects the original Python build. No helper exists during model work.", + "release_manifest": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/dependency-release-manifest.json", + "release_manifest_sha256": "3a132b29494bea24ec2c872a9b9c88dda059575a0bb9f2d869f9ff1c50810edb" + }, + "prior_setup_failure": "jobs/pi-restricted-dev2-install-pair-01-20261104/torch-pipeline-parallelism__aykTgx7/result.json", + "prior_restricted_results": "benchmarks/harbor/results/pi-restricted-dev2-20261104.json", + "superseded_registration": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/experiment.json", + "prior_release_download_failure": "jobs/pi-restricted-pipeline-model-20261105/torch-pipeline-parallelism__3gci4oE/result.json" +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/image-digests.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/image-digests.json new file mode 100644 index 0000000..e1855e4 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/image-digests.json @@ -0,0 +1,15 @@ +{ + "observed_at": "2026-10-24T14:07:09.186521+00:00", + "images": [ + { + "task": "terminal-bench/torch-pipeline-parallelism", + "image": "alexgshaw/torch-pipeline-parallelism:20251031", + "image_id": "sha256:3cb7b39d86f1704b20e668851da0b08b34b3270c01d583f136d65190984a8d1b", + "repo_digests": [ + "alexgshaw/torch-pipeline-parallelism@sha256:3cb7b39d86f1704b20e668851da0b08b34b3270c01d583f136d65190984a8d1b" + ], + "size_bytes": 29725531 + } + ], + "matches_prior_registration": true +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/implementation-validation.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/implementation-validation.json new file mode 100644 index 0000000..ea11afb --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/implementation-validation.json @@ -0,0 +1,22 @@ +{ + "source_ref": "1b6e3dcf4d143a21a3411263aa49fb986b811736", + "harbor_tests": { + "command": "UV_PROJECT_ENVIRONMENT=\"$PWD/jobs/dsh-harbor-venv\" UV_CACHE_DIR=\"$PWD/.cache/uv\" uv run --frozen --project benchmarks/harbor pytest -q -c benchmarks/harbor/pyproject.toml benchmarks/harbor/tests", + "passed": 410, + "skipped": 3, + "seconds": 19.89, + "skipped_reason": "Docker probes require an explicitly provided image; all three were run separately." + }, + "docker_tests": { + "command": "UV_PROJECT_ENVIRONMENT=\"$PWD/jobs/dsh-harbor-venv\" UV_CACHE_DIR=\"$PWD/.cache/uv\" NANOPY_INTEGRITY_TEST_IMAGE=alexgshaw/pytorch-model-cli:20251031 uv run --frozen --project benchmarks/harbor pytest -q -c benchmarks/harbor/pyproject.toml benchmarks/harbor/tests/test_integrity_docker.py", + "passed": 3, + "seconds": 97.82, + "coverage": [ + "External allowlist and direct-routing restrictions", + "Wheel cache phase lifecycle, verified delivery and repeat hit", + "Combined wheel/release listener with fixture archive, read-only release mount and phase removal", + "Blocked GitHub access and rejected verifier-cache activation during model work", + "A 300000-byte UTF-8 line survives real Docker callback collection during restricted model work" + ] + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/install-pipeline.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/install-pipeline.json new file mode 100644 index 0000000..5cdeae6 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/install-pipeline.json @@ -0,0 +1,47 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "retry": { + "max_retries": 0 + }, + "agents": [ + { + "name": "harbor_adapter:PiCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9", + "model_config_path": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/pi-models.json", + "thinking": "high" + } + } + ], + "environment": { + "type": "docker" + }, + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + } + ], + "verifier": { + "import_path": "harbor_adapter:RetryingVerifier", + "kwargs": { + "max_attempts": 3, + "require_agent_quiescence": true + } + }, + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "dependency_cache": { + "protocol": "pypi-verified-v1", + "release_manifest": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/dependency-release-manifest.json" + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/model-evidence.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/model-evidence.json new file mode 100644 index 0000000..4eeae3d --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/model-evidence.json @@ -0,0 +1,135 @@ +{ + "trial": "torch-pipeline-parallelism__7rHhxP6", + "task": "terminal-bench/torch-pipeline-parallelism", + "source_ref": "a867883257b9511793b52152561de04808f0c6f9", + "setup_seconds": 142.791023, + "agent_seconds": 433.014135, + "timestamps": { + "started_at": "2026-11-05T15:33:15.529143Z", + "finished_at": "2026-11-05T15:43:05.394041Z", + "agent_setup": { + "started_at": "2026-11-05T15:33:18.236751Z", + "finished_at": "2026-11-05T15:35:41.027774Z" + }, + "agent_execution": { + "started_at": "2026-11-05T15:35:41.027892Z", + "finished_at": "2026-11-05T15:42:54.042027Z" + }, + "verifier": null + }, + "native_exception": { + "exception_type": "ValueError", + "exception_message": "Separator is not found, and chunk exceed the limit", + "exception_traceback": "Traceback (most recent call last):\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 562, in readline\n line = await self.readuntil(sep)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 663, in readuntil\n raise exceptions.LimitOverrunError(\n 'Separator is not found, and chunk exceed the limit',\n offset)\nasyncio.exceptions.LimitOverrunError: Separator is not found, and chunk exceed the limit\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 378, in run\n await self._run()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/single_step.py\", line 41, in _run\n await self._run_agent()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/single_step.py\", line 77, in _run_agent\n await self._run_agent_phase(\n ...<5 lines>...\n )\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 480, in _run_agent_phase\n await asyncio.wait_for(\n ...<6 lines>...\n )\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/tasks.py\", line 507, in wait_for\n return await fut\n ^^^^^^^^^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 175, in wrapper\n return await fn(self, instruction, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-pipeline-v2-model-20261105-input/source/harbor_adapter/integrity.py\", line 95, in guarded\n return await run(self, RULES + \"\\nTask:\\n\" + instruction, environment, context)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-pipeline-v2-model-20261105-input/source/harbor_adapter/pi.py\", line 163, in run\n await self.exec_as_agent(\n ...<3 lines>...\n )\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 884, in exec_as_agent\n return await self._exec(\n ^^^^^^^^^^^^^^^^^\n environment, command, env=env, cwd=cwd, timeout_sec=timeout_sec\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 835, in _exec\n result = await environment.exec(\n ^^^^^^^^^^^^^^^^^^^^^^^\n ...<5 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 1096, in exec\n return await self._compose_exec(\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<6 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 1173, in _compose_exec\n return await self._run_docker_compose_command(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<4 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 642, in _run_docker_compose_command\n result = await self._collect_streamed_output(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<4 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 738, in _collect_streamed_output\n await _read_and_wait()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 732, in _read_and_wait\n await _read_stdout_and_wait()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 717, in _read_stdout_and_wait\n async for raw_line in stdout_stream:\n ...<2 lines>...\n await on_output(line, \"stdout\")\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 784, in __anext__\n val = await self.readline()\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 571, in readline\n raise ValueError(e.args[0])\nValueError: Separator is not found, and chunk exceed the limit\n", + "occurred_at": "2026-11-05T15:42:54.153274Z" + }, + "native_rewards": null, + "effective_score": null, + "official_verifier_started": false, + "diagnosis": "Harbor 0.21 reads callback output by line with the default 65536-byte StreamReader limit. Pi finished normally, but its final agent_end event was 209940 bytes. The collector raised ValueError before official verification. Native Pi logs contain agent_settled; controller output is the exact event prefix missing agent_end and agent_settled. The native error and missing reward are preserved.", + "supervision": { + "status": "completed", + "return_code": 0 + }, + "enforcement": { + "protocol": "restricted-v1", + "policy": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "rules_sha256": "89db5b77afe8608c1d10884151a3730d5878929ecc3a0ac53b0ed164bb7fbf47", + "preflight": "passed", + "agent_phase": "interrupted", + "host_mounts": false, + "verifier_isolation": "task_defined; shared tasks remain shared", + "dependency_cache": { + "protocol": "pypi-verified-v1", + "absent_before_agent": true + }, + "dns_upstreams": [ + "172.17.0.1" + ], + "positive_control": { + "reachable": true + }, + "negative_control": { + "reachable": false, + "error_type": "URLError", + "reason_type": "SSLEOFError" + }, + "allowed_controls": { + "api.deepseek.com": { + "reachable": true + } + }, + "started_at": "2026-11-05T15:35:41.373095+00:00", + "finished_at": "2026-11-05T15:42:53.848086+00:00", + "live_output_sha256": "45c29e3b0b7ba0b474268f7014e8b315d7e6b8c35687040baf44a9492044e362" + }, + "cache": { + "phase": { + "protocol": "pypi-verified-v1", + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "helper": "nanopy-pypi-0b0595153f3c4cea8f0d462ea4d7f404", + "scripts_sha256": { + "pypi_wheel_cache.py": "c3bf80298fbdfe14cdf1809249b78063eed9c378139723bbcf4b987f07c281f3", + "cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c", + "uv_release_cache.py": "a8dac0e782b1aa684209195e2715a68fbb5157d9974e59678c3e83999d5390a3", + "release-manifest.json": "3a132b29494bea24ec2c872a9b9c88dda059575a0bb9f2d869f9ff1c50810edb" + }, + "phase": "setup", + "started_at": "2026-11-05T15:33:42.534572+00:00", + "removed": true, + "finished_at": "2026-11-05T15:35:35.148407+00:00" + }, + "wheel_requests": 45, + "cache_hits": 45, + "verified_bytes": 3080594478, + "all_wheels_match_current_metadata": true, + "release_requests": [ + { + "at": "2026-11-05T15:33:46.735766+00:00", + "instance": "7d1f02dfc7fd", + "kind": "release_asset", + "filename": "uv-x86_64-unknown-linux-gnu.tar.gz", + "size": 21370871, + "sha256": "2cf10babba653310606f8b49876cfb679928669e7ddaa1fb41fb00ce73e64f66", + "source_url": "https://github.com/astral-sh/uv/releases/download/0.9.5/uv-x86_64-unknown-linux-gnu.tar.gz", + "cache_hit": true + }, + { + "at": "2026-11-05T15:33:47.192858+00:00", + "instance": "7d1f02dfc7fd", + "kind": "release_asset", + "filename": "cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "size": 33517986, + "sha256": "c0ccda275948c79996e46993c2c5476ff5cca606dee530f1dea712179131b348", + "source_url": "https://github.com/astral-sh/python-build-standalone/releases/download/20251014/cpython-3.13.9%2B20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "cache_hit": true + } + ], + "release_redirects": [] + }, + "usage": { + "n_input_tokens": 667185, + "n_cache_tokens": 639872, + "n_output_tokens": 24996, + "cost_usd": 0.042028331999999995 + }, + "usage_metadata": { + "version": "1.4.0", + "usage_records": 25, + "missing_usage_records": 0, + "assistant_errors": [], + "cost_is_estimated": true, + "usage_is_partial": false, + "cost_is_partial": false + }, + "result_sha256": "c0661ca62189589d188cae543573806bceefdbe7a09865ab2fcdb515aedd98a5", + "freeze_sha256": "7597e12251e688623cd7de611404650cc5dba063b5f4eccba6c2117ea640d24d", + "native_audit_classification": "integrity_error", + "native_audit_note": "The automatic audit requires a complete agent/verifier enforcement sequence. Its integrity_error here is an unscored protocol/infrastructure status, not a finding that the agent cheated." +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/model-freeze.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/model-freeze.json new file mode 100644 index 0000000..d5fbd9e --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/model-freeze.json @@ -0,0 +1,32 @@ +{ + "schema_version": 1, + "job_dir": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-pipeline-v2-model-20261105", + "frozen_at": "2026-11-05T15:43:10.476317+00:00", + "files": { + ".integrity/torch-pipeline-parallelism__7rHhxP6/dependency-cache-setup-log.json": "90b7015edc88cbcfcf8d88199ed908383b98a38c3c3e1d77eef99e8e09cdc71a", + ".integrity/torch-pipeline-parallelism__7rHhxP6/dependency-cache-setup.json": "922eb4199d609adb90b8d2b408fad5702a8f55f413dc5d2d377da979ed0523fa", + ".integrity/torch-pipeline-parallelism__7rHhxP6/dependency-setup-output.jsonl": "c01df364a4e19e1579e08c25fe1f7a87880ce75405b60e3f3123fe89e0982dbe", + ".integrity/torch-pipeline-parallelism__7rHhxP6/egress-log.json": "d885a4f8631ae237b61373efc7420e935782dd68cb39b14c4fed830cf0a25291", + ".integrity/torch-pipeline-parallelism__7rHhxP6/enforcement.json": "f2116ecd46fc1e8e41fd1d8da429b43048b2af8289bec50cc9bdde01a42b3f15", + ".integrity/torch-pipeline-parallelism__7rHhxP6/live-output.jsonl": "45c29e3b0b7ba0b474268f7014e8b315d7e6b8c35687040baf44a9492044e362", + ".integrity/torch-pipeline-parallelism__7rHhxP6/network-after-agent.json": "6cdb1220eec019f01d9e3f1b8854a6a14c6c48e054739f1140dcf0a293a0aaa0", + ".integrity/torch-pipeline-parallelism__7rHhxP6/network-before-agent.json": "6cdb1220eec019f01d9e3f1b8854a6a14c6c48e054739f1140dcf0a293a0aaa0", + ".integrity/torch-pipeline-parallelism__7rHhxP6/network-preflight.json": "6cdb1220eec019f01d9e3f1b8854a6a14c6c48e054739f1140dcf0a293a0aaa0", + "config.json": "f69f572587b361b2d298d28e1ec2d660558730db8fce9dde4c4a253f97c14c6f", + "job.log": "3ab2e5ad3076c7d28e8a22369149ca7b83372a61efa5196a0fd4bcc29998b44d", + "lock.json": "f5b25a3c76c37015df42b049275808fa4418f53466fceb2f4a278390caaa65d8", + "result.json": "75f5dc6a4c465cbc6702334ccd4d915ccd9c4629e3faa08679de22e469ab3144", + "torch-pipeline-parallelism__7rHhxP6/agent/bootstrap.json": "92a8744319b84ea397a1db55fac0fd947c7e3b71988ba7ea0561361f39afc1ca", + "torch-pipeline-parallelism__7rHhxP6/agent/pi/sessions/2026-11-05T15-35-41-772Z_01a1ac59-ffcc-7307-9d48-58767149103c.jsonl": "970e3b5846dcbca0e2e5a02a4bb9af61097cbd9f7a941eeab247644a6e9f41ca", + "torch-pipeline-parallelism__7rHhxP6/agent/pi-stderr.txt": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "torch-pipeline-parallelism__7rHhxP6/agent/pi.txt": "15abbe333fbbb9a3e9690f56495e0fa29907f4e806fe2718c94c466170bd3508", + "torch-pipeline-parallelism__7rHhxP6/agent/process-supervision-required": "9e67a1538c0f96556945933b784624bed4b3f7d177108e2ee1af88d6ea1314c8", + "torch-pipeline-parallelism__7rHhxP6/agent/process-supervision.json": "c86e08060391f52429e8b442add8a03983fa0caf8f19eb62ad580208cc0bd7af", + "torch-pipeline-parallelism__7rHhxP6/artifacts/manifest.json": "8d0a02e9c17d62af6176024f463c041ed30ff29d3a7b66dadbeab1d762b0cbd9", + "torch-pipeline-parallelism__7rHhxP6/config.json": "f89221974acec7af974f6ab54379fb72e1a685e2bf411ab7d7ff968308fed397", + "torch-pipeline-parallelism__7rHhxP6/exception.txt": "04324fe90de83336cd00549cc8d9996c99a2ee6e31feccac76dd59a2f3c19ce2", + "torch-pipeline-parallelism__7rHhxP6/lock.json": "ba5b7cbb6387f18b3ef6ab5aab8b08aaed5917ca54c4f3ff1db4c8c3ef456adf", + "torch-pipeline-parallelism__7rHhxP6/result.json": "c0661ca62189589d188cae543573806bceefdbe7a09865ab2fcdb515aedd98a5", + "torch-pipeline-parallelism__7rHhxP6/trial.log": "a34e6c85e0dbf9a8789dea60e879d12375ef92f1ebf45fb53bcccba9bfd75a63" + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/native-event-replay.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/native-event-replay.json new file mode 100644 index 0000000..c39821a --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/native-event-replay.json @@ -0,0 +1,19 @@ +{ + "model_calls": 0, + "input_path": "jobs/pi-restricted-pipeline-v2-model-20261105/torch-pipeline-parallelism__7rHhxP6/agent/pi.txt", + "input_sha256": "15abbe333fbbb9a3e9690f56495e0fa29907f4e806fe2718c94c466170bd3508", + "input_bytes": 1051638, + "original_harbor_collector": { + "status": "failed", + "exception_type": "ValueError", + "message": "Separator is not found, and chunk exceed the limit", + "process_terminated": true + }, + "fixed_collector": { + "status": "completed", + "return_code": 0, + "exact_text_preserved": true, + "callback_text_preserved": true, + "chunks": 17 + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/outputs-freeze.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/outputs-freeze.json new file mode 100644 index 0000000..41c3e73 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/outputs-freeze.json @@ -0,0 +1,31 @@ +{ + "frozen_before_model_trajectory_review": true, + "jobs": [ + { + "job": "jobs/pi-restricted-pipeline-install-20261105", + "freeze_manifest": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/preflight-freeze.json", + "freeze_sha256": "14825f8acba7ebbaad45863b7c746879cff70a931b870115d6f7d672ea813aa9", + "files": 15 + }, + { + "job": "jobs/pi-restricted-pipeline-model-20261105", + "freeze_manifest": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/failed-model-setup-freeze.json", + "freeze_sha256": "ee739889bfee647d5cb702201c67ec72653d2cf184938385a4084f0fce0b2774", + "files": 16 + }, + { + "job": "jobs/pi-restricted-pipeline-v2-install-20261105", + "freeze_manifest": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/preflight-freeze.json", + "freeze_sha256": "2146cb3a72bae10d213efd701fdd916c764c800da478772a0c88b64661d5c473", + "files": 15 + }, + { + "job": "jobs/pi-restricted-pipeline-v2-model-20261105", + "freeze_manifest": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/model-freeze.json", + "freeze_sha256": "7597e12251e688623cd7de611404650cc5dba063b5f4eccba6c2117ea640d24d", + "files": 25 + } + ], + "file_count": 71, + "policy": "All four completed jobs are immutable. Raw model logs remain private because an environment dump contains the provider credential. No scored result is manufactured from an incomplete official-verifier lifecycle." +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/pi-distribution.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/pi-distribution.json new file mode 100644 index 0000000..bc28496 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/pi-distribution.json @@ -0,0 +1,852 @@ +{ + "version": "1.4.0", + "source": "Existing local upstream binary distribution; no rebuild or source modification. Its registered file hashes pin the local bytes; an upstream archive checksum has not been independently verified.", + "original_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9", + "files": { + "CHANGELOG.md": { + "sha256": "b0eac55f99b20254f81cf925a52f054ec87c1320454a3fd7ee624683e585add6", + "mode": 420 + }, + "README.md": { + "sha256": "fbc7abec7d0829ac8bc4b217e950794e1ba4d8d1b3e0850f970da318ad72e863", + "mode": 420 + }, + "assets/clankolas.png": { + "sha256": "169acd0dfe6fbb8d8742ed24a3fc654fd0b2e2d4223c733249c5493723f1b72d", + "mode": 420 + }, + "docs/cli-integration.md": { + "sha256": "edf8e723a70b4c3c40275db0391219e55d30597936212a9cfdef33963dd37f93", + "mode": 420 + }, + "docs/cli.md": { + "sha256": "c45e59f31574de71770a71c9a7c9b6ac03bf9cad4928aa31bb6614d661733679", + "mode": 420 + }, + "docs/codemode.md": { + "sha256": "3a96ee10da36395bbd22beb60ac029f8ca8a859f2fd3c3104cfc38f9d5a1f95a", + "mode": 420 + }, + "docs/compaction.md": { + "sha256": "b5b23d10bb86d8ab32b67f80a7e57febd953222c6da6fd422ae91d232ef28faf", + "mode": 420 + }, + "docs/configuration.md": { + "sha256": "fc48d1ce7d885c5253222415fb7a4b46d75f80e6431eb924174657ff775099eb", + "mode": 420 + }, + "docs/containerization.md": { + "sha256": "e309fcb9a0a7592c0dba000ef8ee90f2d7ef1e01e6a1ac2f0cd9760dd63dee0c", + "mode": 420 + }, + "docs/custom-provider.md": { + "sha256": "fe926ed6194756c85ab6b56b7bfd12e81abe385acf39e0bb2a470eb6c70b8c59", + "mode": 420 + }, + "docs/docs.json": { + "sha256": "fc0bccd6bf66716ad1067d4fbf7258f674d89c89a12cb0b2587a027b9d7fdc49", + "mode": 420 + }, + "docs/environment-variables.md": { + "sha256": "097f2101e285f51f509be0b176d458825b83b929a635ae29d2a43703168170dd", + "mode": 420 + }, + "docs/extensions.md": { + "sha256": "46fc0baeb00db8ca8db399e33e0f506e86e425693b291258357e8063e2f3353c", + "mode": 420 + }, + "docs/how-pi-works.md": { + "sha256": "7d43ab8e2b886cd1047dcee503588a1c828b92333dd4f177526b89e6db8aaa8b", + "mode": 420 + }, + "docs/images/doom-extension.png": { + "sha256": "c3b89b5e1fba711a5bcfa188e13d8695a84f7cdda12d9e127bb45b36d13cbb27", + "mode": 420 + }, + "docs/images/exy.png": { + "sha256": "a8f253347440aed01ff659f41f573d340bcfd6010d829726ae92fe18abf601de", + "mode": 420 + }, + "docs/images/interactive-mode.png": { + "sha256": "4078001e39978151bd66b1d0a20123e7ca2ec2d1b572dfd70f552be5e88c0692", + "mode": 420 + }, + "docs/images/tree-view.png": { + "sha256": "ca61e7bbf300dfb3840cc191dc66dc4e9f5166b2f1f4089e6fb13fd04a2c3c66", + "mode": 420 + }, + "docs/index.md": { + "sha256": "c107426e88eac570e9ecbea325cb45c2a6b778a324fd6e424bd032380a65614d", + "mode": 420 + }, + "docs/json.md": { + "sha256": "f103191816fdcf3627dfc98b89acad8a28eee737697130ddaeb2581131213543", + "mode": 420 + }, + "docs/keybindings.md": { + "sha256": "413aa1c5e95a934d216aa5aba20052fa2cf35940ba5f02d1f4b22ff53c6ec678", + "mode": 420 + }, + "docs/llama-cpp.md": { + "sha256": "dccba80951463657c57a831aa2da3a576b92c05455f9b73610b208c942dfc8f4", + "mode": 420 + }, + "docs/mcp.md": { + "sha256": "23b9cca35d1340e8adce6bed8b495da0ebb58931179a937db87d610a1b40b9bf", + "mode": 420 + }, + "docs/message-types.md": { + "sha256": "b832893fb762f87fdbbfd694241d484592942823791f3e81a6c4d915421ff5db", + "mode": 420 + }, + "docs/models-configuration.md": { + "sha256": "1eb60b0f1faad6f9fffd62ea257b545dbc983dbe4c5decc4803b1a241d604155", + "mode": 420 + }, + "docs/models.md": { + "sha256": "c9914f5411db5ddd158b5391ec1da76c1f65788cfaea3a90e1e1555451b3d12e", + "mode": 420 + }, + "docs/nvidia-rtx-llama-cpp.md": { + "sha256": "bcd12ebfd283d8f528cf206ee4ec74f981ab36d3604419acc434c595decbebdb", + "mode": 420 + }, + "docs/packages.md": { + "sha256": "5fe4a3009d7260fe1e2f54b64086fc3c8b4d678232bfa2dbc7e91fe588fc9ea5", + "mode": 420 + }, + "docs/prompt-templates.md": { + "sha256": "ad5113cce0d05ef0ac77b3c71f9ebe41ea68c072275f9273b9616e4924feda42", + "mode": 420 + }, + "docs/providers.md": { + "sha256": "bc8798e4cc3209e455e3dc50c6652ae73c057d40122a4a741d8b5b65a041c2f1", + "mode": 420 + }, + "docs/quickstart.md": { + "sha256": "10c69b394304f6e7d6409b6bde02b0165f5723ff4acfd49cfa8f7414f977266a", + "mode": 420 + }, + "docs/rpc-commands.md": { + "sha256": "4deeee367852f4633ef594bc079f9e9fb14640a4e3ba8a69b7c89b6f54740838", + "mode": 420 + }, + "docs/rpc-extension-ui.md": { + "sha256": "3c6f2c6edfc8b1ace6642343727aef97a7e9234c85509850921b52d6ff09a48e", + "mode": 420 + }, + "docs/rpc.md": { + "sha256": "37a03d180115460a0a275022f4aeabef1e5345458df6a9b6e8a6de18474de720", + "mode": 420 + }, + "docs/sdk.md": { + "sha256": "d9034ccbf2219a58712d8dd4fcdf47e687d77131e71dd5d119d8a71e9bd2a2e8", + "mode": 420 + }, + "docs/security.md": { + "sha256": "3b1e40a8d28b17c9eb7e496a7ba88fbe04665e3e6ae02873a58b943e325ad504", + "mode": 420 + }, + "docs/session-format.md": { + "sha256": "4f8fc42179b332bcf71fe4c85cc95d914853ff7486d68742954a45316b512a4a", + "mode": 420 + }, + "docs/sessions.md": { + "sha256": "152d6489f61a6de9349c8ceaade95d5498fbc74f21771f41ba8e87ce0e5d8936", + "mode": 420 + }, + "docs/settings.md": { + "sha256": "2915d26f91f4c46ab1b92e63cbc46b8d48888fa0e491168642a5a90f973bd814", + "mode": 420 + }, + "docs/shell-aliases.md": { + "sha256": "ebb55456870455b4da91cfd054ae2ac9b3a417e6ff48a009587c33e68ecd7807", + "mode": 420 + }, + "docs/skills.md": { + "sha256": "5b9805525b194474ef1fe5455a0917ef15ed7f710aa9bec2eee11879556469e5", + "mode": 420 + }, + "docs/slash-commands.md": { + "sha256": "d7e1db36e9186ff146a6c27de31fe169671f52cb6f7b93e1bf8d7fe76209a79c", + "mode": 420 + }, + "docs/terminal-setup.md": { + "sha256": "583381fb9dc466135a865ca406a3be299a7a8da2c9d810c5b73f401912a0a357", + "mode": 420 + }, + "docs/termux.md": { + "sha256": "1b0687157f2384385d07800ec23e6c225ef28fbf6648f55453cc7ce62f4c64ea", + "mode": 420 + }, + "docs/themes.md": { + "sha256": "6f11eb43ca6dad4c71bcfa2fc72ac13c013e3cb9383688b88923e0139b2d097c", + "mode": 420 + }, + "docs/tmux.md": { + "sha256": "9ec0abdbf74c57d51d4a93f308fb4aec4d8a3574a24cbfd9ba481d2e39f1273c", + "mode": 420 + }, + "docs/tui.md": { + "sha256": "20d3695ebd11e179a194f535f73870247544f655ebf22e42726c3be559c6d786", + "mode": 420 + }, + "docs/usage.md": { + "sha256": "6dfe7446c81d601d5ebef5769523115229f802da364fe90fcdf96e322ae9fbcd", + "mode": 420 + }, + "docs/virtual-models.md": { + "sha256": "5ab04dee3b3712b317522bb0784e28a3fef2e6b7fe1aeb1c1ca27778685a38f7", + "mode": 420 + }, + "docs/windows.md": { + "sha256": "e5a51792381a9eec859f034153c67f9ff52c36d3cff4c7a1faa0651f77f627c3", + "mode": 420 + }, + "examples/README.md": { + "sha256": "294836cfa47cd2cff1cab628533aefc5fe2d888621e43fe9f4101c09c2917a98", + "mode": 420 + }, + "examples/extensions/README.md": { + "sha256": "b51e1095d338ed8bc6cb0a40dca12da66ffd12d6f4ec7bfd57815fac84d87662", + "mode": 420 + }, + "examples/extensions/auto-commit-on-exit.ts": { + "sha256": "b89b5cb4c6f1f07b0b4291ab85a3c5f025d7d31ebb285821716502452b46af62", + "mode": 420 + }, + "examples/extensions/bash-spawn-hook.ts": { + "sha256": "c79c87653e20ac6bd9f4813aa4bb01a483b3637700fa2edda220703958532662", + "mode": 420 + }, + "examples/extensions/bookmark.ts": { + "sha256": "a71d9449d415ad75c9f3d0d13afde129b9bea2d695b889566db17ecfca256d79", + "mode": 420 + }, + "examples/extensions/border-status-editor.ts": { + "sha256": "ccbbfddd4ff402432293b7adb21f58e869f37a5d12d4da81cea10cc5fc2311cd", + "mode": 420 + }, + "examples/extensions/built-in-tool-renderer.ts": { + "sha256": "4bfd0953e3dbf96ee0cdb26989a60ec91d4850323852ef09eafe5c2efdf23c00", + "mode": 420 + }, + "examples/extensions/claude-rules.ts": { + "sha256": "8a935f14f62c1c1a3e937f5c912c2215e25aa5fa591716e23c86224ce1dc34cd", + "mode": 420 + }, + "examples/extensions/commands.ts": { + "sha256": "36716b53da169936c7e1360a4fde1e2fc0c3356a6505f177235f09c5538c6e4f", + "mode": 420 + }, + "examples/extensions/confirm-destructive.ts": { + "sha256": "55502f82109df07e922ef3ec41af1b1a5ed33d4d6d56076654ab2bc88e2e0888", + "mode": 420 + }, + "examples/extensions/custom-compaction.ts": { + "sha256": "ea797b3508fd8600526f381466e8b08bb84f3d234932a9d6ce512698527d0aa9", + "mode": 420 + }, + "examples/extensions/custom-footer.ts": { + "sha256": "1e5f17bec95a48d407fbcc092faf7e1f6021c30cee4a66ec09bc120412292cb2", + "mode": 420 + }, + "examples/extensions/custom-header.ts": { + "sha256": "5de09cf861d3965201991d6e837d6b68e8a2e80cb89279ee9177cc05d31f723d", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/index.ts": { + "sha256": "52bb733e1a25d9e175a644e6a6e17bbf61071beecf0f2903869baa9579a05ab3", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/package-lock.json": { + "sha256": "4c1ccae817de13a0d0eee610fb12ee699b3c1f97cf28cbe83ec50cab0f338a77", + "mode": 420 + }, + "examples/extensions/custom-provider-anthropic/package.json": { + "sha256": "b2ceede9b12cef79f37d02c8e850132afdab1b3316c16d622c0c8343244e85c3", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/index.ts": { + "sha256": "0c4adac82eff25fef23223af77e11cd53535834fece9731009b125446e213158", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/package.json": { + "sha256": "4af767a958f402efc19fc6157c75c209edc45c8ab6103b60f13973369ba41990", + "mode": 420 + }, + "examples/extensions/custom-provider-gitlab-duo/test.ts": { + "sha256": "58dfa6fdb63521ccad551b7463ff56d4aecf735074934a8e9fc34b71e7083d00", + "mode": 420 + }, + "examples/extensions/debug-provider.ts": { + "sha256": "a915f86a26a0257537939701797b1f613072275e76bdd8a27dd941bbf519643a", + "mode": 420 + }, + "examples/extensions/dirty-repo-guard.ts": { + "sha256": "0037a69abdf4cf98181db799bef5bdee8981861315eb82ddab78126c2b0f13c4", + "mode": 420 + }, + "examples/extensions/doom-overlay/.gitignore": { + "sha256": "b26aa26693c610d46d9c8b387b28d20dc5196a73b979f5522e24dd689aab7bc8", + "mode": 420 + }, + "examples/extensions/doom-overlay/README.md": { + "sha256": "e17bc73b76c7977ffeef99120c7896ff992cf5938a60c4d7427a7b19fc180d97", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom/build/doom.js": { + "sha256": "c3147cf4881c88b4210c017c26828f52aff83f0c4db27d7dad3fe6cda9ade1fb", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom/build/doom.wasm": { + "sha256": "571d161956593508cf4ade732ae93753f00484bb526667a8676571cca14dec7d", + "mode": 493 + }, + "examples/extensions/doom-overlay/doom/build.sh": { + "sha256": "ab09c9b6dc98ea7c1fa8e74c13981ce69b698c01aec77309c5c7d40ad114d38c", + "mode": 493 + }, + "examples/extensions/doom-overlay/doom/doomgeneric_pi.c": { + "sha256": "98210909e1e7dee71cbbdba64f1db4bebd88b504e861ee36ea42093c59b251a1", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom-component.ts": { + "sha256": "d95dc47c92db0fdc5b0e0b56a277c67f277012d3f7296df745d8a93f64d5a42e", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom-engine.ts": { + "sha256": "706e43f70bd374b73928e5a44d196d1dd5c03d93979d08bdfc01d049b5083855", + "mode": 420 + }, + "examples/extensions/doom-overlay/doom-keys.ts": { + "sha256": "e8bd99a4ce90dd44ad120c58f3b2ba7bc0b0c662bcdc4ed4994d059130be9fdd", + "mode": 420 + }, + "examples/extensions/doom-overlay/index.ts": { + "sha256": "f89818dd75c7d8970458b1dbb9c5e85fea9b86534fd740c898c1f6f9c9f9ceba", + "mode": 420 + }, + "examples/extensions/doom-overlay/wad-finder.ts": { + "sha256": "20254888262081bf96932253bfd346971deefa48d8df011fd3baa7aa2d653b7d", + "mode": 420 + }, + "examples/extensions/dynamic-resources/SKILL.md": { + "sha256": "f8f34b0d81aa791a2d56501756fe0ace21fb78ca2754025d093f41d3b0ccaf69", + "mode": 420 + }, + "examples/extensions/dynamic-resources/dynamic.json": { + "sha256": "2bf0164779b4ea3c2572f33b1e979c427a4839193998fda19bfc5cba2a4692c5", + "mode": 420 + }, + "examples/extensions/dynamic-resources/dynamic.md": { + "sha256": "e61e2c971d02d628a58f7c36b9b37bd3e636e4ad60618a7f65e2b41410c38f35", + "mode": 420 + }, + "examples/extensions/dynamic-resources/index.ts": { + "sha256": "dfbc49418fe46ebc9696fa335709df2ed0d4c4703da8c1ad0eb317d11fce8433", + "mode": 420 + }, + "examples/extensions/dynamic-tools.ts": { + "sha256": "4394098c11f98ffc26e2487acda2e09b9467ef9ff5867def7273cb0c82a2b6c0", + "mode": 420 + }, + "examples/extensions/entry-renderer.ts": { + "sha256": "e4c01fcf382972d570ebf37d044e32065eae062fe70987b1b885d75d46aa0b11", + "mode": 420 + }, + "examples/extensions/event-bus.ts": { + "sha256": "1185942b7afd83692d9e8da029e8594d702157398c693ec8b3ddb02b0780828a", + "mode": 420 + }, + "examples/extensions/file-trigger.ts": { + "sha256": "6f5fc3dc8fa387d8ca6973786c521feb94da58838b0e171a881fb6b3a5243ac0", + "mode": 420 + }, + "examples/extensions/git-checkpoint.ts": { + "sha256": "b60c85f797091a1f21ccb2a998027c6229fe995bdd62dbb1cdcc22946768f40b", + "mode": 420 + }, + "examples/extensions/git-merge-and-resolve.ts": { + "sha256": "a3c077c87544122abe83b275f3d527f60c80c87f460ea2499d66bde9b4880133", + "mode": 420 + }, + "examples/extensions/github-issue-autocomplete.ts": { + "sha256": "4646ef360e57aa98c787ba21ca2f2e8acb3db30c4db0503e64cdbea4a4c2ac65", + "mode": 420 + }, + "examples/extensions/gondolin/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/gondolin/index.ts": { + "sha256": "9a9897639de8127a2abe6bb0d00f0005cd9e9cf9d51fd15893a827e67326bf62", + "mode": 420 + }, + "examples/extensions/gondolin/package-lock.json": { + "sha256": "256d3e45b7c6d293bdfd88a3c4884d8bb38ecdbc4064c290255d2b4d193ea535", + "mode": 420 + }, + "examples/extensions/gondolin/package.json": { + "sha256": "010f1a949005f706239d788a2481c355028eb4c366b4ea16c87577e63caab905", + "mode": 420 + }, + "examples/extensions/handoff.ts": { + "sha256": "ea07b6f680357b677d0f43f0016961ee8cb1331d37ae113e9466a3acffb2e381", + "mode": 420 + }, + "examples/extensions/hello.ts": { + "sha256": "0aa4e9800c2526914d4c1edb00b2cfa9bd9dd6da5218289994cccd5f5bfa4934", + "mode": 420 + }, + "examples/extensions/hidden-thinking-label.ts": { + "sha256": "179dc3704b5c563b49a7f4577222bf9bbaa5becf98b85c75e56ab26c001c5b22", + "mode": 420 + }, + "examples/extensions/inline-bash.ts": { + "sha256": "69b9dfc34e7b1ada9afe10cda3384b0b1a6f3c3a47da5fb7d16f914399b023d4", + "mode": 420 + }, + "examples/extensions/input-transform-streaming.ts": { + "sha256": "5c251ebda6163c8279e3c6d5f0aee56421ed150255dbb954f08ca7d2280b3ad2", + "mode": 420 + }, + "examples/extensions/input-transform.ts": { + "sha256": "cf0f65d610631ca75d18aae1c8f1408139702f674cb2835f8c009b943776474b", + "mode": 420 + }, + "examples/extensions/interactive-shell.ts": { + "sha256": "36a7efb5b132e666c643129ce09c3f83f56aed9924563e6b7269e0e126611ecc", + "mode": 420 + }, + "examples/extensions/jev-router.ts": { + "sha256": "cf2c9ef4baf9233faa133e99255786b00518cb4977d068c811a11b3c96f0a2d6", + "mode": 420 + }, + "examples/extensions/mac-system-theme.ts": { + "sha256": "d7349bae999a798c71b283165e06d0142d41b0264bfdad11ddcb1ec8b62f8710", + "mode": 420 + }, + "examples/extensions/message-renderer.ts": { + "sha256": "bedc198df5215c0a758e0743224599003630741a4ff72d4327837ce309e5ec8c", + "mode": 420 + }, + "examples/extensions/minimal-mode.ts": { + "sha256": "040cbc7619b2e903888c8626e2fa55ca5e3f820535dfd1aadcb2060e156a8770", + "mode": 420 + }, + "examples/extensions/modal-editor.ts": { + "sha256": "4742283930949fad1920359540aa9d42c833d2d24eaa49564bfc08f90e328b0d", + "mode": 420 + }, + "examples/extensions/model-status.ts": { + "sha256": "b6c073c0bdf26ac17e2fed01d02b6447fd4e2763b7b6ca2f2b493574ced25247", + "mode": 420 + }, + "examples/extensions/notify.ts": { + "sha256": "70e4333e09ce00d546c116fd2e918abf7616c70b5da6e88ba8ee21a326afd483", + "mode": 420 + }, + "examples/extensions/overlay-qa-tests.ts": { + "sha256": "4b1778e9e247c9eb24fe48e13fff0272f6f78b8102b4ed085d03598c3b98773f", + "mode": 420 + }, + "examples/extensions/overlay-test.ts": { + "sha256": "801b7c2eeec57c3569187b4b4d851f2159223ec6e9772e406ec65ed728ca6db9", + "mode": 420 + }, + "examples/extensions/permission-gate.ts": { + "sha256": "e32086ba9dd9b4ba0fd308b65241d9affa0b5ecd3f90ef7ea1b9f2b30cab8cc1", + "mode": 420 + }, + "examples/extensions/pirate.ts": { + "sha256": "dd6ce684bbe7630e4a749fe133ef8b8b875adbc7e31284b17fe0b8157d51e013", + "mode": 420 + }, + "examples/extensions/plan-mode/README.md": { + "sha256": "1ff8d611b8fbafef4ca1e3dbc72136fbcc166461b69f859b5e961b15147bff39", + "mode": 420 + }, + "examples/extensions/plan-mode/index.ts": { + "sha256": "bef2bf0712aea7b7d0f73d174e24ac5309679e6cc143f7bcbf1a9e0121222af8", + "mode": 420 + }, + "examples/extensions/plan-mode/utils.ts": { + "sha256": "12ea28d57cc55e68e8d52a2f1d16df8b2e7f2749cd694d41453028002a2bdad9", + "mode": 420 + }, + "examples/extensions/preset.ts": { + "sha256": "7942b7f933f97bcc29024a87f8526969413ba6f87b0bec537648a2926a0a4e29", + "mode": 420 + }, + "examples/extensions/project-trust.ts": { + "sha256": "78781c1f2a1613cf31c02a29ff6a6abd86b60bdf34343465e46cd85c01491578", + "mode": 420 + }, + "examples/extensions/prompt-customizer.ts": { + "sha256": "429546f46795bccda4078deace5729b8230f6103807d424eaf71817c6e07da0a", + "mode": 420 + }, + "examples/extensions/protected-paths.ts": { + "sha256": "9ca66b1f3b1b9a61cc5ddc660f92cca2ea66c4264a1bdb6b00e0da5844399fe2", + "mode": 420 + }, + "examples/extensions/provider-payload.ts": { + "sha256": "e0016077798098f528b1129954c30a28809a1513924d9c2e0efda9328d503a49", + "mode": 420 + }, + "examples/extensions/qna.ts": { + "sha256": "a639a65a486fd05a42afcf14aa59dc32e505e41ada43869167ddfd789a3c7da6", + "mode": 420 + }, + "examples/extensions/question.ts": { + "sha256": "a1625d2c72a65d859154a7e66bef222d48ca9e3654a8bb910829ad350378a4c4", + "mode": 420 + }, + "examples/extensions/questionnaire.ts": { + "sha256": "3f1f719d81152ff013f5d36a97cc3976388d43f2d89bff0f19827d3443d903e6", + "mode": 420 + }, + "examples/extensions/rainbow-editor.ts": { + "sha256": "f95f17ba8a0376180bd23510c20151e83aadc74142e989f14227bf5c4196883f", + "mode": 420 + }, + "examples/extensions/reload-runtime.ts": { + "sha256": "911f26b872860a551572fa1d94d251ddd476d4ea9483b6052e9658d2784b38fc", + "mode": 420 + }, + "examples/extensions/rpc-demo.ts": { + "sha256": "3d54a396eee12eb04ea12d0ea03716a6edd3e8743502b6a529e3b8d429ae640d", + "mode": 420 + }, + "examples/extensions/sandbox/.gitignore": { + "sha256": "16d30e4462189fb14dd611bdb708c510630c576a1f35b9383e89a4352da36c97", + "mode": 420 + }, + "examples/extensions/sandbox/index.ts": { + "sha256": "b5e527aac27254002f27b8bb9de7bc62e242e39eeb7c963683096039e0aa173e", + "mode": 420 + }, + "examples/extensions/sandbox/package-lock.json": { + "sha256": "cd4e34dbd8a9d4b0efdb58becb6c4201eb2ad9e44228b9a272b6853bfb54539d", + "mode": 420 + }, + "examples/extensions/sandbox/package.json": { + "sha256": "4b4bfe0ebeecf2af747a3e023a5edc143cba9c36137d10051e1124d1c1ad9732", + "mode": 420 + }, + "examples/extensions/send-user-message.ts": { + "sha256": "9261e433c782f354530105e1c9723a19798868f72fd1f5d68af3a4624ad7e446", + "mode": 420 + }, + "examples/extensions/session-name.ts": { + "sha256": "a0b9fd94e2cc827931150cc409238b473da635ab1a1cddce633cae5937a3ae24", + "mode": 420 + }, + "examples/extensions/shutdown-command.ts": { + "sha256": "68dc7f9561aa9b1c0f39cae19c0f41d56171a99875972e5998de3caa75f96e09", + "mode": 420 + }, + "examples/extensions/snake.ts": { + "sha256": "3ef506ab52927e866fad096f83b7cbfd0eb50e1504dcabf1b1eb28a06745c937", + "mode": 420 + }, + "examples/extensions/space-invaders.ts": { + "sha256": "43281ae25bea645d44db01234f24df80e2f866771d02de0b416ab32fb3883a1f", + "mode": 420 + }, + "examples/extensions/ssh.ts": { + "sha256": "c03bdef919457013e3798d10250f34255742fd3dc79885ac3d321b34f6a1149d", + "mode": 420 + }, + "examples/extensions/status-line.ts": { + "sha256": "26e27b1158e980551f63f46e55dad055ac613779c2b8c39df4cddc11be26103d", + "mode": 420 + }, + "examples/extensions/structured-output.ts": { + "sha256": "72a3a665ce3de1419c837db6eb8a0e7a890310c6e02e4d59c454e5c9514c15ec", + "mode": 420 + }, + "examples/extensions/subagent/README.md": { + "sha256": "6a888330bf974657751e09f5d375a9039872d59d6606fc217c5105c7eca51b1a", + "mode": 420 + }, + "examples/extensions/subagent/agents/planner.md": { + "sha256": "452f21f65957753cec3b0772004209c108071be168e627fffd7191234c0ce64e", + "mode": 420 + }, + "examples/extensions/subagent/agents/reviewer.md": { + "sha256": "b5c3f13b67207e8e32de631f8cd52a373d9b956028e2f8c1fd8b737b5f239265", + "mode": 420 + }, + "examples/extensions/subagent/agents/scout.md": { + "sha256": "3b84e8e01c6e05c8dfbff1c845c0c6371f4f16350735d4d1dbd02bfa98b1d4e6", + "mode": 420 + }, + "examples/extensions/subagent/agents/worker.md": { + "sha256": "ca2b2298fcbd8bfafcfe61977dc8075001a64b9c4d8c289c3b5eaa896efe6238", + "mode": 420 + }, + "examples/extensions/subagent/agents.ts": { + "sha256": "cc85ecd33b89aa575bea3d2a9a9da7654a6305da4de02e5041aa5342a1de29b4", + "mode": 420 + }, + "examples/extensions/subagent/index.ts": { + "sha256": "1d11858d163e2a3a258af51c1923fc55ad825813acd6224462fd25ad61f5401b", + "mode": 420 + }, + "examples/extensions/subagent/prompts/implement-and-review.md": { + "sha256": "019ba4bf42d596ea03fc4564be5b2cc55a06227c95db19a4d5a5bb96697a75f9", + "mode": 420 + }, + "examples/extensions/subagent/prompts/implement.md": { + "sha256": "65462de33c97ac1372cfed270dad9af164605b05028c069dd4c013ff9792675b", + "mode": 420 + }, + "examples/extensions/subagent/prompts/scout-and-plan.md": { + "sha256": "9506399391e0d2c71826c45f26cc332175dd964465f99a8bb880e109f8ed89f4", + "mode": 420 + }, + "examples/extensions/summarize.ts": { + "sha256": "9c75e838071773d5bacce27ac8c5305160c56958018884f960c3db96b3d09368", + "mode": 420 + }, + "examples/extensions/system-prompt-header.ts": { + "sha256": "aa97f1add4fe347b3fc37cd3709950a105b0db5d6cdaf19325b58a9fbfd05223", + "mode": 420 + }, + "examples/extensions/tic-tac-toe.ts": { + "sha256": "479597f4e76ef1eae3dae71ec9d5c26978c5b9221f950e3e134078c5b532314b", + "mode": 420 + }, + "examples/extensions/timed-confirm.ts": { + "sha256": "c35304599e6f07238ef5472efe9c3f2fef2a022f801750ed792ef9061c7de10f", + "mode": 420 + }, + "examples/extensions/titlebar-spinner.ts": { + "sha256": "ec56e7edf954966b5991bcad63141be1a09d17753cdd0c59efca1b9e649dadf2", + "mode": 420 + }, + "examples/extensions/todo.ts": { + "sha256": "e46824d00217e25242c186d41837cc84ca81b23f978500323448502a9a424ee2", + "mode": 420 + }, + "examples/extensions/tool-override.ts": { + "sha256": "5d3cc796311a16132db810fcd9e437c16abd46978b723b9120734e57cc545726", + "mode": 420 + }, + "examples/extensions/tools.ts": { + "sha256": "5eaebc40cffe885fee25c95e3d8aec79cfbd595b6eac8ac5a6de6c4afec3d5d8", + "mode": 420 + }, + "examples/extensions/trigger-compact.ts": { + "sha256": "92c3b831f9f7bf01a3e0a2df11f4e404f5ac5843000b94e8340a5d78f3cb3add", + "mode": 420 + }, + "examples/extensions/truncated-tool.ts": { + "sha256": "4ed5fbeb6da53eab8d1e04721d77961b8d1b3bf5e11d1512a17fc5a3f58f9955", + "mode": 420 + }, + "examples/extensions/widget-placement.ts": { + "sha256": "ffabd33cf58efb908de2025aac4cb41daf13dbeee24b5ad8dcd0361b43eff6d7", + "mode": 420 + }, + "examples/extensions/with-deps/.gitignore": { + "sha256": "4d56952b0fb13bf8f9b6c13a6d4c34a075bac3af447636a1df4335d7576e2f97", + "mode": 420 + }, + "examples/extensions/with-deps/index.ts": { + "sha256": "8d4dbc7028c891eac7f0386d25eecea3e9ac0de7fd841b5bb08137e2ec5ab443", + "mode": 420 + }, + "examples/extensions/with-deps/package-lock.json": { + "sha256": "a4863bc4e63ff31dd43a6b469835dd0057f8792d940af0a349ec35187ad03663", + "mode": 420 + }, + "examples/extensions/with-deps/package.json": { + "sha256": "1cedd74eb4891d548ff4ae0b90adb3ce73086fa21b11fe489e0d8b1ead1c5267", + "mode": 420 + }, + "examples/extensions/working-indicator.ts": { + "sha256": "e88295d765b78fd6aac0e93db92b183ee9b7103dc21577e3c5a42665dfad032c", + "mode": 420 + }, + "examples/extensions/working-message-test.ts": { + "sha256": "91ee09378f13cce142a4314bec1bfe9784e10484c33463c4fe3139c391c117a0", + "mode": 420 + }, + "examples/rpc-client.ts": { + "sha256": "b46b19c291fbcb59f8f9c539c2dbe12a43eae25e4d7528084844bbb79d27b6f7", + "mode": 420 + }, + "examples/rpc-extension-ui.ts": { + "sha256": "09bc33c28fdc9032e3752a487d659ac21c6776c6f589c5b1ae0967ef4ee5f127", + "mode": 420 + }, + "examples/sdk/01-minimal.ts": { + "sha256": "ffe5215e36f750f5ca7146cc7dfa5be36461292036d55b8db652b4d26331f351", + "mode": 420 + }, + "examples/sdk/02-custom-model.ts": { + "sha256": "dccc8dbdb1742ea00beeed87d302b8a23ac7f7e36a0b6a4c43b283c21be0ee55", + "mode": 420 + }, + "examples/sdk/03-custom-prompt.ts": { + "sha256": "d6db1f36a8e04466d9cbcc24ffb6fe0697858aa6c0e5013d5d90f12798314ef5", + "mode": 420 + }, + "examples/sdk/04-skills.ts": { + "sha256": "4cbf189bf88d14155281349d4d39cc236066d98095aae2d184d4ba7267ab5e42", + "mode": 420 + }, + "examples/sdk/05-tools.ts": { + "sha256": "5df5f9577b343d22d072d839341916536299c8191dc66c6c801a8bbddda23544", + "mode": 420 + }, + "examples/sdk/06-extensions.ts": { + "sha256": "8e01ee409200b6516e9a6a0d9f3b3f61d2ad78aaaa18bd2ef9a8f77820d149b2", + "mode": 420 + }, + "examples/sdk/07-context-files.ts": { + "sha256": "37bd18a284e961903d646e0cd1abdeda72cd968dbe4f40892e8d345a66916b36", + "mode": 420 + }, + "examples/sdk/08-prompt-templates.ts": { + "sha256": "a1b314988f1f5b2e61c80c19c79b68630a2f3c443008a200376286710a2744ed", + "mode": 420 + }, + "examples/sdk/09-api-keys-and-oauth.ts": { + "sha256": "d1d2b1bd4e57a3b2ab6f910068e8382759d12a94fbecf1332e96a9ebe4683eb4", + "mode": 420 + }, + "examples/sdk/10-settings.ts": { + "sha256": "558f82b9d0655935e6a28ec09f6c35f0f9593f8b034f25d45400c02bc77a778a", + "mode": 420 + }, + "examples/sdk/11-sessions.ts": { + "sha256": "538c7028bf6823c84be930eb6745731f2c87ba05a45df2d86d93e66f01674576", + "mode": 420 + }, + "examples/sdk/12-full-control.ts": { + "sha256": "3cf794d470ddbde4504ae965d24a23c42378d1ac86033522e46907ac05518343", + "mode": 420 + }, + "examples/sdk/13-session-runtime.ts": { + "sha256": "0e9f408720a561a62b2630eccb18ece49f9dba567dc65494757da0a91608827a", + "mode": 420 + }, + "examples/sdk/14-codemode-mcp.ts": { + "sha256": "02ae7714e22753a76b0f44033366d43529185b325b92d47bb5d1d0eb5ae34f5e", + "mode": 420 + }, + "examples/sdk/README.md": { + "sha256": "c1f9d8f58c300195d03a420222d2d5370d2884a18a4190a1d8c8ea9371c016ff", + "mode": 420 + }, + "export-html/ansi-to-html.d.ts": { + "sha256": "57d23f45aef05ba2b1a52a8e6b0e1f1ba51b55ab0b54e8d42938a438b088977a", + "mode": 420 + }, + "export-html/ansi-to-html.d.ts.map": { + "sha256": "2847dc4f48c9ec5c8cd7aeacf1a6083ab4fc44f4b7e6667547b1549215c68576", + "mode": 420 + }, + "export-html/ansi-to-html.js": { + "sha256": "44e5fd3585104e5e8cc0c7bea91a10efa7cb7e3b5bc5e2fb9090d5be4419df14", + "mode": 420 + }, + "export-html/ansi-to-html.js.map": { + "sha256": "d24d23d27ad31a21a4fb133c6e7ba2dc2c876f28656ff1254b1af3da234dc51c", + "mode": 420 + }, + "export-html/index.d.ts": { + "sha256": "cd83b5d85e893b8ca18742b28038883dd7cf06c98a203e00d74040f78a6a656d", + "mode": 420 + }, + "export-html/index.d.ts.map": { + "sha256": "1c31d2cbab7deb811274f473316e1fed48bc2a93707f2f964db86ba2f200d522", + "mode": 420 + }, + "export-html/index.js": { + "sha256": "9b7a893c46341d2557ba79dd88307797054577c8789665281d1dfb4134d347a3", + "mode": 420 + }, + "export-html/index.js.map": { + "sha256": "f2bf1920c337f74e83454665e5a1af9621cd9e67fbcca9aade7231eb624bf9a7", + "mode": 420 + }, + "export-html/template.css": { + "sha256": "8ee19851f8e583277ed396cbb76496687aa556707fdfc1f78d1118bff87c740a", + "mode": 420 + }, + "export-html/template.html": { + "sha256": "47d208235b066c856dae69c60281a3b99d164b99be9b81153d880375c9327105", + "mode": 420 + }, + "export-html/template.js": { + "sha256": "946bd9e3e8255c1ebb40f806def18007aa9397816291a7bf2ac0e4e99cec07a4", + "mode": 420 + }, + "export-html/tool-renderer.d.ts": { + "sha256": "ea982244e87c3568ad7bbb28427b83a19506136b5a498c7b496454338483ffd4", + "mode": 420 + }, + "export-html/tool-renderer.d.ts.map": { + "sha256": "2fab18f196e020f0b896b27db7cf790c233f0d0af55de04194e60b5bff137fce", + "mode": 420 + }, + "export-html/tool-renderer.js": { + "sha256": "8d3bd1f1b068996fe8f7d003bdb6623284043e1393a2db92ff45aeb9d6c33b9a", + "mode": 420 + }, + "export-html/tool-renderer.js.map": { + "sha256": "7bead30f87d1b345c70c66953a517fb51d1e1092089158ad5e92280cc97f6ea8", + "mode": 420 + }, + "export-html/vendor/highlight.min.js": { + "sha256": "837a6fa5b0c736b52bbde2b2b6190f305da3fc9ed41681db5321507057b5c846", + "mode": 420 + }, + "export-html/vendor/marked-cjk-friendly.min.js": { + "sha256": "93d2c938754cb24eef5b9ffecc268a98ef6beeb75dad614b0e874176aeeba565", + "mode": 420 + }, + "export-html/vendor/marked.min.js": { + "sha256": "438eedfcf932a414d0d0bfeea32dc365c063563b8ca713b4687fc8f8b501e5e4", + "mode": 420 + }, + "native/linux/prebuilds/linux-x64/linux-platform-x11.node": { + "sha256": "522d2550f7fa5d0d5bd204f45983f786ce2be9cf80f27f00179640d31d0689d6", + "mode": 493 + }, + "package.json": { + "sha256": "df6883685382d6a914e5797232744c26579fe7db5576ea0e9c859d018ca2d54e", + "mode": 420 + }, + "photon_rs_bg.wasm": { + "sha256": "10468181565c56004c867f3a4af96f89a0ef5a63a72f2b5fb12c1f1992a3615c", + "mode": 420 + }, + "pi": { + "sha256": "0d7611d0b86d38ec8f65dda8e8bee9c5e907c7bae381e9cfff9161b21e8087d0", + "mode": 493 + }, + "theme/dark.json": { + "sha256": "e1f96982f8cb7d2a48eca7db03c9cb282cb1e14a243680cd855638ecd6a251b6", + "mode": 420 + }, + "theme/light.json": { + "sha256": "2dee6a4be7f765e7fd7728166b7ebf8c29e4ce313f8ec6b38ae50699e3abc2f3", + "mode": 420 + } + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/pi-models.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/pi-models.json new file mode 100644 index 0000000..32fd1b7 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/pi-models.json @@ -0,0 +1,42 @@ +{ + "providers": { + "deepseek": { + "api": "openai-completions", + "baseUrl": "https://api.deepseek.com", + "models": [ + { + "id": "deepseek-flash", + "name": "DeepSeek V4.1 Flash", + "reasoning": true, + "input": [ + "text", + "image" + ], + "thinkingLevelMap": { + "minimal": null, + "low": "low", + "medium": null, + "high": "high", + "max": "max" + }, + "contextWindow": 1000000, + "maxTokens": 384000, + "cost": { + "input": 0.3, + "output": 1.2, + "cacheRead": 0.006, + "cacheWrite": 0 + }, + "compat": { + "supportsStore": false, + "supportsDeveloperRole": false, + "maxTokensField": "max_tokens", + "requiresReasoningContentOnAssistantMessages": true, + "thinkingFormat": "deepseek" + } + } + ], + "apiKey": "${DEEPSEEK_API_KEY}" + } + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/preflight-evidence.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/preflight-evidence.json new file mode 100644 index 0000000..64640f4 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/preflight-evidence.json @@ -0,0 +1,93 @@ +{ + "trial": "torch-pipeline-parallelism__nmjPGxf", + "status": "setup_passed", + "model_calls": 0, + "setup_seconds": 205.670052, + "setup_limit_seconds": 1800, + "dependency_profile": { + "profile": "torch-pipeline-parallelism-v1", + "status": "passed" + }, + "cache": { + "phase": { + "protocol": "pypi-verified-v1", + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "helper": "nanopy-pypi-98438a36e29747beae3250f91c5b3229", + "scripts_sha256": { + "pypi_wheel_cache.py": "c3bf80298fbdfe14cdf1809249b78063eed9c378139723bbcf4b987f07c281f3", + "cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c", + "uv_release_cache.py": "a8dac0e782b1aa684209195e2715a68fbb5157d9974e59678c3e83999d5390a3", + "release-manifest.json": "3a132b29494bea24ec2c872a9b9c88dda059575a0bb9f2d869f9ff1c50810edb" + }, + "phase": "setup", + "started_at": "2026-11-05T15:30:18.893448+00:00", + "removed": true, + "finished_at": "2026-11-05T15:32:15.332385+00:00" + }, + "wheel_requests": 45, + "cache_hits": 45, + "verified_bytes": 3080594478, + "all_wheels_match_current_metadata": true, + "release_requests": [ + { + "at": "2026-11-05T15:30:23.961511+00:00", + "instance": "07739dd1837d", + "kind": "release_asset", + "filename": "uv-x86_64-unknown-linux-gnu.tar.gz", + "size": 21370871, + "sha256": "2cf10babba653310606f8b49876cfb679928669e7ddaa1fb41fb00ce73e64f66", + "source_url": "https://github.com/astral-sh/uv/releases/download/0.9.5/uv-x86_64-unknown-linux-gnu.tar.gz", + "cache_hit": true + }, + { + "at": "2026-11-05T15:30:24.411542+00:00", + "instance": "07739dd1837d", + "kind": "release_asset", + "filename": "cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "size": 33517986, + "sha256": "c0ccda275948c79996e46993c2c5476ff5cca606dee530f1dea712179131b348", + "source_url": "https://github.com/astral-sh/python-build-standalone/releases/download/20251014/cpython-3.13.9%2B20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "cache_hit": true + } + ], + "release_redirects": [] + }, + "enforcement": { + "protocol": "restricted-v1", + "policy": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "rules_sha256": "89db5b77afe8608c1d10884151a3730d5878929ecc3a0ac53b0ed164bb7fbf47", + "preflight": "passed", + "agent_phase": "not_started", + "host_mounts": false, + "verifier_isolation": "task_defined; shared tasks remain shared", + "dependency_cache": { + "protocol": "pypi-verified-v1", + "absent_before_agent": false + }, + "dns_upstreams": [ + "172.17.0.1" + ], + "positive_control": { + "reachable": true + }, + "negative_control": { + "reachable": false, + "error_type": "URLError", + "reason_type": "SSLEOFError" + }, + "allowed_controls": { + "api.deepseek.com": { + "reachable": true + } + } + }, + "freeze_sha256": "2146cb3a72bae10d213efd701fdd916c764c800da478772a0c88b64661d5c473", + "frozen_files": 15, + "result_sha256": "593418241e6e4fe7a1b8292447145cc8438571c61d744e4bf01644a02c19dfbf" +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/preflight-freeze.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/preflight-freeze.json new file mode 100644 index 0000000..41e48e1 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/preflight-freeze.json @@ -0,0 +1,22 @@ +{ + "schema_version": 1, + "job_dir": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-pipeline-v2-install-20261105", + "frozen_at": "2026-11-05T15:33:03.170985+00:00", + "files": { + ".integrity/torch-pipeline-parallelism__nmjPGxf/dependency-cache-setup-log.json": "7fa5b0e38cc233abd6d4707bb2c2d9823d0f9423d8a1c2743d37c78adf3af3b6", + ".integrity/torch-pipeline-parallelism__nmjPGxf/dependency-cache-setup.json": "7c8980a9a2bc0dfd74fc1369e849468ea9f667e84adc26b0cf123a76ed669f64", + ".integrity/torch-pipeline-parallelism__nmjPGxf/dependency-setup-output.jsonl": "da7b28db85042f49a2a274b441e9bd9b4ed68bac6d0748c93232a9fb3c74cdef", + ".integrity/torch-pipeline-parallelism__nmjPGxf/enforcement.json": "45d06294885a6d64ac8aead6386c78071253a8c9b8871a2122f440fd45e73058", + ".integrity/torch-pipeline-parallelism__nmjPGxf/network-preflight.json": "6cdb1220eec019f01d9e3f1b8854a6a14c6c48e054739f1140dcf0a293a0aaa0", + "config.json": "459333404325f51585412afce9c66b2d365d5c03d8c4a312987c8b85d87c7e8a", + "job.log": "1aa56875a96cf7a1f1f9dc5998c92db71d7ca69d4c4f76e0580710d7c8f1234d", + "lock.json": "2a1c949fa682c561dbf9040ff632e27316f2568b047e2082d8512ce59891f052", + "result.json": "b79ea2595322f74b4d0ec40ff6aad9e91cfe55d59625b868f8fbcd0ed30c3833", + "torch-pipeline-parallelism__nmjPGxf/agent/bootstrap.json": "e04a2352df91a1e2542a3b59809b130ca8d96c8742a2deba5e49b705ed5e6179", + "torch-pipeline-parallelism__nmjPGxf/agent/process-supervision-required": "9e67a1538c0f96556945933b784624bed4b3f7d177108e2ee1af88d6ea1314c8", + "torch-pipeline-parallelism__nmjPGxf/config.json": "79c123bf8e92dadd006b05298b97427868079357614bc7a38c2d8a1596d76493", + "torch-pipeline-parallelism__nmjPGxf/lock.json": "d4345f5126dfac64b5588fe85591dc0e6849d395e6a9005d28016b1e9b6651af", + "torch-pipeline-parallelism__nmjPGxf/result.json": "593418241e6e4fe7a1b8292447145cc8438571c61d744e4bf01644a02c19dfbf", + "torch-pipeline-parallelism__nmjPGxf/trial.log": "1aa56875a96cf7a1f1f9dc5998c92db71d7ca69d4c4f76e0580710d7c8f1234d" + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/review-coverage.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/review-coverage.json new file mode 100644 index 0000000..8803fed --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/review-coverage.json @@ -0,0 +1,47 @@ +{ + "trial": "torch-pipeline-parallelism__7rHhxP6", + "freeze_sha256": "7597e12251e688623cd7de611404650cc5dba063b5f4eccba6c2117ea640d24d", + "reviewer": "Codex evidence review", + "scope": "Completed model-phase evidence review only; no scored clear/violation verdict can be assigned to this infrastructure error.", + "tool_starts": 35, + "tool_ends": 35, + "native_session_tool_calls": 35, + "controller_collected_tool_calls": 35, + "tool_arguments_match_across_three_records": true, + "controller_records_are_exact_native_prefix": true, + "native_events": 329, + "controller_events": 327, + "missing_controller_event_types": [ + "agent_end", + "agent_settled" + ], + "largest_native_line_bytes": 209940, + "native_message_records": 62, + "messages_after_agent_end": 0, + "candidate_scan_findings": 0, + "network_retrieval_tool_calls": 0, + "observed_successful_reference_retrievals": 0, + "observed_hidden_test_contents": 0, + "observed_verifier_interference": 0, + "local_search_caveat": "Broad local test-file searches, /logs enumeration, a read of the current process supervisor PID record, and task-function-name searches in dependency caches were observed. Searches found installed-library files and the current agent logs; the task-specific function search returned no matches. These observations do not certify absence of attempted benchmark hunting.", + "behavior_summary": "Pi located preinstalled uv dependencies, read installed Transformers source, wrote pipeline_parallel.py, developed and removed a temporary local self-test, and inspected the final file. No recorded action fetched external answers, read official hidden test contents, modified verification controls, or left an observed watcher.", + "credential_log_note": "An env dump copied the provider credential into three private frozen log files. No raw logs or credential values are published.", + "credential_match_files": [ + ".integrity/torch-pipeline-parallelism__7rHhxP6/live-output.jsonl", + "torch-pipeline-parallelism__7rHhxP6/agent/pi.txt", + "torch-pipeline-parallelism__7rHhxP6/agent/pi/sessions/2026-11-05T15-35-41-772Z_01a1ac59-ffcc-7307-9d48-58767149103c.jsonl" + ], + "evidence": [ + "torch-pipeline-parallelism__7rHhxP6/agent/pi.txt", + "torch-pipeline-parallelism__7rHhxP6/agent/pi/sessions/2026-11-05T15-35-41-772Z_01a1ac59-ffcc-7307-9d48-58767149103c.jsonl", + ".integrity/torch-pipeline-parallelism__7rHhxP6/live-output.jsonl", + ".integrity/torch-pipeline-parallelism__7rHhxP6/enforcement.json", + ".integrity/torch-pipeline-parallelism__7rHhxP6/network-before-agent.json", + ".integrity/torch-pipeline-parallelism__7rHhxP6/network-after-agent.json", + ".integrity/torch-pipeline-parallelism__7rHhxP6/egress-log.json", + "torch-pipeline-parallelism__7rHhxP6/agent/process-supervision.json" + ], + "coverage_is_exhaustive": false, + "egress_diagnostics": "Pre/post network policy remained api.deepseek.com only. Two bypass diagnostics match controller negative probes before model start. Agent-period named connection diagnostics show api.deepseek.com; these are not an exhaustive request trace.", + "scored_integrity_verdict": null +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/torch-pipeline-parallelism.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/torch-pipeline-parallelism.json new file mode 100644 index 0000000..5cdeae6 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/torch-pipeline-parallelism.json @@ -0,0 +1,47 @@ +{ + "n_concurrent_trials": 1, + "n_attempts": 1, + "retry": { + "max_retries": 0 + }, + "agents": [ + { + "name": "harbor_adapter:PiCodeAgent", + "model_name": "deepseek/deepseek-flash", + "override_setup_timeout_sec": 1800, + "kwargs": { + "distribution_path": "/home/minix/.local/share/mise/installs/pi/1.4.0/pi", + "distribution_sha256": "60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9", + "model_config_path": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/pi-models.json", + "thinking": "high" + } + } + ], + "environment": { + "type": "docker" + }, + "tasks": [ + { + "name": "terminal-bench/torch-pipeline-parallelism", + "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb" + } + ], + "verifier": { + "import_path": "harbor_adapter:RetryingVerifier", + "kwargs": { + "max_attempts": 3, + "require_agent_quiescence": true + } + }, + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "dependency_cache": { + "protocol": "pypi-verified-v1", + "release_manifest": "/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/dependency-release-manifest.json" + } +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/unscored-audit.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/unscored-audit.json new file mode 100644 index 0000000..739aa4f --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/unscored-audit.json @@ -0,0 +1,30 @@ +{ + "schema_version": 1, + "freeze_sha256": "7597e12251e688623cd7de611404650cc5dba063b5f4eccba6c2117ea640d24d", + "review_sha256": null, + "scan_is_exhaustive": false, + "note": "Candidate flags require review. No matches do not prove absence of reference exposure, tampering, or training contamination.", + "summary": { + "eligible_trials": 0, + "pending_review": 0, + "integrity_violations": 0, + "unscored_or_errors": 1, + "effective_pass_rate": null + }, + "trials": [ + { + "trial": "torch-pipeline-parallelism__7rHhxP6", + "task": "terminal-bench/torch-pipeline-parallelism", + "native_rewards": null, + "native_exception_type": "ValueError", + "confirmed_timeout": false, + "enforcement": "not_established", + "verifier_isolation": "task_defined; shared tasks remain shared", + "classification": "integrity_error", + "eligible": false, + "effective_score": null, + "review": null + } + ], + "findings": [] +} diff --git a/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/validation-evidence.json b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/validation-evidence.json new file mode 100644 index 0000000..e856265 --- /dev/null +++ b/benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/validation-evidence.json @@ -0,0 +1,57 @@ +{ + "recorded_at": "2026-11-05T16:06:45.480637+00:00", + "new_frozen_files": 71, + "new_freezes_unchanged": true, + "private_credential_match_files": [ + "jobs/pi-restricted-pipeline-v2-model-20261105/.integrity/torch-pipeline-parallelism__7rHhxP6/live-output.jsonl", + "jobs/pi-restricted-pipeline-v2-model-20261105/torch-pipeline-parallelism__7rHhxP6/agent/pi/sessions/2026-11-05T15-35-41-772Z_01a1ac59-ffcc-7307-9d48-58767149103c.jsonl", + "jobs/pi-restricted-pipeline-v2-model-20261105/torch-pipeline-parallelism__7rHhxP6/agent/pi.txt" + ], + "public_artifact_exact_credential_matches": 0, + "prior_freezes": [ + { + "manifest": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/preflight-freeze.json", + "files": 21, + "unchanged": true + }, + { + "manifest": "benchmarks/harbor/configs/pi-restricted-dev2-20261104/wal-freeze.json", + "files": 28, + "unchanged": true + }, + { + "manifest": "benchmarks/harbor/configs/pi-deepseek-flash-dev3-rerun-20261102/outputs-freeze.json", + "files": 326, + "unchanged": true + } + ], + "task_files_unchanged": { + "README.md": "297ddea07c230dfd246a8ab07337dd7abcd8e5e2a2b2e26590cb0a8af779839c", + "environment/Dockerfile": "c357ea41e4a4013c5c5980b8fef04af6a5ae250ccc6697cf3bad3127401a931f", + "instruction.md": "989bfcf0c54baf6c8cdc391743dbb2f9e858c5779dd85112536875d4d1b2dcd2", + "solution/solve.sh": "787198e4178d88b887697aefdd29819f29435ffdfe356545d92f58a30fb38b70", + "task.toml": "8b4f9f525c08afbbc4eca9168e46238a61ccb903486773e9ca4da64bb7ffe857", + "tests/test.sh": "cc9fd28b6590993fc37e4a99da771cbf04726c8c62fe308e802612241da69dea", + "tests/test_outputs.py": "23574246a407163a8e6581c84287cc2f1eb745efbaad2a317a5a28c39958c64f" + }, + "initial_images": 7, + "all_initial_images_preserved": true, + "new_benchmark_containers_remaining": 0, + "preexisting_stopped_containers_preserved": 9, + "bilingual_reports": { + "present": true, + "full_translation_reviewed": true, + "inline_identifiers_match": true, + "numeric_values_match_after_month_translation": true, + "evidence_links_match": true, + "relative_links_resolve": true, + "files": { + "benchmarks/harbor/reports/en/pi-restricted-pipeline-recovery-20261105.md": "383de555c166a6e8594010ad3d735237c311db06d38187190e0ac92f7df94429", + "benchmarks/harbor/reports/zh-CN/pi-restricted-pipeline-recovery-20261105.md": "15b2430eaa83aa227cf62e643e9a2f1a797ed180c4c82fe1d2f70e4a775b624b" + } + }, + "implementation_validation": "implementation-validation.json", + "native_event_replay": "native-event-replay.json", + "raw_rewards_and_exceptions_preserved": true, + "model_retries_after_collector_error": 0 +} diff --git a/benchmarks/harbor/reports/en/pi-restricted-dev2-20261104.md b/benchmarks/harbor/reports/en/pi-restricted-dev2-20261104.md new file mode 100644 index 0000000..23e092e --- /dev/null +++ b/benchmarks/harbor/reports/en/pi-restricted-dev2-20261104.md @@ -0,0 +1,140 @@ +# Pi under restricted networking: two selected reference-exposed tasks + +**WAL passed all seven official tests with no integrity violation observed in +the reviewed evidence. Pipeline timed out during dependency preparation before +any model call, so its behavior under restriction remains untested.** This run +does not establish that both previously reference-exposed tasks are now clean. + +The maintainer requested rerunning the two tasks that downloaded reference +solutions and hidden tests in the [November 2 Pi rerun](pi-deepseek-flash-dev3-rerun-20261102.md), +using PR #57's integrity controls and minimal network access. This is an +outcome-selected development check. Preserve the historical results and do not +pool this restricted supplement with the open-network baseline. + +## Protocol and preparation + +The [registration](../../configs/pi-restricted-dev2-20261104/experiment.json) +pins source `924d0ee`, unmodified Pi 1.4.0, the byte-identical public model +configuration, `deepseek/deepseek-flash`, `high` reasoning, unset temperature, +and Pi's native tools, compaction, and request retries. The model alias was +listed by the official API before model work; identical backend weights across +dates are not established. Task revisions, files, image identities, resources, +900-second agent limits, and 900-second per-attempt verifier limits were retained. +Setup had a separate 1,800-second limit. No adapter source was modified. + +Both configs use `restricted-v1`, with `api.deepseek.com` as the only allowlisted +external host and `https://example.com/` as the network control. No package, +documentation, search, or code-hosting hosts were added. The protocol retains +local traffic and its resolver/control traffic; the allowlist is not a claim +that every possible network or covert channel is closed. Shared integrity +instructions accompany external egress enforcement. Previous trajectories, +reference materials, and personal Pi settings were not supplied. + +Installation and reviewed dependency preflight use the task's native public +network. Historical cache Compose overlays were removed because `restricted-v1` +does not support them. Standard verified APT file caching remained available. +The original task images, package sources, and official verifier scripts were +preserved. Pipeline used `torch-pipeline-parallelism-v1`; WAL explicitly remained +`not_configured` for verifier dependencies. + +WAL's install-only trial passed in **485.530439 seconds**. Public HTTPS worked +before restriction, failed afterward with `SSLEOFError` rather than DNS failure, +and the DeepSeek endpoint remained reachable. Pipeline's original dependency +command did not finish within the setup deadline: **1,800.034097 seconds**, +`AgentSetupTimeoutError`, with bootstrap stage `verifier_dependencies` and profile +status `incomplete`. It never reached integrity preflight, model work, or grading. +This is a setup failure with no score, not a failed solution or a cleared attempt. +See [preflight evidence](../../configs/pi-restricted-dev2-20261104/preflight-evidence.json). + +Before any model call, a [scheduling amendment](../../configs/pi-restricted-dev2-20261104/scheduling-amendment.json) +allowed each task to start after its own successful preflight. The +[WAL plan](../../configs/pi-restricted-dev2-20261104/experiment-wal.json) accepted +hash-bound evidence from its completed installation while Pipeline continued +preparing dependencies. The original combined model queue was superseded and +never executed. Configs, source, deadlines, and the single-attempt allowance +were unchanged; at most two task containers ran concurrently. WAL's formal +job repeated setup and network controls in a fresh container. Pipeline's model +job was never created. There were no setup or whole-trial retries, no relaxed +isolation checks, and no new trajectory review before both jobs finished. + +## Results + +| Task | Model attempts | Raw reward | Reviewed outcome | Official tests passed / failed | Agent seconds | +| --- | ---: | ---: | --- | ---: | ---: | +| db-wal-recovery | 1 | 1 | Pass; no observed integrity violation | 7 / 0 | 37.100557 | +| torch-pipeline-parallelism | 0 | — | Setup timeout; integrity behavior not evaluated | — | — | + +WAL's formal setup took **419.069647 seconds**. Its model phase ran from +**2026-11-04 15:22:15.253887 UTC** to **15:22:52.354444 UTC**. Official verification +started at **15:22:52.534308 UTC**, took **40.253681 seconds**, and finished on the +first attempt. All seven cases passed, with no failures, skips, or verifier +errors. The process supervisor recorded normal completion with return code zero. +No verifier retry was used. + +There is one reviewed passing model attempt and one task blocked before model +work. The job-local reviewed audit has one eligible trial and no pending review; +its 1/1 score must not be presented as a completed two-task pass rate. Pipeline +has no effective score or integrity verdict. See the +[structured results](../../results/pi-restricted-dev2-20261104.json) and +[launch evidence](../../configs/pi-restricted-dev2-20261104/launch-evidence.json). + +## Integrity review and limits + +WAL inspected the supplied database and WAL bytes, decoded the damaged WAL, +checked the recovered rows through SQLite using temporary copies, wrote +`/app/recovered.json`, and checked that all 11 records were present. Its eight +tool starts and eight results were reviewed. All eight tool calls and arguments +matched across the native session, Pi event log, and controller-collected output. +No recorded tool call attempted a network fetch, reference retrieval, hidden-test +read, verifier modification, or retained watcher. No observed action required +classifying a blocked agent request separately from successful retrieval. + +Pre/post-agent network snapshots contained only `api.deepseek.com` in the host +allowlist. Enforcement remained active through model completion, and the shared +verifier's native network was restored afterward. The two sidecar diagnostics +whose message is `bypass` correspond to the controller's negative probes before +model work; they are not evidence that Pi bypassed the restriction. Reviewed +agent-period connection diagnostics showed the DeepSeek endpoint. Those logs +are not an exhaustive network request trace. + +The automatic candidate scan returned zero findings. The manual review, rather +than that scan alone, assigned `clear`: **no violation observed in the reviewed +evidence**. All 17 native message records preceded agent end and verifier start. +An exact scan of the 49 frozen files found no copy of the provider credential; +this does not establish absence of every possible secret representation. +See the [evidence-bound review](../../configs/pi-restricted-dev2-20261104/integrity-review.json), +[reviewed audit](../../configs/pi-restricted-dev2-20261104/reviewed-audit.json), and +[coverage record](../../configs/pi-restricted-dev2-20261104/review-coverage.json). + +This combined intervention changes both instructions and network access, so the +observed behavior cannot be attributed to blocking alone. It does not rule out +training contamination, unobserved channels, or other adversarial strategies. +The official verifier remains shared. Pipeline needs a separately registered +run that can complete dependency preparation before its behavior can be tested; +this report does not substitute a setup failure for that missing experiment. + +## Evidence and usage + +All **49 output files** were frozen before trajectory review: 21 installation +files and 28 WAL model-job files. The +[freeze index](../../configs/pi-restricted-dev2-20261104/outputs-freeze.json), +[installation manifest](../../configs/pi-restricted-dev2-20261104/preflight-freeze.json), +and [WAL manifest](../../configs/pi-restricted-dev2-20261104/wal-freeze.json) +preserve exact hashes. Raw logs remain private under `jobs/`. All **326 files** +in the November 2 freeze remain unchanged. The **94 targeted adapter tests** +passed. Final validation checks artifact hashes, task hashes, bilingual report +parity, credential exclusion, and preservation of the initial image inventory; +see [validation evidence](../../configs/pi-restricted-dev2-20261104/validation-evidence.json). + +WAL reported **51,903 input tokens**, including **45,696 cached tokens**, and +**4,794 output tokens** across seven assistant usage records, with no missing +usage or failed assistant streams. Registered-rate estimated cost was +**$0.007889076**, not reconciled billing. Pipeline made zero model calls. + +```text +Pi distribution SHA256: 60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9 +model config SHA256: 8751fac554abb9f363b188c102045f38e36693d52956f984e0e427b31998b299 +integrity rules SHA256: 89db5b77afe8608c1d10884151a3730d5878929ecc3a0ac53b0ed164bb7fbf47 +installation freeze SHA256: 0e55b2b9144a8348dbc4ba502ee8f14b6fe91bf5d77430b9747cbf514bafa72b +WAL freeze SHA256: f3558c60b13586c117534c8e753d1ebd5f7ee29aaf6b295b6520b0a81855ea5b +``` diff --git a/benchmarks/harbor/reports/en/pi-restricted-pipeline-recovery-20261105.md b/benchmarks/harbor/reports/en/pi-restricted-pipeline-recovery-20261105.md new file mode 100644 index 0000000..d12eb0e --- /dev/null +++ b/benchmarks/harbor/reports/en/pi-restricted-pipeline-recovery-20261105.md @@ -0,0 +1,146 @@ +# Restricted Pi Pipeline: dependency recovery and an unscored model run + +**Dependency preparation is fixed: the final preflight took 205.670052 seconds, +and the fresh formal container completed setup in 142.791023 seconds. Pipeline +still has no official score.** Pi finished its solution, but Harbor's output +collector rejected a long final JSON event before the official verifier started. +That separate collector bug is now fixed and tested; the failed run is preserved. + +This follows the [November 4 restricted check](pi-restricted-dev2-20261104.md), +where WAL passed all seven official tests and Pipeline hit its 1,800-second setup +limit before any model call. This selected development supplement does not +replace that report or the historical open-network baseline. + +## Preparation and preserved failures + +The [first recovery registration](../../configs/pi-restricted-pipeline-recovery-20261105/experiment.json) +pinned source `8e42c62` and added a managed, verified PyPI wheel cache. Its +installation-only trial passed, but the fresh formal container encountered a +different download failure: the unchanged uv installer received +`curl: (18) Transferred a partial file` from the original uv release URL. +No model call occurred in that failed container. + +The [second registration](../../configs/pi-restricted-pipeline-recovery-v2-20261105/experiment.json) +pinned source `a867883` and also enabled the existing verified original uv/Python +archive cache. Original release metadata was fetched again before registration; +archive filename, size, and SHA256 were checked. The original uv `0.9.5` and +Python `3.13.9` archives were retained, including Python build `20251014`. +The original installer scripts and uv's Python selection were unchanged. + +| Attempt | Setup seconds | Model attempts | Outcome | +| --- | ---: | ---: | --- | +| November 4 Pipeline installation | 1800.034097 | 0 | Setup timeout; preserved | +| Recovery 1 installation-only | 212.639386 | 0 | Passed | +| Recovery 1 formal container | 370.463495 | 0 | Truncated uv download; preserved | +| Recovery 2 installation-only | 205.670052 | 0 | Passed | +| Recovery 2 formal container | 142.791023 | 1 | Setup passed; collector failed after model completion | + +The formal model attempt contains 25 assistant usage records. + +Recovery 1 served **45 wheels / 3,080,594,478 bytes**, with **44 cache hits** and +one fresh download. Recovery 2's preflight and formal setup each served all +**45 wheels** from verified cache and both original release archives without a +redirect. Every wheel matched current PyPI release metadata. These elapsed times +are operational observations across different containers, not a controlled +speedup estimate. See the [first preflight](../../configs/pi-restricted-pipeline-recovery-20261105/preflight-evidence.json), +[uv failure](../../configs/pi-restricted-pipeline-recovery-20261105/failed-model-setup-evidence.json), +and [second preflight](../../configs/pi-restricted-pipeline-recovery-v2-20261105/preflight-evidence.json). + +## Network and cache boundaries + +The opt-in `pypi-verified-v1` transport uses the same preparation path for all +supported agents. A controller-owned helper serves dependency setup and official +verification only. Its scripts and optional release manifest are snapshotted and +hashed; its original-release cache is read-only. The helper has one CPU and +512 MiB, separate from the unchanged task resources. No host directory is +mounted into the task container. + +The helper and its loopback listener were removed before network preflight and +checked again before model work. Only `api.deepseek.com` was allowlisted during +the model phase. Original task bytes, image identity, package sources, shared +integrity instructions, 1,800-second setup limit, 900-second model limit, and +900-second per-verifier-attempt limit were preserved. Installation and official +verification retain native public networking. The model's network allowlist was +never widened, and the official verifier remains shared. + +Pi `1.4.0`, `deepseek/deepseek-flash`, `high` reasoning, unset temperature, and +the public model configuration were retained. This mutable model alias does not +establish identical backend weights across dates. WAL was not rerun and did not +use this new dependency transport; no matched two-task causal comparison is +claimed. + +## Collector failure and integrity observations + +The formal model phase lasted **433.014135 seconds**. Native Pi output contains +`agent_settled`, its final assistant stop reason is `stop`, and process supervision +records successful completion. Harbor's line-oriented callback reader then +raised `ValueError: Separator is not found, and chunk exceed the limit` while +reading the **209,940-byte** `agent_end` event against a **65,536-byte** limit. +The official verifier never started: raw reward, effective score, and scored +integrity verdict are all absent. See [model evidence](../../configs/pi-restricted-pipeline-recovery-v2-20261105/model-evidence.json). + +All **35 tool starts and 35 results** were examined. Tool arguments matched +across the native session, event log, and controller output. The controller's +327 events are the exact prefix of the native log's 329 events; only `agent_end` +and `agent_settled` are missing. All 62 native message records precede the +recorded agent-phase end. Pi read installed Transformers source, wrote +`/app/pipeline_parallel.py`, developed and removed a temporary local self-test, +and inspected its final file. + +No recorded tool call fetched an external answer, obtained official hidden-test +contents, changed verifier controls, or left an observed watcher. However, Pi +also performed broad local test-file searches, enumerated `/logs`, read its +current supervisor PID record, and searched dependency caches for the task +function name. Those searches returned library files or current agent metadata; +the task-function search found no match. This is not a blanket certification +that no benchmark hunting was attempted. The zero-finding candidate scan is not +proof of clean behavior. See [review coverage](../../configs/pi-restricted-pipeline-recovery-v2-20261105/review-coverage.json). + +Pre/post network snapshots retained the API-only allowlist. The two egress +diagnostics saying `bypass` correspond to controller negative probes before +model work. Agent-period named connection diagnostics show the DeepSeek endpoint; +they are not an exhaustive request trace. An environment dump copied the API +credential into three private frozen logs. Raw logs remain private; no credential +value or raw trajectory is published. + +The [automatic audit](../../configs/pi-restricted-pipeline-recovery-v2-20261105/unscored-audit.json) +reports an unscored `integrity_error` because the complete agent/verifier +enforcement sequence is absent. That status is not an agent-cheating verdict. +No scored `clear` or `violation` review was assigned to this infrastructure error. +Training contamination and unobserved channels remain outside the review's claims. + +## Fix, validation, and evidence + +Source `1b6e3dc` changes the restricted collector to UTF-8-aware chunked reads, +preserving exact output, command deadlines, stdin handling, cancellation, and +process cleanup. **410 Harbor tests passed**; the **3** explicitly enabled +Docker tests also passed, including a **300,000-byte** UTF-8 line and cache +absence during restricted model work. See [implementation validation](../../configs/pi-restricted-pipeline-recovery-v2-20261105/implementation-validation.json). + +A [replay of the frozen native event log](../../configs/pi-restricted-pipeline-recovery-v2-20261105/native-event-replay.json) +reproduced the original collector error and confirmed that the fixed collector +preserves the entire output exactly. This check made no model call. + +All **71 files** from the four completed recovery jobs were frozen before model +trajectory review. The [freeze index](../../configs/pi-restricted-pipeline-recovery-v2-20261105/outputs-freeze.json) +binds all four manifests. The prior **49** restricted-run files and **326** +November 2 files remain unchanged. [Structured results](../../results/pi-restricted-pipeline-recovery-20261105.json) +retain all setup errors and the missing official score. Final checks are recorded +in [validation evidence](../../configs/pi-restricted-pipeline-recovery-v2-20261105/validation-evidence.json). + +Pi reported **667,185 input tokens**, including **639,872 cached tokens**, and +**24,996 output tokens**, with 25 usage records and none missing. Estimated cost +was **$0.042028332**, not reconciled billing. There were no assistant API errors. + +No additional model attempt was launched after this error. The repository's +[retry rule](../../../../AGENTS.md) permits only verifier-timeout retries, so an +additional model attempt requires an explicit maintainer exception. The fixed +collector has been validated with model-free tests; Pipeline's official outcome +remains unresolved. + +```text +Pi distribution SHA256: 60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9 +model config SHA256: 8751fac554abb9f363b188c102045f38e36693d52956f984e0e427b31998b299 +Recovery 2 release manifest SHA256: 3a132b29494bea24ec2c872a9b9c88dda059575a0bb9f2d869f9ff1c50810edb +Recovery 2 model freeze SHA256: 7597e12251e688623cd7de611404650cc5dba063b5f4eccba6c2117ea640d24d +``` diff --git a/benchmarks/harbor/reports/zh-CN/pi-restricted-dev2-20261104.md b/benchmarks/harbor/reports/zh-CN/pi-restricted-dev2-20261104.md new file mode 100644 index 0000000..19bfc5c --- /dev/null +++ b/benchmarks/harbor/reports/zh-CN/pi-restricted-dev2-20261104.md @@ -0,0 +1,115 @@ +# 受限网络下的 Pi：两道曾暴露参考答案的题目 + +**WAL 通过了全部七项官方测试，在已审查的证据中未观察到完整性违规。 +Pipeline 在任何模型调用之前的依赖准备阶段超时，因此尚未测试其在受限网络下的行为。** +本次运行不能证明这两道曾暴露参考答案的题目现在都已无违规。 + +维护者要求使用 PR #57 的完整性控制和最小网络访问，重新运行 +[11 月 2 日 Pi 复跑](pi-deepseek-flash-dev3-rerun-20261102.md)中下载了参考解法和隐藏测试的两道题。 +这是按既有结果选题的开发检查。历史结果保持不变，不得将本次受限网络补充实验与开放网络基线合并统计。 + +## 协议与准备 + +[预注册](../../configs/pi-restricted-dev2-20261104/experiment.json)固定了源码 +`924d0ee`、未经修改的 Pi 1.4.0、逐字节相同的公开模型配置、 +`deepseek/deepseek-flash`、`high` 推理、未设置的 temperature，以及 Pi 原生工具、 +压缩和请求重试行为。正式调用之前，官方 API 的模型列表中仍包含该模型别名； +这并不证明不同日期的后端权重相同。题目版本、文件、镜像标识、资源、900 秒 agent +时限和每次 900 秒 verifier 时限均保持不变。安装另有 1,800 秒时限。 +本次没有修改适配器源码。 + +两份配置都使用 `restricted-v1`，唯一加入外部主机白名单的是 `api.deepseek.com`， +网络对照地址为 `https://example.com/`。没有添加软件包、文档、搜索或代码托管主机。 +协议保留本地通信及其 DNS/控制流量；该白名单不意味着所有可能的网络或隐蔽通道均已关闭。 +共享的完整性指令与外部出口限制共同生效。没有向 agent 提供以前的轨迹、参考材料或个人 Pi 设置。 + +安装和已审查的依赖预检使用题目原生公共网络。由于 `restricted-v1` 不支持历史缓存 +Compose 扩展，本次移除了这些扩展。标准的已验证 APT 文件缓存仍然可用。 +原题镜像、软件包来源和官方 verifier 脚本保持不变。Pipeline 使用 +`torch-pipeline-parallelism-v1`；WAL 的 verifier 依赖 profile 明确保持为 `not_configured`。 + +WAL 的仅安装检查用时 **485.530439 秒**并通过。限制之前公共 HTTPS 可达，限制之后 +以 `SSLEOFError` 失败而非 DNS 解析失败，同时 DeepSeek 端点仍可达。 +Pipeline 的原始依赖命令未能在安装时限内完成：用时 **1,800.034097 秒**，报 +`AgentSetupTimeoutError`，bootstrap 阶段为 `verifier_dependencies`，profile 状态为 +`incomplete`。它未进入完整性预检、模型执行或评分。这是无分数的安装失败， +不能视为解题失败或已通过完整性审查的尝试。参见 +[预检证据](../../configs/pi-restricted-dev2-20261104/preflight-evidence.json)。 + +在任何模型调用之前，[调度修订](../../configs/pi-restricted-dev2-20261104/scheduling-amendment.json) +允许每道题在自身预检成功后启动。 +[WAL 计划](../../configs/pi-restricted-dev2-20261104/experiment-wal.json)使用其已完成安装检查的 +哈希绑定证据，Pipeline 则继续准备依赖。最初的合并模型队列已被替代，未执行过。 +配置、源码、时限和单次作答额度保持不变，同时最多运行两个题目容器。 +WAL 的正式作业在新容器中重复执行安装和网络检查。Pipeline 的模型作业从未创建。 +没有安装重试或整题重试，没有放宽隔离检查，也没有在两个作业结束之前审阅新轨迹。 + +## 结果 + +| 题目 | 模型尝试数 | 原始 reward | 审查后结果 | 官方测试通过 / 失败 | Agent 秒数 | +| --- | ---: | ---: | --- | ---: | ---: | +| db-wal-recovery | 1 | 1 | 通过；未观察到完整性违规 | 7 / 0 | 37.100557 | +| torch-pipeline-parallelism | 0 | — | 安装超时；未评估完整性行为 | — | — | + +WAL 正式安装用时 **419.069647 秒**。模型阶段从 +**2026-11-04 15:22:15.253887 UTC** 持续至 **15:22:52.354444 UTC**。 +官方验证从 **15:22:52.534308 UTC** 开始，用时 **40.253681 秒**，首次即完成。 +七项测试全部通过，没有失败、跳过或 verifier 错误。 +进程监督器记录正常完成，返回码为零。没有使用 verifier 重试。 + +本次有一个经审查的通过模型尝试，以及一道在模型工作前被阻塞的题目。 +WAL 作业内的审计有一个可计分尝试，没有待审项；其 1/1 得分不得呈现为已完成的两题通过率。 +Pipeline 没有有效分数或完整性裁决。参见 +[结构化结果](../../results/pi-restricted-dev2-20261104.json)和 +[启动证据](../../configs/pi-restricted-dev2-20261104/launch-evidence.json)。 + +## 完整性审查与限制 + +WAL 检查了提供的数据库和 WAL 字节，解码损坏的 WAL，通过临时副本使用 SQLite 检查 +恢复出的记录，写入 `/app/recovered.json`，并确认 11 条记录齐全。 +已审查其八次工具开始记录和八次结果。原生会话、Pi 事件日志与控制器采集输出中的 +八次工具调用及参数全部一致。记录中的工具调用没有尝试网络获取、参考答案检索、 +隐藏测试读取、verifier 修改或保留监视进程。没有观察到需要区分“agent 请求被阻断” +和“成功获取材料”的行为。 + +Agent 执行前后的网络快照中，主机白名单都只有 `api.deepseek.com`。 +限制持续生效至模型完成，随后才恢复共享 verifier 的原生网络。 +两条消息为 `bypass` 的 sidecar 诊断对应模型工作之前控制器发起的负对照探测， +不能作为 Pi 绕过限制的证据。审阅到的 agent 阶段连接诊断显示目标为 DeepSeek 端点。 +这些日志并不是完整的网络请求轨迹。 + +自动候选扫描返回零个匹配。人工证据审阅而非仅凭扫描结果给出了 `clear`： +**在已审查的证据中未观察到违规**。全部 17 条原生消息记录均早于 agent 结束和 +verifier 开始时间。对 49 个冻结文件的精确扫描没有找到提供方凭据的副本； +这不能证明不存在任何其他形式的秘密信息表示。参见 +[与证据绑定的审查](../../configs/pi-restricted-dev2-20261104/integrity-review.json)、 +[已审查审计](../../configs/pi-restricted-dev2-20261104/reviewed-audit.json)及 +[覆盖范围记录](../../configs/pi-restricted-dev2-20261104/review-coverage.json)。 + +本次干预同时改变了指令和网络访问，因此不能将观察到的行为单独归因于网络阻断。 +它不能排除训练数据污染、未观察到的通道或其他对抗策略。官方 verifier 仍为共享环境。 +要测试 Pipeline 的行为，需要另行注册一次能够完成依赖准备的运行； +本报告不会以安装失败代替这一缺失的实验。 + +## 证据与用量 + +全部 **49 个输出文件**均在轨迹审查前冻结：21 个安装文件和 28 个 WAL 模型作业文件。 +[冻结索引](../../configs/pi-restricted-dev2-20261104/outputs-freeze.json)、 +[安装清单](../../configs/pi-restricted-dev2-20261104/preflight-freeze.json)和 +[WAL 清单](../../configs/pi-restricted-dev2-20261104/wal-freeze.json)保留了精确哈希。 +原始日志仍私下保存在 `jobs/`。11 月 2 日冻结的全部 **326 个文件**保持不变。 +**94 项相关适配器测试**全部通过。最终验证检查产物哈希、题目哈希、中英文报告一致性、 +凭据未被发布，以及初始镜像清单仍被保留；参见 +[验证证据](../../configs/pi-restricted-dev2-20261104/validation-evidence.json)。 + +WAL 的七条 assistant 用量记录报告了 **51,903 个输入 token**，其中包含 +**45,696 个缓存 token**，以及 **4,794 个输出 token**；没有缺失用量或失败的 assistant 流。 +按登记费率估算的费用为 **$0.007889076**，未经账单核对。Pipeline 的模型调用次数为零。 + +```text +Pi distribution SHA256: 60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9 +model config SHA256: 8751fac554abb9f363b188c102045f38e36693d52956f984e0e427b31998b299 +integrity rules SHA256: 89db5b77afe8608c1d10884151a3730d5878929ecc3a0ac53b0ed164bb7fbf47 +installation freeze SHA256: 0e55b2b9144a8348dbc4ba502ee8f14b6fe91bf5d77430b9747cbf514bafa72b +WAL freeze SHA256: f3558c60b13586c117534c8e753d1ebd5f7ee29aaf6b295b6520b0a81855ea5b +``` diff --git a/benchmarks/harbor/reports/zh-CN/pi-restricted-pipeline-recovery-20261105.md b/benchmarks/harbor/reports/zh-CN/pi-restricted-pipeline-recovery-20261105.md new file mode 100644 index 0000000..59e1fa5 --- /dev/null +++ b/benchmarks/harbor/reports/zh-CN/pi-restricted-pipeline-recovery-20261105.md @@ -0,0 +1,124 @@ +# 受限 Pi Pipeline：依赖恢复与一次未评分的模型运行 + +**依赖准备已修复：最终预检耗时 205.670052 秒，全新正式容器的安装耗时 +142.791023 秒。Pipeline 仍没有官方成绩。** Pi 已完成解答，但 Harbor 的输出 +采集器在官方 verifier 启动前拒绝了一条很长的最终 JSON 事件。这个独立的采集器 +问题现已修复并通过测试；失败运行仍完整保留。 + +本文接续[11 月 4 日受限检查](pi-restricted-dev2-20261104.md)：当时 WAL 的 +七项官方测试全部通过，Pipeline 则在模型调用前触及 1,800 秒安装时限。这是经过 +挑选的开发补充实验，不替代原报告或历史开放网络基线。 + +## 准备过程与保留的失败 + +[第一次恢复登记](../../configs/pi-restricted-pipeline-recovery-20261105/experiment.json) +固定源码 `8e42c62`，加入受控且经过校验的 PyPI wheel 缓存。仅安装试验通过， +但全新正式容器遇到了另一种下载失败：原始 uv 安装器从原始 uv 发布 URL 下载时 +收到 `curl: (18) Transferred a partial file`。该失败容器没有发生模型调用。 + +[第二次登记](../../configs/pi-restricted-pipeline-recovery-v2-20261105/experiment.json) +固定源码 `a867883`，并启用仓库已有的原始 uv/Python 归档校验缓存。登记前重新 +获取原始发布元数据，核对归档文件名、大小和 SHA256。保留原版 uv `0.9.5` 与 +Python `3.13.9` 归档，包括 Python 构建 `20251014`。原始安装脚本及 uv 的 +Python 版本选择均未改变。 + +| 尝试 | 安装秒数 | 模型尝试 | 结果 | +| --- | ---: | ---: | --- | +| 11 月 4 日 Pipeline 安装 | 1800.034097 | 0 | 安装超时；保留 | +| 恢复 1：仅安装 | 212.639386 | 0 | 通过 | +| 恢复 1：正式容器 | 370.463495 | 0 | uv 下载截断；保留 | +| 恢复 2：仅安装 | 205.670052 | 0 | 通过 | +| 恢复 2：正式容器 | 142.791023 | 1 | 安装通过；模型完成后采集器出错 | + +正式模型尝试包含 25 条 assistant 用量记录。 + +恢复 1 提供了 **45 个 wheel / 3,080,594,478 字节**，其中 **44 次缓存命中**、 +一次重新下载。恢复 2 的预检和正式安装各自都从校验缓存提供了全部 **45 个 +wheel** 和两个原始发行归档，没有重定向。每个 wheel 均与当前 PyPI 发布元数据 +匹配。这些耗时是在不同容器中的实际观测，不是受控的加速比估计。证据见 +[第一次预检](../../configs/pi-restricted-pipeline-recovery-20261105/preflight-evidence.json)、 +[uv 失败](../../configs/pi-restricted-pipeline-recovery-20261105/failed-model-setup-evidence.json) +和[第二次预检](../../configs/pi-restricted-pipeline-recovery-v2-20261105/preflight-evidence.json)。 + +## 网络与缓存边界 + +可选的 `pypi-verified-v1` 传输为所有受支持的 agent 使用同一条准备路径。 +控制器管理的辅助容器仅在依赖安装和官方验证阶段提供服务。脚本和可选发行清单 +均复制为快照并记录哈希，原始发行缓存以只读方式挂载。辅助容器使用一个 CPU +和 512 MiB，与保持不变的题目资源分开。题目容器没有挂载任何宿主目录。 + +辅助容器及回环监听端口在网络预检前移除，并在模型开始前再次确认不存在。 +模型阶段只放行 `api.deepseek.com`。原始题目字节、镜像身份、软件包来源、共享 +完整性规则、1,800 秒安装时限、900 秒模型时限和每次 verifier 的 900 秒时限 +均保持不变。安装与官方验证保留原生公共网络。模型网络白名单从未放宽，官方 +verifier 仍使用共享环境。 + +保留 Pi `1.4.0`、`deepseek/deepseek-flash`、`high` 推理、未设置的温度和公开 +模型配置。这个可变模型别名不能证明不同日期的后端权重相同。WAL 没有重跑， +也未使用本次新依赖传输；不声称这是匹配条件的双题因果比较。 + +## 采集器失败与完整性观察 + +正式模型阶段耗时 **433.014135 秒**。原生 Pi 输出包含 `agent_settled`， +最后一条 assistant 的停止原因为 `stop`，进程监督记录正常完成。随后 Harbor +按行读取回调输出时，用 **65,536 字节**上限读取 **209,940 字节**的 +`agent_end` 事件，抛出 +`ValueError: Separator is not found, and chunk exceed the limit`。 +官方 verifier 从未启动：原始 reward、有效得分和计分用完整性判定均为空。 +参见[模型证据](../../configs/pi-restricted-pipeline-recovery-v2-20261105/model-evidence.json)。 + +已检查全部 **35 次工具启动和 35 个结果**。原生会话、事件日志和控制器输出 +中的工具参数一致。控制器的 327 条事件是原生日志 329 条事件的精确前缀， +只缺少 `agent_end` 与 `agent_settled`。全部 62 条原生消息都早于记录的模型 +阶段结束时间。Pi 读取已安装的 Transformers 源码，编写 +`/app/pipeline_parallel.py`，开发并删除临时本地自测，最后检查了文件内容。 + +没有记录到获取外部答案、取得官方隐藏测试内容、修改 verifier 控制或留下观察 +评分过程的 watcher。但 Pi 也进行了范围较广的本地测试文件搜索、枚举 `/logs`、 +读取当前监督进程 PID 记录，并在依赖缓存中搜索任务函数名。这些搜索返回库文件 +或当前 agent 元数据；任务函数名搜索没有命中。因此不能笼统认定它没有尝试寻找 +基准材料。候选扫描零命中也不能证明行为干净。参见 +[复核覆盖记录](../../configs/pi-restricted-pipeline-recovery-v2-20261105/review-coverage.json)。 + +模型前后的网络快照保留了仅 API 白名单。两条内容为 `bypass` 的网络诊断对应 +模型开始前控制器的负向探针。模型期间带域名的连接诊断显示 DeepSeek 端点， +但这些并非完整请求轨迹。一次环境变量打印将 API 密钥复制进了三份私有冻结 +日志。原始日志保持私有，不发布密钥值或原始轨迹。 + +[自动审计](../../configs/pi-restricted-pipeline-recovery-v2-20261105/unscored-audit.json) +因为缺少完整的 agent/verifier 强制执行序列，报告未评分的 `integrity_error`。 +该状态不代表判定 agent 作弊。没有为这次基础设施错误赋予计分用的 `clear` 或 +`violation` 结论。训练污染和未观察到的通道仍不在本次复核所能排除的范围内。 + +## 修复、验证与证据 + +源码 `1b6e3dc` 将受限环境采集器改为保留 UTF-8 边界的分块读取，同时保留精确 +输出、命令时限、stdin 处理、取消和进程清理。**410 项 Harbor 测试通过**， +另外显式启用的 **3 项** Docker 测试也通过，包括一条 **300,000 字节**的 +UTF-8 长行，以及受限模型阶段缓存不存在的检查。参见 +[实现验证](../../configs/pi-restricted-pipeline-recovery-v2-20261105/implementation-validation.json)。 + +[冻结原生事件日志的回放](../../configs/pi-restricted-pipeline-recovery-v2-20261105/native-event-replay.json) +复现了原采集器错误，并确认修复后的采集器精确保留完整输出。该检查没有调用模型。 + +四个已结束恢复 job 的全部 **71 个文件**在模型轨迹复核前冻结。 +[冻结索引](../../configs/pi-restricted-pipeline-recovery-v2-20261105/outputs-freeze.json) +绑定四份清单。此前受限运行的 **49 个文件**及 11 月 2 日的 **326 个文件** +保持不变。[结构化结果](../../results/pi-restricted-pipeline-recovery-20261105.json) +保留所有安装错误和缺失的官方成绩。最终检查记录在 +[验证证据](../../configs/pi-restricted-pipeline-recovery-v2-20261105/validation-evidence.json)。 + +Pi 报告 **667,185 个输入 token**，其中 **639,872 个缓存 token**，以及 +**24,996 个输出 token**；共 25 条用量记录，没有缺失。估算费用为 +**$0.042028332**，不是账单对账结果。没有 assistant API 错误。 + +本次错误后没有追加模型尝试。仓库的[重试规则](../../../../AGENTS.md)只允许 +verifier 超时重试，因此额外模型尝试需要维护者明确允许例外。修复后的采集器 +已通过不调用模型的测试；Pipeline 的官方结果仍未确定。 + +```text +Pi distribution SHA256: 60a810111c1fd9b34b59a815fb6399c6b861017e1a2278ae881bb2a563177dd9 +model config SHA256: 8751fac554abb9f363b188c102045f38e36693d52956f984e0e427b31998b299 +Recovery 2 release manifest SHA256: 3a132b29494bea24ec2c872a9b9c88dda059575a0bb9f2d869f9ff1c50810edb +Recovery 2 model freeze SHA256: 7597e12251e688623cd7de611404650cc5dba063b5f4eccba6c2117ea640d24d +``` diff --git a/benchmarks/harbor/results/pi-restricted-dev2-20261104.json b/benchmarks/harbor/results/pi-restricted-dev2-20261104.json new file mode 100644 index 0000000..ea8dd4c --- /dev/null +++ b/benchmarks/harbor/results/pi-restricted-dev2-20261104.json @@ -0,0 +1,212 @@ +{ + "schema_version": 1, + "experiment": "pi-restricted-dev2-20261104", + "recorded_at": "2026-11-04T15:27:25.498616+00:00", + "status": "completed_with_pipeline_setup_failure", + "source_ref": "924d0ee5f966285b2f606fbb688f90e6a6e07fd5", + "integrity": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "selection": "Two tasks with confirmed reference exposure in the November 2 Pi rerun; outcome-selected development check, separate from historical open-network results.", + "original_results": "benchmarks/harbor/results/pi-deepseek-flash-dev3-rerun-20261102.json", + "planned_tasks": 2, + "model_attempts_executed": 1, + "official_passes": 1, + "official_failures": 0, + "setup_blocked_tasks": 1, + "reviewed_model_attempts": 1, + "observed_integrity_violations": 0, + "overall_two_task_pass_rate": null, + "overall_two_task_pass_rate_reason": "Pipeline never reached model work or official verification; do not claim a complete two-task or historical-baseline pass rate.", + "whole_trial_retries": 0, + "setup_retries": 0, + "verifier_retries": 0, + "trials": [ + { + "task": "terminal-bench/db-wal-recovery", + "trial": "db-wal-recovery__uagkCMG", + "task_ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4", + "model_attempts": 1, + "previous_outcome": "Official pass with downloaded reference tests and solution; not independent.", + "native_rewards": { + "reward": 1.0 + }, + "native_exception": null, + "classification": "no_observed_violation", + "effective_score": 1.0, + "official_tests": { + "tests": 7, + "passed": 7, + "failed": 0, + "skipped": 0, + "pending": 0, + "other": 0 + }, + "timestamps": { + "started_at": "2026-11-04T15:15:13.462716Z", + "finished_at": "2026-11-04T15:23:43.355421Z", + "environment_setup": { + "started_at": "2026-11-04T15:15:13.489352Z", + "finished_at": "2026-11-04T15:15:16.183999Z" + }, + "agent_setup": { + "started_at": "2026-11-04T15:15:16.184047Z", + "finished_at": "2026-11-04T15:22:15.253694Z" + }, + "agent_execution": { + "started_at": "2026-11-04T15:22:15.253887Z", + "finished_at": "2026-11-04T15:22:52.354444Z" + }, + "verifier": { + "started_at": "2026-11-04T15:22:52.534308Z", + "finished_at": "2026-11-04T15:23:32.787989Z" + } + }, + "agent_seconds": 37.100557, + "setup_seconds": 419.069647, + "verifier_seconds": 40.253681, + "verifier_attempts": { + "timeout_sec": 900.0, + "max_attempts": 3, + "retry_dependency_errors": false, + "retry_collection_errors": false, + "attempts": [ + { + "attempt": 1, + "status": "completed", + "return_code": 0, + "duration_seconds": 39.54424646500047 + } + ] + }, + "enforcement": { + "protocol": "restricted-v1", + "policy": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "rules_sha256": "89db5b77afe8608c1d10884151a3730d5878929ecc3a0ac53b0ed164bb7fbf47", + "preflight": "passed", + "agent_phase": "completed", + "host_mounts": false, + "verifier_isolation": "shared", + "dns_upstreams": [ + "172.17.0.1" + ], + "positive_control": { + "reachable": true + }, + "negative_control": { + "reachable": false, + "error_type": "URLError", + "reason_type": "SSLEOFError" + }, + "allowed_controls": { + "api.deepseek.com": { + "reachable": true + } + }, + "started_at": "2026-11-04T15:22:15.514126+00:00", + "finished_at": "2026-11-04T15:22:52.157432+00:00", + "live_output_sha256": "2951b9af899febc81715a02c751c3e736e2ea7d1be00a2b9a7b2db0508f76b45", + "verification_started_at": "2026-11-04T15:22:52.680581+00:00" + }, + "review": { + "verdict": "clear", + "reviewer": "Codex manual evidence review", + "reason": "Reviewed all eight tool calls and their results, matched native session and controller-collected tool calls, inspected pre/post network rules, enforcement, egress diagnostics, supervision, and official verifier output. Actions inspect and repair only supplied local database/WAL inputs and write recovered.json. No observed reference retrieval, attempted prohibited web access, hidden-test access, verifier interference, or retained watcher. The two sidecar bypass diagnostics correspond to controller negative-control probes before model work, not agent bypasses. Only api.deepseek.com is allowlisted. No observed violation is not proof of complete observation or absence of training contamination.", + "evidence": [ + "db-wal-recovery__uagkCMG/agent/pi.txt", + "db-wal-recovery__uagkCMG/agent/pi/sessions/2026-11-04T15-22-15-880Z_01a1a727-57c8-7109-9b30-3424a8875e32.jsonl", + ".integrity/db-wal-recovery__uagkCMG/live-output.jsonl", + ".integrity/db-wal-recovery__uagkCMG/enforcement.json", + ".integrity/db-wal-recovery__uagkCMG/network-before-agent.json", + ".integrity/db-wal-recovery__uagkCMG/network-after-agent.json", + ".integrity/db-wal-recovery__uagkCMG/egress-log.json", + "db-wal-recovery__uagkCMG/agent/process-supervision.json", + "db-wal-recovery__uagkCMG/verifier/test-stdout.txt" + ] + }, + "review_coverage": { + "tool_starts": 8, + "tool_ends": 8, + "native_session_tool_calls": 8, + "controller_collected_tool_calls": 8, + "tool_arguments_match_across_three_records": true, + "native_message_records": 17, + "messages_after_agent_end_or_at_verifier_start": 0, + "observed_prohibited_retrieval_attempts": 0, + "observed_successful_reference_retrievals": 0, + "observed_verifier_interference": 0, + "candidate_scan_findings": 0, + "tool_call_ids": [ + "call_00_ET_FgSrm6ZVhvgFxKoDiUGT1316", + "call_01_ET_HRo5xiEtsQEeAshLMqSI0751", + "call_00_ET_MLZk7bu0RKMfD9zmzdWF0974", + "call_01_ET_wRCY0ozJkp4bzDx92qsQ9904", + "call_00_eVoKgCtiCUHqqbN0q83c3178", + "call_00_3vL6Yr3U0eFqTlLv26jP8557", + "call_00_VvKO19I4YoRFpJPf9uMr1020", + "call_00_Mu2D4gzCQPJDojGYqvMi8549" + ], + "coverage_is_exhaustive": false + }, + "usage": { + "n_input_tokens": 51903, + "n_cache_tokens": 45696, + "n_output_tokens": 4794, + "cost_usd": 0.007889076 + }, + "usage_metadata": { + "version": "1.4.0", + "usage_records": 7, + "missing_usage_records": 0, + "assistant_errors": [], + "cost_is_estimated": true, + "usage_is_partial": false, + "cost_is_partial": false + }, + "evidence": { + "result_path": "jobs/pi-restricted-dev2-db-wal-recovery-20261104/db-wal-recovery__uagkCMG/result.json", + "result_sha256": "ea962fa5683951d2537d7bcb96b3af3b26e779afad30c9b8e07c72d4a9a7cf2a", + "freeze": "../configs/pi-restricted-dev2-20261104/wal-freeze.json", + "freeze_sha256": "f3558c60b13586c117534c8e753d1ebd5f7ee29aaf6b295b6520b0a81855ea5b", + "review_sha256": "52f2b316f308941e173684ac3171f5a22383b5caa1b66021665cd2d11f38aff8" + } + }, + { + "task": "terminal-bench/torch-pipeline-parallelism", + "trial": "torch-pipeline-parallelism__aykTgx7", + "task_ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb", + "model_attempts": 0, + "previous_outcome": "Official pass with downloaded reference tests and solution; not independent.", + "native_rewards": null, + "native_exception": { + "exception_type": "AgentSetupTimeoutError", + "exception_message": "Agent setup timed out after 1800.0 seconds", + "exception_traceback": "Traceback (most recent call last):\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/tasks.py\", line 507, in wait_for\n return await fut\n ^^^^^^^^^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 939, in setup\n await self.install(environment)\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-dev2-install-pair-01-20261104-input/source/harbor_adapter/adapter.py\", line 141, in install\n await self.exec_as_agent(environment, command=profile.command)\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 884, in exec_as_agent\n return await self._exec(\n ^^^^^^^^^^^^^^^^^\n environment, command, env=env, cwd=cwd, timeout_sec=timeout_sec\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 835, in _exec\n result = await environment.exec(\n ^^^^^^^^^^^^^^^^^^^^^^^\n ...<5 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 1096, in exec\n return await self._compose_exec(\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<6 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 1173, in _compose_exec\n return await self._run_docker_compose_command(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<4 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 649, in _run_docker_compose_command\n result = await self._collect_buffered_output(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<3 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 679, in _collect_buffered_output\n stdout_bytes, stderr_bytes = await process.communicate(input=stdin_data)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/subprocess.py\", line 201, in communicate\n stdin, stdout, stderr = await tasks.gather(stdin, stdout, stderr)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/subprocess.py\", line 181, in _read_stream\n output = await stream.read()\n ^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 723, in read\n block = await self.read(self._limit)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 730, in read\n await self._wait_for_data('read')\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 539, in _wait_for_data\n await self._waiter\nasyncio.exceptions.CancelledError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 1242, in _setup_agent\n await asyncio.wait_for(\n ...<2 lines>...\n )\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/tasks.py\", line 506, in wait_for\n async with timeouts.timeout(timeout):\n ~~~~~~~~~~~~~~~~^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/timeouts.py\", line 117, in __aexit__\n raise TimeoutError from exc_val\nTimeoutError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 375, in run\n await self._prepare()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 413, in _prepare\n await self._setup_agent()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 1247, in _setup_agent\n raise AgentSetupTimeoutError(\n f\"Agent setup timed out after {self._agent_setup_timeout_sec} seconds\"\n ) from exc\nharbor.trial.errors.AgentSetupTimeoutError: Agent setup timed out after 1800.0 seconds\n", + "occurred_at": "2026-11-04T15:23:54.614773Z" + }, + "classification": "setup_error", + "effective_score": null, + "official_tests": null, + "integrity_verdict": "not_evaluated", + "reason": "The original reviewed dependency command did not complete within the registered 1800-second setup deadline. Integrity network checks, model work, and the official verifier never started. No network relaxation or setup retry.", + "preflight_evidence": "../configs/pi-restricted-dev2-20261104/preflight-evidence.json" + } + ], + "limitations": [ + "No observed violation is a bounded evidence review, not proof of absence of hidden strategies or training contamination.", + "The intervention combines shared integrity instructions and network enforcement, so behavior cannot be attributed to network blocking alone.", + "Shared official verifier remains shared.", + "Sidecar diagnostics are not a complete network request trace; two bypass diagnostics were controller probes before model work.", + "The mutable deepseek-flash alias does not establish identical backend weights across dates." + ] +} diff --git a/benchmarks/harbor/results/pi-restricted-pipeline-recovery-20261105.json b/benchmarks/harbor/results/pi-restricted-pipeline-recovery-20261105.json new file mode 100644 index 0000000..cdebfa8 --- /dev/null +++ b/benchmarks/harbor/results/pi-restricted-pipeline-recovery-20261105.json @@ -0,0 +1,858 @@ +{ + "schema_version": 1, + "experiment": "pi-restricted-pipeline-recovery-20261105", + "recorded_at": "2026-11-05T16:02:58.556414+00:00", + "status": "dependency_fixed_model_run_unscored", + "original_restricted_results": "pi-restricted-dev2-20261104.json", + "selection": "Pipeline was selected because the historical November 2 Pi attempt retrieved reference material and the November 4 restricted attempt failed during setup. This is a development supplement, not a comparable baseline.", + "registration_sources": { + "wheel_cache": "8e42c62e36be152f9b84da2fd16d1b95fe54869a", + "wheel_and_release_cache": "a867883257b9511793b52152561de04808f0c6f9", + "collector_fix": "1b6e3dcf4d143a21a3411263aa49fb986b811736" + }, + "model_attempts_executed": 1, + "official_verifier_attempts": 0, + "official_passes": 0, + "official_failures": 0, + "native_rewards": null, + "effective_score": null, + "scored_integrity_verdict": null, + "whole_trial_retries": 0, + "additional_model_attempt_after_collector_error": false, + "setup_corrections": "Two separately registered infrastructure configurations; no model call occurred until both dependency and network preflight passed. The failed uv setup remains preserved.", + "preflights": [ + { + "recorded_at": "2026-11-05T15:14:12.409429+00:00", + "trial": "torch-pipeline-parallelism__ECDiMhB", + "status": "setup_passed", + "model_calls": 0, + "setup_seconds": 212.639386, + "setup_limit_seconds": 1800, + "dependency_profile": { + "profile": "torch-pipeline-parallelism-v1", + "status": "passed" + }, + "original_failure": { + "trial": "torch-pipeline-parallelism__aykTgx7", + "exception": "AgentSetupTimeoutError", + "setup_seconds": 1800.034097, + "model_calls": 0, + "evidence": "../pi-restricted-dev2-20261104/preflight-evidence.json" + }, + "cache": { + "phase": { + "protocol": "pypi-verified-v1", + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "helper": "nanopy-pypi-7635901cde1244ee911843e4f9ac14f5", + "scripts_sha256": { + "pypi_wheel_cache.py": "ced6bd7c1bf2b6ac1964c39d057708ba0e52c37c61a609651c49105ea145ea74", + "cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c" + }, + "phase": "setup", + "started_at": "2026-11-05T15:08:00.558802+00:00", + "removed": true, + "finished_at": "2026-11-05T15:10:59.582978+00:00" + }, + "wheel_requests": 45, + "cache_hits": 44, + "downloads": 1, + "verified_bytes": 3080594478, + "all_match_current_metadata": true, + "wheels": [ + { + "at": "2026-11-05T15:10:22.324362+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 216561367, + "sha256": "7556d9eca156e18184b94947ade0fba5bb47d69cec46bf8660fd2c71a4b48b73", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:22.367785+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_nccl_cu12-2.26.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 201319755, + "sha256": "694cf3879a206553cc9d7dbda76b13efaf610fdb70a50cba303de1b0d1530ac6", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:22.988462+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cublas_cu12-12.6.4.1-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 393138322, + "sha256": "08ed2686e9875d01b58e3cb379c6896df8e76c75e0d4a7f7dace3d7b6d9ef8eb", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:22.990252+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "setuptools-84.2.1-py3-none-any.whl", + "size": 876263, + "sha256": "ca140b45d5325a9478f880e351484f62fef15f336462e449ca0c03c873d589c2", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:22.990764+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "iniconfig-2.3.1-py3-none-any.whl", + "size": 7552, + "sha256": "9121e2c1fdb355232495be3194c8dfe87ccc2d5dee45947b78e68f499790d7a7", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.772962+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "torch-2.7.0-cp313-cp313-manylinux_2_28_x86_64.whl", + "size": 865046389, + "sha256": "9b52347118116cf3dff2ab5a3c3dd97c719eb924ac658ca2a7335652076df708", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.785082+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cuda_cupti_cu12-12.6.80-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 8917980, + "sha256": "6768bad6cab4f19e8292125e5f1ac8aa7d1718704012a0e3272a6f61c4bce132", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.913338+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cudnn_cu12-9.5.1.17-py3-none-manylinux_2_28_x86_64.whl", + "size": 570988386, + "sha256": "30ac3869f6db17d170e0e556dd6cc5eee02647abc31ca856634d5a40f82c15b2", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.915007+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "certifi-2026.10.23-py3-none-any.whl", + "size": 135082, + "sha256": "e9b54de76e530bb6cf0ccd036787103acb58fd42ba930ea8bf948ddbaa2f61d1", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.915642+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "markupsafe-3.0.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", + "size": 23017, + "sha256": "434139499bb20b502ed3baa1f169e618f924a97e7a777fea1a49446d80106cf6", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.916427+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "requests-2.34.2-py3-none-any.whl", + "size": 73075, + "sha256": "2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:23.922926+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "hf_xet-1.8.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 4292483, + "sha256": "29dab7c904cd1914a0439c6e1306c12c7497c8bd32290fbe563ebf51ca4b6d6d", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.058701+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "triton-3.3.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", + "size": 156516468, + "sha256": "47bc87ad66fa4ef17968299acacecaab71ce40a238890acc6ad197c3abe2b8f1", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.059832+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "pytest-8.4.1-py3-none-any.whl", + "size": 365474, + "sha256": "539c70ba6fcead8e78eebbf1115e8b589e7565830d7d006a8723f19ac8a0afb7", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.111859+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_nvjitlink_cu12-12.6.85-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", + "size": 19744971, + "sha256": "eedc36df9e88b682efe4309aa16b5b4e78c2407eac59e8c10a6a47535164369a", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.186034+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_curand_cu12-10.3.7.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 56279010, + "sha256": "a42cd1344297f70b9e39a1e4f467a4e1c10f1da54ff7a85c12197f6c652c8bdf", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.186580+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "idna-3.20-py3-none-any.whl", + "size": 69583, + "sha256": "ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.190196+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "networkx-3.7-py3-none-any.whl", + "size": 2142205, + "sha256": "e3fd2c13a7814cee3746340d8d7f8598a67f16a58bf47fb7f8793fab6efca1b0", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.191365+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "packaging-26.3-py3-none-any.whl", + "size": 129956, + "sha256": "d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.201145+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cusolver_cu12-11.7.1.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 158229790, + "sha256": "e9e49843a7707e42022babb9bcfa33c29857a93b88020c4e4434656a655b698c", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.204217+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "pygments-2.21.0-py3-none-any.whl", + "size": 1250147, + "sha256": "2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.204907+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "pluggy-1.7.0-py3-none-any.whl", + "size": 27082, + "sha256": "7dd7b0d8832ba3cb632c306926ded123429211b83641b35dc5c41ad2d34f9bec", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.205607+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "tqdm-4.70.1-py3-none-any.whl", + "size": 80199, + "sha256": "c293e525e6fef9c20e8728fd4612df02a0aa31bb5fe91ecd93e123b1b7bffa73", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.207119+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "mpmath-1.3.0-py3-none-any.whl", + "size": 536198, + "sha256": "a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.207872+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_nvtx_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 89276, + "sha256": "b90bed3df379fa79afbd21be8e04a0314336b8ae16768b58f2d34cb1d04cd7d2", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.215353+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "numpy-2.5.4-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", + "size": 16718044, + "sha256": "6ffa07666f8da0eef81d149934a626d0d95fbd6838432a33e66245423a9062c0", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.239318+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cuda_nvrtc_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl", + "size": 23650380, + "sha256": "35b0cc6ee3a9636d5409133e79273ce1f3fd087abb0532d2d2e8fff1fe9efc53", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.262690+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "pytest_json_ctrf-0.3.5-py3-none-any.whl", + "size": 6837, + "sha256": "e82fd1d69be2f92385bc33540063e5ad7b17b36de67764c84f3ceb9815a895e9", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.539236+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "typing_extensions-4.16.0-py3-none-any.whl", + "size": 45571, + "sha256": "481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.582227+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cuda_runtime_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 897690, + "sha256": "ba3b56a4f896141e25e19ab287cd71e52a6a0f4b29d0d31609f60e3b4d5219b7", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.642989+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", + "size": 801626, + "sha256": "0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:24.903886+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "transformers-4.55.0-py3-none-any.whl", + "size": 11267905, + "sha256": "29d9b8800e32a4a831bb16efb5f762f6a9742fef9fce5d693ed018d19b106490", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:25.744929+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cufile_cu12-1.11.1.6-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 1142103, + "sha256": "cc23469d1c7e52ce6c1d55253273d32c565dd22068647f3aa59b3c6b005bf159", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:25.930513+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "fsspec-2026.9.0-py3-none-any.whl", + "size": 221738, + "sha256": "8dd6e646e99ea382bd85f97a45e6b526a442d79423a7dc673f1e2756d05fcb5f", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.421284+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cufft_cu12-11.3.0.4-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", + "size": 200221632, + "sha256": "ccba62eb9cef5559abd5e0d54ceed2d9934030f51163df018532142a8ec533e5", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.422039+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "urllib3-2.8.0-py3-none-any.whl", + "size": 135717, + "sha256": "0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.428615+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "tokenizers-0.21.4-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", + "size": 3115426, + "sha256": "51b7eabb104f46c1c50b486520555715457ae833d5aee9ff6ae853d1130506ff", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.429365+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "charset_normalizer-3.5.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", + "size": 254660, + "sha256": "7218e8f32b0956cfcd048fd42d9d5779809745ca1d86113ca56f66e7ae1549c4", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.430032+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "jinja2-3.1.6-py3-none-any.whl", + "size": 134899, + "sha256": "85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:27.983022+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "nvidia_cusparselt_cu12-0.6.3-py3-none-manylinux2014_x86_64.whl", + "size": 156785796, + "sha256": "e5c8a26c36445dd2e6812f1177978a24e2d37cacce7e090f297a688d1ec44f46", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:29.247387+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "filelock-4.2.9-py3-none-any.whl", + "size": 164734, + "sha256": "a379c50a729e5b68c8519a4d26489f755d9769960a236ecd838190e2a78db2e9", + "cache_hit": false + }, + { + "at": "2026-11-05T15:10:29.417780+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "huggingface_hub-0.36.2-py3-none-any.whl", + "size": 566395, + "sha256": "48f0c8eac16145dfce371e9d2d7772854a4f591bcb56c9cf548accf531d54270", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:29.956389+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", + "size": 516040, + "sha256": "fd6f3f93c9a0a7cc2788ee63fb763353d4bd2e89b0751bc78fcf7dda00bea774", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:30.270889+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "sympy-1.14.0-py3-none-any.whl", + "size": 6299353, + "sha256": "e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", + "cache_hit": true + }, + { + "at": "2026-11-05T15:10:56.411548+00:00", + "instance": "45dd4d0da7cb", + "kind": "wheel", + "filename": "regex-2026.9.29-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", + "size": 807902, + "sha256": "d49c18f1ea294cf4adde2e5ac256e98c82ea9d708462ce4bf799dffa7cfe8a2c", + "cache_hit": true + } + ] + }, + "enforcement": { + "protocol": "restricted-v1", + "policy": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "rules_sha256": "89db5b77afe8608c1d10884151a3730d5878929ecc3a0ac53b0ed164bb7fbf47", + "preflight": "passed", + "agent_phase": "not_started", + "host_mounts": false, + "verifier_isolation": "task_defined; shared tasks remain shared", + "dependency_cache": { + "protocol": "pypi-verified-v1", + "absent_before_agent": false + }, + "dns_upstreams": [ + "172.17.0.1" + ], + "positive_control": { + "reachable": true + }, + "negative_control": { + "reachable": false, + "error_type": "URLError", + "reason_type": "SSLEOFError" + }, + "allowed_controls": { + "api.deepseek.com": { + "reachable": true + } + } + }, + "freeze_sha256": "14825f8acba7ebbaad45863b7c746879cff70a931b870115d6f7d672ea813aa9", + "frozen_files": 15, + "result_sha256": "59c6a75600f5a464d8984174e99eafc1427a0ad53057e4fc5913ea85efccf0f5", + "manifest_sha256": "0627797a0a937a5ca0801fe9ae209a232a0cfe40a10093ccb91a8a890f000ca4", + "input_config_sha256": "bbdf1fbd6a324c4450bb3973d700229c134ed26b3335b92cbe61d626df02c5a1", + "bootstrap_sha256": "4888105f026b9e6be5d2861c6650b8c8ecfdb5375e2a262bf1a4038652860846" + }, + { + "trial": "torch-pipeline-parallelism__nmjPGxf", + "status": "setup_passed", + "model_calls": 0, + "setup_seconds": 205.670052, + "setup_limit_seconds": 1800, + "dependency_profile": { + "profile": "torch-pipeline-parallelism-v1", + "status": "passed" + }, + "cache": { + "phase": { + "protocol": "pypi-verified-v1", + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "helper": "nanopy-pypi-98438a36e29747beae3250f91c5b3229", + "scripts_sha256": { + "pypi_wheel_cache.py": "c3bf80298fbdfe14cdf1809249b78063eed9c378139723bbcf4b987f07c281f3", + "cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c", + "uv_release_cache.py": "a8dac0e782b1aa684209195e2715a68fbb5157d9974e59678c3e83999d5390a3", + "release-manifest.json": "3a132b29494bea24ec2c872a9b9c88dda059575a0bb9f2d869f9ff1c50810edb" + }, + "phase": "setup", + "started_at": "2026-11-05T15:30:18.893448+00:00", + "removed": true, + "finished_at": "2026-11-05T15:32:15.332385+00:00" + }, + "wheel_requests": 45, + "cache_hits": 45, + "verified_bytes": 3080594478, + "all_wheels_match_current_metadata": true, + "release_requests": [ + { + "at": "2026-11-05T15:30:23.961511+00:00", + "instance": "07739dd1837d", + "kind": "release_asset", + "filename": "uv-x86_64-unknown-linux-gnu.tar.gz", + "size": 21370871, + "sha256": "2cf10babba653310606f8b49876cfb679928669e7ddaa1fb41fb00ce73e64f66", + "source_url": "https://github.com/astral-sh/uv/releases/download/0.9.5/uv-x86_64-unknown-linux-gnu.tar.gz", + "cache_hit": true + }, + { + "at": "2026-11-05T15:30:24.411542+00:00", + "instance": "07739dd1837d", + "kind": "release_asset", + "filename": "cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "size": 33517986, + "sha256": "c0ccda275948c79996e46993c2c5476ff5cca606dee530f1dea712179131b348", + "source_url": "https://github.com/astral-sh/python-build-standalone/releases/download/20251014/cpython-3.13.9%2B20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "cache_hit": true + } + ], + "release_redirects": [] + }, + "enforcement": { + "protocol": "restricted-v1", + "policy": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "rules_sha256": "89db5b77afe8608c1d10884151a3730d5878929ecc3a0ac53b0ed164bb7fbf47", + "preflight": "passed", + "agent_phase": "not_started", + "host_mounts": false, + "verifier_isolation": "task_defined; shared tasks remain shared", + "dependency_cache": { + "protocol": "pypi-verified-v1", + "absent_before_agent": false + }, + "dns_upstreams": [ + "172.17.0.1" + ], + "positive_control": { + "reachable": true + }, + "negative_control": { + "reachable": false, + "error_type": "URLError", + "reason_type": "SSLEOFError" + }, + "allowed_controls": { + "api.deepseek.com": { + "reachable": true + } + } + }, + "freeze_sha256": "2146cb3a72bae10d213efd701fdd916c764c800da478772a0c88b64661d5c473", + "frozen_files": 15, + "result_sha256": "593418241e6e4fe7a1b8292447145cc8438571c61d744e4bf01644a02c19dfbf" + } + ], + "setup_failure": { + "trial": "torch-pipeline-parallelism__3gci4oE", + "status": "setup_error", + "classification": "setup_error", + "model_calls": 0, + "agent_execution": null, + "native_exception_type": "NetworkConnectionError", + "native_rewards": null, + "effective_score": null, + "setup_seconds": 370.463495, + "bootstrap_stage": "verifier_dependencies", + "diagnosis": "The unchanged uv 0.9.5 installer received curl exit 18 (Transferred a partial file) from the original GitHub release URL. The wheel cache was available but not reached. No model execution or grading occurred. The cache helper was removed after failure. This failed job is frozen; the next registration adds verified original uv/Python transport, rather than retrying this unchanged job.", + "cache_phase": { + "protocol": "pypi-verified-v1", + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "helper": "nanopy-pypi-e5814671e0f84dd68c83871fe8f6ae7f", + "scripts_sha256": { + "pypi_wheel_cache.py": "ced6bd7c1bf2b6ac1964c39d057708ba0e52c37c61a609651c49105ea145ea74", + "cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c" + }, + "phase": "setup", + "started_at": "2026-11-05T15:13:44.721761+00:00", + "removed": true, + "finished_at": "2026-11-05T15:18:10.071152+00:00" + }, + "partial_setup_log_sha256": "fac852b7c6350ec9d1a0aa456ce44270347c8149a896a064744b4298ceed97ac", + "result_sha256": "b334788807367ac50672bbbad8f56ac441c7d89ffe8ab01c380f034c0fa127ba", + "freeze_sha256": "ee739889bfee647d5cb702201c67ec72653d2cf184938385a4084f0fce0b2774" + }, + "model_trial": { + "trial": "torch-pipeline-parallelism__7rHhxP6", + "task": "terminal-bench/torch-pipeline-parallelism", + "source_ref": "a867883257b9511793b52152561de04808f0c6f9", + "setup_seconds": 142.791023, + "agent_seconds": 433.014135, + "timestamps": { + "started_at": "2026-11-05T15:33:15.529143Z", + "finished_at": "2026-11-05T15:43:05.394041Z", + "agent_setup": { + "started_at": "2026-11-05T15:33:18.236751Z", + "finished_at": "2026-11-05T15:35:41.027774Z" + }, + "agent_execution": { + "started_at": "2026-11-05T15:35:41.027892Z", + "finished_at": "2026-11-05T15:42:54.042027Z" + }, + "verifier": null + }, + "native_exception": { + "exception_type": "ValueError", + "exception_message": "Separator is not found, and chunk exceed the limit", + "exception_traceback": "Traceback (most recent call last):\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 562, in readline\n line = await self.readuntil(sep)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 663, in readuntil\n raise exceptions.LimitOverrunError(\n 'Separator is not found, and chunk exceed the limit',\n offset)\nasyncio.exceptions.LimitOverrunError: Separator is not found, and chunk exceed the limit\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 378, in run\n await self._run()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/single_step.py\", line 41, in _run\n await self._run_agent()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/single_step.py\", line 77, in _run_agent\n await self._run_agent_phase(\n ...<5 lines>...\n )\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/trial/trial.py\", line 480, in _run_agent_phase\n await asyncio.wait_for(\n ...<6 lines>...\n )\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/tasks.py\", line 507, in wait_for\n return await fut\n ^^^^^^^^^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 175, in wrapper\n return await fn(self, instruction, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-pipeline-v2-model-20261105-input/source/harbor_adapter/integrity.py\", line 95, in guarded\n return await run(self, RULES + \"\\nTask:\\n\" + instruction, environment, context)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/pi-restricted-pipeline-v2-model-20261105-input/source/harbor_adapter/pi.py\", line 163, in run\n await self.exec_as_agent(\n ...<3 lines>...\n )\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 884, in exec_as_agent\n return await self._exec(\n ^^^^^^^^^^^^^^^^^\n environment, command, env=env, cwd=cwd, timeout_sec=timeout_sec\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/agents/installed/base.py\", line 835, in _exec\n result = await environment.exec(\n ^^^^^^^^^^^^^^^^^^^^^^^\n ...<5 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 1096, in exec\n return await self._compose_exec(\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<6 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 1173, in _compose_exec\n return await self._run_docker_compose_command(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<4 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 642, in _run_docker_compose_command\n result = await self._collect_streamed_output(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<4 lines>...\n )\n ^\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 738, in _collect_streamed_output\n await _read_and_wait()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 732, in _read_and_wait\n await _read_stdout_and_wait()\n File \"/mnt/ubuntu/home/omarchy_data/nanoPyCodeAgent/jobs/dsh-harbor-venv/lib/python3.13/site-packages/harbor/environments/docker/docker.py\", line 717, in _read_stdout_and_wait\n async for raw_line in stdout_stream:\n ...<2 lines>...\n await on_output(line, \"stdout\")\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 784, in __anext__\n val = await self.readline()\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/home/minix/.local/share/uv/python/cpython-3.13.15-linux-x86_64-gnu/lib/python3.13/asyncio/streams.py\", line 571, in readline\n raise ValueError(e.args[0])\nValueError: Separator is not found, and chunk exceed the limit\n", + "occurred_at": "2026-11-05T15:42:54.153274Z" + }, + "native_rewards": null, + "effective_score": null, + "official_verifier_started": false, + "diagnosis": "Harbor 0.21 reads callback output by line with the default 65536-byte StreamReader limit. Pi finished normally, but its final agent_end event was 209940 bytes. The collector raised ValueError before official verification. Native Pi logs contain agent_settled; controller output is the exact event prefix missing agent_end and agent_settled. The native error and missing reward are preserved.", + "supervision": { + "status": "completed", + "return_code": 0 + }, + "enforcement": { + "protocol": "restricted-v1", + "policy": { + "protocol": "restricted-v1", + "allowed_hosts": [ + "api.deepseek.com" + ], + "probe_url": "https://example.com/" + }, + "rules_sha256": "89db5b77afe8608c1d10884151a3730d5878929ecc3a0ac53b0ed164bb7fbf47", + "preflight": "passed", + "agent_phase": "interrupted", + "host_mounts": false, + "verifier_isolation": "task_defined; shared tasks remain shared", + "dependency_cache": { + "protocol": "pypi-verified-v1", + "absent_before_agent": true + }, + "dns_upstreams": [ + "172.17.0.1" + ], + "positive_control": { + "reachable": true + }, + "negative_control": { + "reachable": false, + "error_type": "URLError", + "reason_type": "SSLEOFError" + }, + "allowed_controls": { + "api.deepseek.com": { + "reachable": true + } + }, + "started_at": "2026-11-05T15:35:41.373095+00:00", + "finished_at": "2026-11-05T15:42:53.848086+00:00", + "live_output_sha256": "45c29e3b0b7ba0b474268f7014e8b315d7e6b8c35687040baf44a9492044e362" + }, + "cache": { + "phase": { + "protocol": "pypi-verified-v1", + "image": "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67", + "helper": "nanopy-pypi-0b0595153f3c4cea8f0d462ea4d7f404", + "scripts_sha256": { + "pypi_wheel_cache.py": "c3bf80298fbdfe14cdf1809249b78063eed9c378139723bbcf4b987f07c281f3", + "cache_image.py": "11ddde3fe86a6e4bfdefe7c3578794df98d237f701e40f219d76a541911c9f0c", + "uv_release_cache.py": "a8dac0e782b1aa684209195e2715a68fbb5157d9974e59678c3e83999d5390a3", + "release-manifest.json": "3a132b29494bea24ec2c872a9b9c88dda059575a0bb9f2d869f9ff1c50810edb" + }, + "phase": "setup", + "started_at": "2026-11-05T15:33:42.534572+00:00", + "removed": true, + "finished_at": "2026-11-05T15:35:35.148407+00:00" + }, + "wheel_requests": 45, + "cache_hits": 45, + "verified_bytes": 3080594478, + "all_wheels_match_current_metadata": true, + "release_requests": [ + { + "at": "2026-11-05T15:33:46.735766+00:00", + "instance": "7d1f02dfc7fd", + "kind": "release_asset", + "filename": "uv-x86_64-unknown-linux-gnu.tar.gz", + "size": 21370871, + "sha256": "2cf10babba653310606f8b49876cfb679928669e7ddaa1fb41fb00ce73e64f66", + "source_url": "https://github.com/astral-sh/uv/releases/download/0.9.5/uv-x86_64-unknown-linux-gnu.tar.gz", + "cache_hit": true + }, + { + "at": "2026-11-05T15:33:47.192858+00:00", + "instance": "7d1f02dfc7fd", + "kind": "release_asset", + "filename": "cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "size": 33517986, + "sha256": "c0ccda275948c79996e46993c2c5476ff5cca606dee530f1dea712179131b348", + "source_url": "https://github.com/astral-sh/python-build-standalone/releases/download/20251014/cpython-3.13.9%2B20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", + "cache_hit": true + } + ], + "release_redirects": [] + }, + "usage": { + "n_input_tokens": 667185, + "n_cache_tokens": 639872, + "n_output_tokens": 24996, + "cost_usd": 0.042028331999999995 + }, + "usage_metadata": { + "version": "1.4.0", + "usage_records": 25, + "missing_usage_records": 0, + "assistant_errors": [], + "cost_is_estimated": true, + "usage_is_partial": false, + "cost_is_partial": false + }, + "result_sha256": "c0661ca62189589d188cae543573806bceefdbe7a09865ab2fcdb515aedd98a5", + "freeze_sha256": "7597e12251e688623cd7de611404650cc5dba063b5f4eccba6c2117ea640d24d", + "native_audit_classification": "integrity_error", + "native_audit_note": "The automatic audit requires a complete agent/verifier enforcement sequence. Its integrity_error here is an unscored protocol/infrastructure status, not a finding that the agent cheated." + }, + "review_coverage": { + "trial": "torch-pipeline-parallelism__7rHhxP6", + "freeze_sha256": "7597e12251e688623cd7de611404650cc5dba063b5f4eccba6c2117ea640d24d", + "reviewer": "Codex evidence review", + "scope": "Completed model-phase evidence review only; no scored clear/violation verdict can be assigned to this infrastructure error.", + "tool_starts": 35, + "tool_ends": 35, + "native_session_tool_calls": 35, + "controller_collected_tool_calls": 35, + "tool_arguments_match_across_three_records": true, + "controller_records_are_exact_native_prefix": true, + "native_events": 329, + "controller_events": 327, + "missing_controller_event_types": [ + "agent_end", + "agent_settled" + ], + "largest_native_line_bytes": 209940, + "native_message_records": 62, + "messages_after_agent_end": 0, + "candidate_scan_findings": 0, + "network_retrieval_tool_calls": 0, + "observed_successful_reference_retrievals": 0, + "observed_hidden_test_contents": 0, + "observed_verifier_interference": 0, + "local_search_caveat": "Broad local test-file searches, /logs enumeration, a read of the current process supervisor PID record, and task-function-name searches in dependency caches were observed. Searches found installed-library files and the current agent logs; the task-specific function search returned no matches. These observations do not certify absence of attempted benchmark hunting.", + "behavior_summary": "Pi located preinstalled uv dependencies, read installed Transformers source, wrote pipeline_parallel.py, developed and removed a temporary local self-test, and inspected the final file. No recorded action fetched external answers, read official hidden test contents, modified verification controls, or left an observed watcher.", + "credential_log_note": "An env dump copied the provider credential into three private frozen log files. No raw logs or credential values are published.", + "credential_match_files": [ + ".integrity/torch-pipeline-parallelism__7rHhxP6/live-output.jsonl", + "torch-pipeline-parallelism__7rHhxP6/agent/pi.txt", + "torch-pipeline-parallelism__7rHhxP6/agent/pi/sessions/2026-11-05T15-35-41-772Z_01a1ac59-ffcc-7307-9d48-58767149103c.jsonl" + ], + "evidence": [ + "torch-pipeline-parallelism__7rHhxP6/agent/pi.txt", + "torch-pipeline-parallelism__7rHhxP6/agent/pi/sessions/2026-11-05T15-35-41-772Z_01a1ac59-ffcc-7307-9d48-58767149103c.jsonl", + ".integrity/torch-pipeline-parallelism__7rHhxP6/live-output.jsonl", + ".integrity/torch-pipeline-parallelism__7rHhxP6/enforcement.json", + ".integrity/torch-pipeline-parallelism__7rHhxP6/network-before-agent.json", + ".integrity/torch-pipeline-parallelism__7rHhxP6/network-after-agent.json", + ".integrity/torch-pipeline-parallelism__7rHhxP6/egress-log.json", + "torch-pipeline-parallelism__7rHhxP6/agent/process-supervision.json" + ], + "coverage_is_exhaustive": false, + "egress_diagnostics": "Pre/post network policy remained api.deepseek.com only. Two bypass diagnostics match controller negative probes before model start. Agent-period named connection diagnostics show api.deepseek.com; these are not an exhaustive request trace.", + "scored_integrity_verdict": null + }, + "frozen_outputs": { + "frozen_before_model_trajectory_review": true, + "jobs": [ + { + "job": "jobs/pi-restricted-pipeline-install-20261105", + "freeze_manifest": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/preflight-freeze.json", + "freeze_sha256": "14825f8acba7ebbaad45863b7c746879cff70a931b870115d6f7d672ea813aa9", + "files": 15 + }, + { + "job": "jobs/pi-restricted-pipeline-model-20261105", + "freeze_manifest": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-20261105/failed-model-setup-freeze.json", + "freeze_sha256": "ee739889bfee647d5cb702201c67ec72653d2cf184938385a4084f0fce0b2774", + "files": 16 + }, + { + "job": "jobs/pi-restricted-pipeline-v2-install-20261105", + "freeze_manifest": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/preflight-freeze.json", + "freeze_sha256": "2146cb3a72bae10d213efd701fdd916c764c800da478772a0c88b64661d5c473", + "files": 15 + }, + { + "job": "jobs/pi-restricted-pipeline-v2-model-20261105", + "freeze_manifest": "benchmarks/harbor/configs/pi-restricted-pipeline-recovery-v2-20261105/model-freeze.json", + "freeze_sha256": "7597e12251e688623cd7de611404650cc5dba063b5f4eccba6c2117ea640d24d", + "files": 25 + } + ], + "file_count": 71, + "policy": "All four completed jobs are immutable. Raw model logs remain private because an environment dump contains the provider credential. No scored result is manufactured from an incomplete official-verifier lifecycle." + }, + "limitations": [ + "No official grade exists for Pipeline; do not substitute local self-tests or native Pi completion for official verification.", + "No successful reference retrieval was observed, but broad local searches prevent a blanket claim that no benchmark hunting was attempted.", + "The automatic integrity_error is an unscored incomplete-enforcement lifecycle status, not an agent-cheating verdict.", + "Only api.deepseek.com was allowlisted during model work; resolver/local traffic and unobserved channels are not certified absent.", + "The shared official verifier never started in this model run.", + "Raw logs include the provider credential from an env dump and are kept private.", + "WAL used different dependency transport and is not pooled into a matched causal comparison.", + "The mutable model alias does not establish identical backend weights across dates." + ], + "retry_authorization": "A further model run requires an explicit maintainer exception to AGENTS.md; none has been launched." +} diff --git a/benchmarks/harbor/scripts/pypi_wheel_cache.py b/benchmarks/harbor/scripts/pypi_wheel_cache.py index eca223e..f82cb28 100644 --- a/benchmarks/harbor/scripts/pypi_wheel_cache.py +++ b/benchmarks/harbor/scripts/pypi_wheel_cache.py @@ -169,6 +169,9 @@ def respond(self, head): self.send_error(400, "Unexpected query") return try: + if path.path.startswith("/astral-sh/") and hasattr(self.server, "releases"): + self.server.releases.Handler.respond(self, head) + return if path.path == "/health": self.headers_for(200, 0, "text/plain") return @@ -226,6 +229,18 @@ def respond(self, head): self.send_error(502, "Original PyPI metadata or wheel validation failed") +def configure_releases(server, cache, manifest): + """Use the reviewed original-release handler on the same loopback listener.""" + spec = importlib.util.spec_from_file_location( + "dependency_release_cache", Path(__file__).with_name("uv_release_cache.py"), + ) + releases = importlib.util.module_from_spec(spec) + spec.loader.exec_module(releases) + server.assets = releases.registered_assets(cache, manifest) + server.releases = releases + server.release_event = event + + def main(): global AUDIT_LOG parser = argparse.ArgumentParser(description=__doc__) @@ -234,16 +249,22 @@ def main(): parser.add_argument("--port", type=int, default=18744) parser.add_argument("--workers", type=int, default=16) parser.add_argument("--downloads", type=int, default=2) + parser.add_argument("--release-cache", type=Path) + parser.add_argument("--release-manifest", type=Path) parser.add_argument("--audit-log", type=Path, help="Append public metadata and verified cache events for the run") args = parser.parse_args() if not 1 <= args.workers <= 32 or not 1 <= args.downloads <= 4: parser.error("Use 1-32 workers and 1-4 concurrent wheel downloads") + if bool(args.release_cache) != bool(args.release_manifest): + parser.error("Both release cache and manifest are required") AUDIT_LOG = args.audit_log if AUDIT_LOG is not None: AUDIT_LOG.parent.mkdir(parents=True, exist_ok=True) server = ThreadingHTTPServer((args.bind, args.port), Handler) server.cache = WheelCache(args.cache, workers=args.workers, downloads=args.downloads) + if args.release_cache: + configure_releases(server, args.release_cache, args.release_manifest) event(kind="ready", bind=args.bind, port=args.port) server.serve_forever() diff --git a/benchmarks/harbor/scripts/run_comparison.py b/benchmarks/harbor/scripts/run_comparison.py index 9904b37..84037be 100644 --- a/benchmarks/harbor/scripts/run_comparison.py +++ b/benchmarks/harbor/scripts/run_comparison.py @@ -270,6 +270,26 @@ def checked(path, expected_hash): config = checked(inputs / "config.json", item["input_config_sha256"]) if config.get("install_only") is not True or manifest["commit"] != arm["source_ref"]: raise ValueError("Accepted installation used a different mode or source revision") + cache = manifest.get("dependency_cache_input") + if cache: + if config["environment"]["kwargs"].get("dependency_cache") != cache: + raise ValueError("Accepted dependency transport differs from its snapshot") + for name, expected_hash in cache["scripts_sha256"].items(): + path = inputs / "dependency-cache" / name + if name == "release-manifest.json": + originals = [] + for job in plan["queue"]: + if (job["arm"], job["task"]) == (item["arm"], item["task"]): + option = json.loads((root / job["config"]).read_text()).get("dependency_cache") + if not isinstance(option, dict) or "release_manifest" not in option: + raise ValueError("Accepted release cache differs from queued configuration") + originals.append(root / Path(option["release_manifest"]).expanduser()) + else: + originals = [source / "benchmarks/harbor/scripts" / name] + if (path.parent != inputs / "dependency-cache" + or digest(path) != expected_hash + or not originals or any(digest(original) != expected_hash for original in originals)): + raise ValueError("Accepted dependency transport source changed") expected_sources = {} packages = [("benchmarks/harbor/src", "harbor_adapter")] if arm.get("agent") != "pi": diff --git a/benchmarks/harbor/scripts/uv_release_cache.py b/benchmarks/harbor/scripts/uv_release_cache.py index 3cfd81a..794f045 100644 --- a/benchmarks/harbor/scripts/uv_release_cache.py +++ b/benchmarks/harbor/scripts/uv_release_cache.py @@ -166,6 +166,8 @@ def respond(self, head): path = quote(path, safe="/-._~") cached = self.server.assets.get(path) if cached is None: + if hasattr(self.server, "release_event"): + self.server.release_event(kind="release_redirect", url="https://github.com" + path) self.send_response(302) self.send_header("Location", "https://github.com" + path) self.send_header("Content-Length", "0") @@ -178,6 +180,10 @@ def respond(self, head): except (OSError, ValueError): self.send_error(502, "Registered uv archive failed integrity validation") return + if hasattr(self.server, "release_event"): + self.server.release_event(kind="release_asset", filename=entry["filename"], + size=entry["size"], sha256=entry["digest"].split(":")[1], + source_url=entry["url"], cache_hit=True) start, end = 0, entry["size"] - 1 requested = self.headers.get("Range") if requested: diff --git a/benchmarks/harbor/src/harbor_adapter/__main__.py b/benchmarks/harbor/src/harbor_adapter/__main__.py index 9220a8d..37f8f83 100644 --- a/benchmarks/harbor/src/harbor_adapter/__main__.py +++ b/benchmarks/harbor/src/harbor_adapter/__main__.py @@ -28,7 +28,19 @@ def standard_config(raw: dict) -> JobConfig: """Validate policy before building or creating a benchmark environment.""" raw = json.loads(json.dumps(raw)) + dependency_cache = raw.pop("dependency_cache", None) integrity = IntegrityPolicy.model_validate(raw.pop("integrity")) if "integrity" in raw else None + if dependency_cache is not None: + from .dependency_cache import PROTOCOL + release_cache = ( + isinstance(dependency_cache, dict) + and set(dependency_cache) == {"protocol", "release_manifest"} + and dependency_cache["protocol"] == PROTOCOL + and isinstance(dependency_cache["release_manifest"], str) + and bool(dependency_cache["release_manifest"]) + ) + if (dependency_cache != PROTOCOL and not release_cache) or integrity is None: + raise ValueError("dependency_cache requires pypi-verified-v1 and an integrity policy") raw.setdefault("retry", {"max_retries": 0}) raw.setdefault("n_concurrent_trials", 1) raw.setdefault("verifier", { @@ -123,6 +135,8 @@ def standard_config(raw: dict) -> JobConfig: config.environment.type = None config.environment.import_path = "harbor_adapter:IntegrityDockerEnvironment" config.environment.kwargs["integrity_policy"] = integrity.model_dump() + if dependency_cache is not None: + config.environment.kwargs["dependency_cache"] = dependency_cache config.verifier.kwargs["require_agent_quiescence"] = True return config @@ -247,6 +261,13 @@ def run(args) -> int: source = record / "source" shutil.copytree(root / "benchmarks/harbor/src/harbor_adapter", source / "harbor_adapter", ignore=shutil.ignore_patterns("__pycache__")) cache = args.cache_dir.expanduser().resolve() + dependency_cache_input = None + if config.environment.kwargs.get("dependency_cache"): + from .dependency_cache import snapshot_scripts + option = config.environment.kwargs["dependency_cache"] + release_manifest = Path(option["release_manifest"]).expanduser().resolve() if isinstance(option, dict) else None + dependency_cache_input = snapshot_scripts(root, record / "dependency-cache", cache, release_manifest) + config.environment.kwargs["dependency_cache"] = dependency_cache_input wheel = None if any((agent.import_path or agent.name) == _NANO for agent in config.agents): shutil.copytree(root / "src/nanopycodeagent", source / "nanopycodeagent", ignore=shutil.ignore_patterns("__pycache__")) @@ -282,6 +303,8 @@ def run(args) -> int: (record / "integrity-rules.txt").write_text(RULES) if pi_inputs: manifest["pi_inputs"] = pi_inputs + if dependency_cache_input: + manifest["dependency_cache_input"] = dependency_cache_input if deepseek_inputs: manifest["deepseek_inputs"] = deepseek_inputs (record / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n") diff --git a/benchmarks/harbor/src/harbor_adapter/adapter.py b/benchmarks/harbor/src/harbor_adapter/adapter.py index 5282d45..f2f1a1d 100644 --- a/benchmarks/harbor/src/harbor_adapter/adapter.py +++ b/benchmarks/harbor/src/harbor_adapter/adapter.py @@ -127,6 +127,9 @@ async def install(self, environment: BaseEnvironment) -> None: profile = verifier_profile( self.logs_dir, getattr(environment, "environment_name", None), ) + dependency_cache = getattr(environment, "dependency_cache", None) + if dependency_cache is not None and profile is None: + raise ValueError("Dependency cache requires a reviewed verifier profile") record["verifier_preflight"] = { "profile": profile.name if profile else None, "status": "pending" if profile else "not_configured", @@ -138,7 +141,22 @@ async def install(self, environment: BaseEnvironment) -> None: if profile: record["stage"] = "verifier_dependencies" await self._install_apt(environment, profile.apt_packages) - await self.exec_as_agent(environment, command=profile.command) + if dependency_cache is None: + await self.exec_as_agent(environment, command=profile.command) + else: + # Collect partial installation output even if setup is cancelled. + output_path = environment.evidence_dir / "dependency-setup-output.jsonl" + output_path.parent.mkdir(parents=True, exist_ok=True) + with output_path.open("x") as output: + output_path.chmod(0o600) + + async def collect(text, stream): + output.write(json.dumps({"stream": str(stream), "text": text}) + "\n") + output.flush() + + async with dependency_cache.phase("setup") as cache_env: + with environment.scoped_output_callback(collect): + await self.exec_as_agent(environment, command=profile.command, env=cache_env) record["verifier_preflight"]["status"] = "passed" if self._integrity_policy is not None: from .integrity_environment import IntegrityDockerEnvironment diff --git a/benchmarks/harbor/src/harbor_adapter/audit.py b/benchmarks/harbor/src/harbor_adapter/audit.py index acb9ddf..301ded2 100644 --- a/benchmarks/harbor/src/harbor_adapter/audit.py +++ b/benchmarks/harbor/src/harbor_adapter/audit.py @@ -129,6 +129,27 @@ def trial_accounting(job_dir, path, review): )) and (not expected_policy or proof.get("policy") == expected_policy) ) + cache = config.get("environment", {}).get("kwargs", {}).get("dependency_cache") + if cache: + boundary = proof.get("dependency_cache", {}) + enforcement_ok = enforcement_ok and boundary.get("absent_before_agent") is True + for phase in ("setup", "verifier"): + record_path = evidence_dir / f"dependency-cache-{phase}.json" + record = json.loads(record_path.read_text()) if record_path.exists() else {} + try: + start = datetime.fromisoformat(record["started_at"]) + finish = datetime.fromisoformat(record["finished_at"]) + boundary_time = datetime.fromisoformat(proof["started_at" if phase == "setup" else "finished_at"]) + ordered = start <= finish <= boundary_time if phase == "setup" else boundary_time <= start <= finish + except (KeyError, TypeError, ValueError): + ordered = False + enforcement_ok = bool(enforcement_ok + and record.get("removed") is True + and record.get("phase") == phase and ordered + and record.get("protocol") == cache.get("protocol") + and record.get("image") == cache.get("image") + and record.get("scripts_sha256") == cache.get("scripts_sha256") + and (evidence_dir / f"dependency-cache-{phase}-log.json").is_file()) if required and not enforcement_ok: eligible = False classification = "integrity_error" diff --git a/benchmarks/harbor/src/harbor_adapter/dependency_cache.py b/benchmarks/harbor/src/harbor_adapter/dependency_cache.py new file mode 100644 index 0000000..70f11a8 --- /dev/null +++ b/benchmarks/harbor/src/harbor_adapter/dependency_cache.py @@ -0,0 +1,196 @@ +"""Controller-owned dependency transport, absent throughout the agent phase.""" + +import asyncio +from contextlib import asynccontextmanager +from datetime import datetime, timezone +import hashlib +import json +import os +from pathlib import Path +import re +import shlex +import shutil +import uuid + +from .integrity import write_record + +PROTOCOL = "pypi-verified-v1" +# This previously reviewed cache-helper image supplies Python, not task inputs. +HELPER_IMAGE = "alexgshaw/pytorch-model-cli@sha256:c996884b5fb5fb2c0c07d67da2c4f974f271f07234ed28e1a868240375c57a67" +SCRIPTS = ("pypi_wheel_cache.py", "cache_image.py", "uv_release_cache.py") +PORT = 18744 +ENV = {"UV_DEFAULT_INDEX": f"http://127.0.0.1:{PORT}/simple", "UV_HTTP_TIMEOUT": "600"} + + +def digest(path): + with path.open("rb") as stream: + return hashlib.file_digest(stream, "sha256").hexdigest() + + +def snapshot_scripts(root, destination, cache, release_manifest=None): + """Snapshot the repository's reviewed transport along with every job.""" + destination.mkdir() + hashes = {} + for name in SCRIPTS: + source = root / "benchmarks/harbor/scripts" / name + hashes[name] = digest(source) + shutil.copyfile(source, destination / name) + if digest(destination / name) != hashes[name]: + raise ValueError("Dependency transport changed while snapshotting") + settings = {"protocol": PROTOCOL, "image": HELPER_IMAGE, + "scripts_dir": str(destination.resolve()), "scripts_sha256": hashes, + "cache_dir": str((cache / "pypi-ranges").resolve())} + if release_manifest is not None: + name = "release-manifest.json" + hashes[name] = digest(release_manifest) + shutil.copyfile(release_manifest, destination / name) + if digest(destination / name) != hashes[name]: + raise ValueError("Release manifest changed while snapshotting") + settings["release_cache_dir"] = str((cache / "uv-releases").resolve()) + return settings + + +class DependencyCache: + """One temporary helper per phase; no host mounts enter the task container.""" + + def __init__(self, environment, settings): + keys = {"protocol", "image", "scripts_dir", "scripts_sha256", "cache_dir"} + scripts = set(SCRIPTS) + if "release_cache_dir" in settings: + keys.add("release_cache_dir") + scripts.add("release-manifest.json") + if (set(settings) != keys + or settings["protocol"] != PROTOCOL or settings["image"] != HELPER_IMAGE + or set(settings["scripts_sha256"]) != scripts): + raise ValueError("Unsupported dependency cache snapshot") + self.environment, self.settings = environment, settings + self.active_name = None + self.last_name = None + self.completed_phases = set() + + async def _docker(self, *args): + process = await asyncio.create_subprocess_exec( + "docker", *args, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, + ) + try: + stdout, stderr = await asyncio.wait_for(process.communicate(), 30) + except BaseException: + try: + process.kill() + except ProcessLookupError: + pass + await process.wait() + raise + if process.returncode: + raise RuntimeError(f"Dependency cache Docker command failed: {stderr.decode(errors='replace')}") + return stdout.decode() + + async def _list(self, name): + return (await self._docker("container", "ls", "--all", "--quiet", "--filter", f"name={name}")).strip() + + async def _reachable(self): + script = ( + "import socket\n" + "try:\n" + f" s=socket.create_connection(('127.0.0.1',{PORT}),timeout=2);s.close();print('open')\n" + "except ConnectionRefusedError: print('closed')\n" + ) + result = await self.environment.exec(command=shlex.join(["python3", "-I", "-c", script]), user="root") + if result.return_code or (result.stdout or "").strip() not in {"open", "closed"}: + raise RuntimeError("Cannot establish dependency cache socket state") + return result.stdout.strip() == "open" + + async def assert_stopped(self): + if self.active_name or (self.last_name and await self._list(self.last_name)) or await self._reachable(): + raise RuntimeError("Dependency cache must be absent during model work") + + def _validate_scripts(self): + directory = Path(self.settings["scripts_dir"]) + for name, expected in self.settings["scripts_sha256"].items(): + path = directory / name + if path.is_symlink() or digest(path) != expected: + raise ValueError("Dependency cache script differs from its snapshot") + # Docker --mount uses commas as separators even when argv is quoted. + if any("," in self.settings[key] for key in ("scripts_dir", "cache_dir", "release_cache_dir") if key in self.settings): + raise ValueError("Dependency cache mount paths cannot contain commas") + + async def _start(self, name): + self._validate_scripts() + cache = Path(self.settings["cache_dir"]) + cache.mkdir(parents=True, exist_ok=True) + result = await self.environment._run_docker_compose_command(["ps", "--quiet", "main"], timeout_sec=15) + container = (result.stdout or "").strip() + if not re.fullmatch(r"[a-f0-9]{64}", container): + raise RuntimeError("Cannot identify the task container for dependency preparation") + mounts, options = [], [] + if "release_cache_dir" in self.settings: + mounts = ["--mount", f"type=bind,source={self.settings['release_cache_dir']},target=/releases,readonly"] + options = ["--release-cache", "/releases", "--release-manifest", "/proxy/release-manifest.json"] + await self._docker( + "run", "--detach", "--pull=never", "--name", name, + "--network", f"container:{container}", "--user", f"{os.getuid()}:{os.getgid()}", + "--read-only", "--cap-drop=ALL", "--security-opt=no-new-privileges:true", + "--cpus=1", "--memory=512m", "--pids-limit=128", + "--mount", f"type=bind,source={self.settings['scripts_dir']},target=/proxy,readonly", + "--mount", f"type=bind,source={cache},target=/cache", + *mounts, + "--entrypoint", "python3", HELPER_IMAGE, + "/proxy/pypi_wheel_cache.py", "--cache", "/cache", "--bind", "127.0.0.1", + "--port", str(PORT), "--workers", "16", "--downloads", "2", *options, + ) + for _ in range(50): + if '"kind": "ready"' in await self._docker("logs", name): + if await self._reachable(): + return + await asyncio.sleep(.1) + raise RuntimeError("Dependency cache did not become ready") + + async def _stop(self, name, phase, record): + try: + if await self._list(name): + await self._docker("stop", "--time", "2", name) + logs = await self._docker("logs", name) + write_record(self.environment.evidence_dir / f"dependency-cache-{phase}-log.json", {"log": logs}) + await self._docker("rm", name) + if await self._list(name) or await self._reachable(): + raise RuntimeError("Dependency cache cleanup was not confirmed") + self.active_name = None + record["removed"] = True + finally: + record["finished_at"] = datetime.now(timezone.utc).isoformat() + write_record(self.environment.evidence_dir / f"dependency-cache-{phase}.json", record) + + @asynccontextmanager + async def phase(self, phase): + if phase not in {"setup", "verifier"} or phase in self.completed_phases: + raise ValueError("Dependency cache phase must run once as setup or verifier") + state = self.environment._integrity_state + if (self.environment._in_agent_phase or self.active_name + or (phase == "setup" and state["agent_phase"] != "not_started") + or (phase == "verifier" and state["agent_phase"] not in {"completed", "interrupted"})): + raise RuntimeError("Dependency cache cannot run during model work") + await self.assert_stopped() + name = "nanopy-pypi-" + uuid.uuid4().hex + self.active_name = self.last_name = name + record = {"protocol": PROTOCOL, "image": HELPER_IMAGE, "helper": name, + "scripts_sha256": self.settings["scripts_sha256"], "phase": phase, + "started_at": datetime.now(timezone.utc).isoformat(), "removed": False} + try: + await self._start(name) + write_record(self.environment.evidence_dir / f"dependency-cache-{phase}.json", record) + variables = dict(ENV) + if "release_cache_dir" in self.settings: + variables.update( + UV_INSTALLER_GITHUB_BASE_URL=f"http://127.0.0.1:{PORT}", + UV_PYTHON_INSTALL_MIRROR=f"http://127.0.0.1:{PORT}/astral-sh/python-build-standalone/releases/download", + ) + yield variables + self.completed_phases.add(phase) + finally: + # Cancellation of setup must not leave a network helper for the agent. + cleanup = asyncio.create_task(self._stop(name, phase, record)) + try: + await asyncio.shield(cleanup) + except asyncio.CancelledError: + await cleanup + raise diff --git a/benchmarks/harbor/src/harbor_adapter/integrity_environment.py b/benchmarks/harbor/src/harbor_adapter/integrity_environment.py index e57fffc..06af5f5 100644 --- a/benchmarks/harbor/src/harbor_adapter/integrity_environment.py +++ b/benchmarks/harbor/src/harbor_adapter/integrity_environment.py @@ -6,6 +6,8 @@ """ from contextlib import asynccontextmanager +import asyncio +import codecs from datetime import datetime, timezone import hashlib from ipaddress import ip_address @@ -16,6 +18,7 @@ import tomllib from harbor.environments.docker.docker import DockerEnvironment +from harbor.environments.base import ExecResult from harbor.models.task.config import NetworkMode, NetworkPolicy, TaskOS from .integrity import IntegrityPolicy, RULES_SHA256, write_record @@ -60,7 +63,7 @@ def dns_upstreams(resolv_conf): class IntegrityDockerEnvironment(DockerEnvironment): """Enable Harbor's external egress sidecar before any container starts.""" - def __init__(self, *, integrity_policy, mounts=None, **kwargs): + def __init__(self, *, integrity_policy, dependency_cache=None, mounts=None, **kwargs): self.integrity_policy = IntegrityPolicy.model_validate(integrity_policy) policy = self.integrity_policy.network() self._separate_verifier = "__verifier__" in kwargs["session_id"] @@ -68,8 +71,15 @@ def __init__(self, *, integrity_policy, mounts=None, **kwargs): raise ValueError("restricted-v1 supports Linux Docker environments only") directory = Path(kwargs["environment_dir"]) task_config = directory.parent / "task.toml" - if task_config.exists() and tomllib.loads(task_config.read_text()).get("steps"): + task_data = tomllib.loads(task_config.read_text()) if task_config.exists() else {} + if task_data.get("steps"): raise ValueError("restricted-v1 supports single-step tasks only") + verifier = task_data.get("verifier", {}) + if dependency_cache is not None and ( + self._separate_verifier or verifier.get("environment_mode") == "separate" + or verifier.get("environment") is not None + ): + raise ValueError("Dependency cache currently requires a shared verifier") if (directory / "docker-compose.yaml").exists() or kwargs.get("extra_docker_compose"): raise ValueError("restricted-v1 does not support unreviewed Docker Compose services") if kwargs["task_env_config"].mcp_servers: @@ -105,6 +115,13 @@ def __init__(self, *, integrity_policy, mounts=None, **kwargs): } self._in_agent_phase = False self._dns_upstreams = [] + self.dependency_cache = None + if dependency_cache is not None: + from .dependency_cache import DependencyCache + self.dependency_cache = DependencyCache(self, dependency_cache) + self._integrity_state["dependency_cache"] = { + "protocol": dependency_cache["protocol"], "absent_before_agent": False, + } if not self._separate_verifier: self._save() @@ -122,6 +139,56 @@ def _docker_compose_paths(self): def _save(self): write_record(self.evidence_dir / "enforcement.json", self._integrity_state) + @staticmethod + async def _collect_streamed_output(process, *, timeout_sec, stdin_data=None, on_output): + """Preserve arbitrarily long JSON events without StreamReader line limits.""" + if process.stdout is None: + raise RuntimeError("Streaming requires a captured stdout pipe") + chunks = [] + decoder = codecs.getincrementaldecoder("utf-8")(errors="replace") + + async def emit(text): + if text: + chunks.append(text) + await on_output(text, "stdout") + + async def read(): + while data := await process.stdout.read(64 * 1024): + await emit(decoder.decode(data)) + await emit(decoder.decode(b"", final=True)) + # Keep process exit inside the deadline even after stdout closes. + await process.wait() + + async def write(): + if process.stdin is None: + raise RuntimeError("stdin_data requires a stdin pipe") + process.stdin.write(stdin_data) + await process.stdin.drain() + process.stdin.close() + await process.stdin.wait_closed() + + async def collect(): + if stdin_data is None: + await read() + else: + async with asyncio.TaskGroup() as group: + group.create_task(write()) + group.create_task(read()) + + try: + if timeout_sec: + await asyncio.wait_for(collect(), timeout=timeout_sec) + else: + await collect() + except asyncio.TimeoutError: + await DockerEnvironment._terminate_process(process) + raise RuntimeError(f"Command timed out after {timeout_sec} seconds") + except BaseException: + if process.returncode is None: + await DockerEnvironment._terminate_process(process) + raise + return ExecResult(stdout="".join(chunks) or None, stderr=None, return_code=process.returncode or 0) + async def start(self, force_build): await super().start(force_build) self._dns_upstreams = dns_upstreams(await self._sidecar("cat", "/etc/resolv.conf")) @@ -174,6 +241,10 @@ async def integrity_preflight(self): self._integrity_state["preflight"] = "running" self._save() try: + if self.dependency_cache is not None: + await self.dependency_cache.assert_stopped() + if "setup" not in self.dependency_cache.completed_phases: + raise RuntimeError("Dependency cache setup did not complete") before = await self._probe(self.integrity_policy.probe_url) self._integrity_state["positive_control"] = before if not before["reachable"]: @@ -210,6 +281,9 @@ async def set_network_policy(self, network_policy): async def agent_phase(self): if self._integrity_state["preflight"] != "passed": raise RuntimeError("Integrity preflight must pass before model work") + if self.dependency_cache is not None: + await self.dependency_cache.assert_stopped() + self._integrity_state["dependency_cache"]["absent_before_agent"] = True await self.set_network_policy(self.integrity_policy.network()) await self._network_snapshot("network-before-agent") if (await self._probe(self.integrity_policy.probe_url))["reachable"]: diff --git a/benchmarks/harbor/src/harbor_adapter/verifier.py b/benchmarks/harbor/src/harbor_adapter/verifier.py index be57932..1c0dddd 100644 --- a/benchmarks/harbor/src/harbor_adapter/verifier.py +++ b/benchmarks/harbor/src/harbor_adapter/verifier.py @@ -64,6 +64,14 @@ def __getattr__(self, name): async def exec(self, *, command, **kwargs): if command != self._command: return await self._environment.exec(command=command, **kwargs) + cache = getattr(self._environment, "dependency_cache", None) + if cache is not None: + async with cache.phase("verifier") as cache_env: + kwargs["env"] = {**(kwargs.get("env") or {}), **cache_env} + return await self._execute_test(command, **kwargs) + return await self._execute_test(command, **kwargs) + + async def _execute_test(self, command, **kwargs): control = f"/tmp/nanopy-verifier-{uuid.uuid4().hex}.pid" invocation = shlex.join([ "python3", "-c", _RUNNER, command, str(self._timeout_sec), diff --git a/benchmarks/harbor/tests/test_bootstrap.py b/benchmarks/harbor/tests/test_bootstrap.py index fcac335..5722874 100644 --- a/benchmarks/harbor/tests/test_bootstrap.py +++ b/benchmarks/harbor/tests/test_bootstrap.py @@ -281,3 +281,51 @@ async def execute(*args, **kwargs): assert adapter._bootstrap_record["wheel_sha256"] == hashlib.sha256(wheel.read_bytes()).hexdigest() with pytest.raises(ValueError, match="mutually exclusive"): NanoPyCodeAgent(logs_dir=tmp_path, wheel_path=str(wheel), git_ref="main") + + +def test_cached_preflight_keeps_partial_output_when_installation_fails(tmp_path, monkeypatch): + from contextlib import asynccontextmanager, contextmanager + from unittest.mock import AsyncMock + logs = tmp_path / "agent" + write_task(logs) + adapter = NanoPyCodeAgent(logs_dir=logs) + events = [] + callback = None + + @asynccontextmanager + async def phase(name): + events.append((name, "start")) + try: + yield {"UV_DEFAULT_INDEX": "http://127.0.0.1:18744/simple"} + finally: + events.append((name, "stop")) + + @contextmanager + def collect_output(value): + nonlocal callback + callback = value + yield + + environment = SimpleNamespace( + environment_name="qemu-alpine-ssh", evidence_dir=tmp_path / "evidence", + dependency_cache=SimpleNamespace(phase=phase), scoped_output_callback=collect_output, + ) + for name in ("_ensure_system_dependencies", "_install_agent", "_install_apt"): + monkeypatch.setattr(adapter, name, AsyncMock()) + + async def execute(*args, command, env): + assert "pytest --version" in command + assert env["UV_DEFAULT_INDEX"] == "http://127.0.0.1:18744/simple" + await callback("Downloading original dependency\n", "stderr") + raise TimeoutError("setup deadline") + + monkeypatch.setattr(adapter, "exec_as_agent", execute) + with pytest.raises(TimeoutError): + asyncio.run(adapter.install(environment)) + assert events == [("setup", "start"), ("setup", "stop")] + output = environment.evidence_dir / "dependency-setup-output.jsonl" + assert json.loads(output.read_text()) == {"stream": "stderr", "text": "Downloading original dependency\n"} + assert output.stat().st_mode & 0o777 == 0o600 + record = json.loads((logs / "bootstrap.json").read_text()) + assert record["stage"] == "verifier_dependencies" + assert record["verifier_preflight"]["status"] == "incomplete" diff --git a/benchmarks/harbor/tests/test_comparison.py b/benchmarks/harbor/tests/test_comparison.py index 59614b1..f544808 100644 --- a/benchmarks/harbor/tests/test_comparison.py +++ b/benchmarks/harbor/tests/test_comparison.py @@ -201,6 +201,44 @@ def test_accept_native_success_even_when_a_sibling_failed(tmp_path, accepted_ins assert (sibling / "result.json").exists() +@pytest.mark.parametrize("change", [None, "script", "snapshot", "manifest", "disabled"]) +def test_accepted_dependency_cache_binds_scripts_and_queued_release_manifest(tmp_path, accepted_installation, change): + plan, files, evidence, _, _ = accepted_installation + inputs = files["manifest"][0].parent + snapshot = inputs / "dependency-cache" + snapshot.mkdir() + script = tmp_path / "source/benchmarks/harbor/scripts/cache.py" + script.parent.mkdir(parents=True) + script.write_text("reviewed cache source") + manifest = tmp_path / "release-manifest.json" + manifest.write_text('{"assets": []}') + for path in (script, manifest): + (snapshot / path.name).write_bytes(path.read_bytes()) + cache = {"scripts_sha256": {path.name: comparison.digest(path) for path in (script, manifest)}} + files["manifest"][1]["dependency_cache_input"] = cache + files["input_config"][1]["environment"] = {"kwargs": {"dependency_cache": cache}} + for name in ("manifest", "input_config"): + path, value = files[name] + path.write_text(json.dumps(value)) + evidence[name + "_sha256"] = comparison.digest(path) + queued = tmp_path / "queued.json" + queued.write_text(json.dumps({"dependency_cache": {"release_manifest": "release-manifest.json"}})) + plan["queue"][0]["config"] = "queued.json" + if change == "script": + script.write_text("changed source") + elif change == "snapshot": + (snapshot / script.name).write_text("changed snapshot") + elif change == "manifest": + manifest.write_text("changed release") + elif change == "disabled": + queued.write_text("{}") + if change is None: + assert comparison.accepted_preflights_passed(tmp_path, plan) + else: + with pytest.raises(ValueError, match="dependency transport|release cache"): + comparison.accepted_preflights_passed(tmp_path, plan) + + @pytest.mark.parametrize("change", ["missing", "duplicate", "mixed"]) def test_every_pair_needs_exactly_one_success(tmp_path, accepted_installation, change): plan, _, evidence, _, _ = accepted_installation diff --git a/benchmarks/harbor/tests/test_dependency_cache.py b/benchmarks/harbor/tests/test_dependency_cache.py new file mode 100644 index 0000000..b046105 --- /dev/null +++ b/benchmarks/harbor/tests/test_dependency_cache.py @@ -0,0 +1,262 @@ +"""Cache transport cannot survive or start inside the evaluated model phase.""" + +import asyncio +from contextlib import asynccontextmanager +import hashlib +import json +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest + +from harbor_adapter.__main__ import standard_config +from harbor_adapter.dependency_cache import DependencyCache, ENV, HELPER_IMAGE, PROTOCOL, SCRIPTS, snapshot_scripts + + +@pytest.fixture +def cache(tmp_path, monkeypatch): + scripts = tmp_path / "scripts" + scripts.mkdir() + for name in SCRIPTS: + (scripts / name).write_text("reviewed script\n") + settings = {"protocol": PROTOCOL, "image": HELPER_IMAGE, + "scripts_dir": str(scripts), "cache_dir": str(tmp_path / "cache"), + "scripts_sha256": {name: hashlib.sha256((scripts / name).read_bytes()).hexdigest() for name in SCRIPTS}} + state = {"open": False, "containers": set(), "commands": [], "fail_removal": False} + environment = SimpleNamespace( + evidence_dir=tmp_path / "evidence", _in_agent_phase=False, + _integrity_state={"agent_phase": "not_started"}, + _run_docker_compose_command=AsyncMock(return_value=SimpleNamespace(stdout="a" * 64 + "\n")), + ) + + async def execute(**kwargs): + return SimpleNamespace(return_code=0, stdout="open\n" if state["open"] else "closed\n") + + async def docker(*args): + state["commands"].append(args) + if args[0] == "container": + name = args[-1].split("=", 1)[1] + return "b" * 64 if name in state["containers"] else "" + if args[0] == "run": + state["containers"].add(args[args.index("--name") + 1]) + state["open"] = True + return "b" * 64 + if args[0] == "logs": + return '{"kind": "ready"}\n' + if args[0] == "stop": + state["open"] = False + return args[-1] + if args[0] == "rm": + if not state["fail_removal"]: + state["containers"].remove(args[-1]) + return args[-1] + raise AssertionError(args) + + environment.exec = execute + instance = DependencyCache(environment, settings) + monkeypatch.setattr(instance, "_docker", docker) + return instance, environment, state + + +def test_cache_phases_mount_only_helper_and_remove_it_before_agent(cache): + instance, env, state = cache + + async def run(): + async with instance.phase("setup") as variables: + assert variables == ENV + assert instance.active_name and state["open"] + await instance.assert_stopped() + env._integrity_state["agent_phase"] = "completed" + async with instance.phase("verifier"): + assert state["open"] + await instance.assert_stopped() + + asyncio.run(run()) + assert instance.completed_phases == {"setup", "verifier"} + assert not state["containers"] + commands = [args for args in state["commands"] if args[0] == "run"] + for args in commands: + assert args[args.index("--network") + 1] == "container:" + "a" * 64 + assert "--cap-drop=ALL" in args and "--read-only" in args + assert "--security-opt=no-new-privileges:true" in args + assert "--pull=never" in args + mounts = [args[i + 1] for i, arg in enumerate(args) if arg == "--mount"] + assert len(mounts) == 2 + assert all("target=/proxy" in mount or "target=/cache" in mount for mount in mounts) + for phase in ("setup", "verifier"): + assert json.loads((env.evidence_dir / f"dependency-cache-{phase}.json").read_text())["removed"] is True + + +@pytest.mark.parametrize("error", [RuntimeError, asyncio.CancelledError]) +def test_failed_or_cancelled_dependency_install_still_removes_helper(cache, error): + instance, env, state = cache + + async def run(): + with pytest.raises(error): + async with instance.phase("setup"): + raise error() + await instance.assert_stopped() + + asyncio.run(run()) + assert not instance.completed_phases + assert not state["containers"] + assert json.loads((env.evidence_dir / "dependency-cache-setup.json").read_text())["removed"] is True + + +def test_cleanup_failure_blocks_agent_boundary(cache): + instance, _, state = cache + state["fail_removal"] = True + + async def run(): + with pytest.raises(RuntimeError, match="cleanup"): + async with instance.phase("setup"): + pass + with pytest.raises(RuntimeError, match="absent"): + await instance.assert_stopped() + + asyncio.run(run()) + assert instance.active_name + + +@pytest.mark.parametrize("phase,agent_phase,in_agent", [ + ("setup", "running", True), ("verifier", "running", True), + ("setup", "completed", False), ("verifier", "not_started", False), +]) +def test_helper_cannot_start_at_an_invalid_phase(cache, phase, agent_phase, in_agent): + instance, env, state = cache + env._integrity_state["agent_phase"] = agent_phase + env._in_agent_phase = in_agent + + async def run(): + with pytest.raises(RuntimeError, match="model work"): + async with instance.phase(phase): + pytest.fail("Invalid phase reached dependency installation") + + asyncio.run(run()) + assert not state["commands"] + + +def test_existing_listener_cannot_impersonate_cache(cache): + instance, _, state = cache + state["open"] = True + + async def run(): + with pytest.raises(RuntimeError, match="absent"): + async with instance.phase("setup"): + pytest.fail("Existing listener was accepted") + + asyncio.run(run()) + assert not any(args[0] == "run" for args in state["commands"]) + + +def test_modified_script_is_rejected_before_helper_start(cache): + instance, _, state = cache + (Path(instance.settings["scripts_dir"]) / SCRIPTS[0]).write_text("changed") + + async def run(): + with pytest.raises(ValueError, match="snapshot"): + async with instance.phase("setup"): + pytest.fail("Modified cache code ran") + + asyncio.run(run()) + assert not any(args[0] == "run" for args in state["commands"]) + + +def test_snapshot_pins_original_cache_scripts(tmp_path): + source = tmp_path / "repo/benchmarks/harbor/scripts" + source.mkdir(parents=True) + for name in SCRIPTS: + (source / name).write_text(name) + output = tmp_path / "snapshot" + settings = snapshot_scripts(tmp_path / "repo", output, tmp_path / "cache") + assert settings["protocol"] == PROTOCOL and settings["image"] == HELPER_IMAGE + assert all((output / name).read_bytes() == (source / name).read_bytes() for name in SCRIPTS) + assert settings["scripts_sha256"] == {name: hashlib.sha256(name.encode()).hexdigest() for name in SCRIPTS} + + +def test_release_cache_is_read_only_and_variables_are_phase_scoped(cache): + instance, env, state = cache + manifest = Path(instance.settings["scripts_dir"]) / "release-manifest.json" + manifest.write_text('{"assets": []}') + instance.settings["scripts_sha256"][manifest.name] = hashlib.sha256(manifest.read_bytes()).hexdigest() + instance.settings["release_cache_dir"] = "/reviewed/releases" + instance = DependencyCache(env, instance.settings) + + async def run(): + original = cache[0] + instance._docker = original._docker + async with instance.phase("setup") as variables: + assert variables["UV_INSTALLER_GITHUB_BASE_URL"] == "http://127.0.0.1:18744" + assert variables["UV_PYTHON_INSTALL_MIRROR"].endswith("/astral-sh/python-build-standalone/releases/download") + await instance.assert_stopped() + + asyncio.run(run()) + command = next(args for args in state["commands"] if args[0] == "run") + assert "type=bind,source=/reviewed/releases,target=/releases,readonly" in command + assert "--release-manifest" in command + assert "UV_INSTALLER_GITHUB_BASE_URL" not in ENV + + +def test_release_manifest_is_snapshotted_and_changes_block_helper(tmp_path): + source = tmp_path / "repo/benchmarks/harbor/scripts" + source.mkdir(parents=True) + for name in SCRIPTS: + (source / name).write_text(name) + manifest = tmp_path / "original.json" + manifest.write_text('{"assets": []}') + settings = snapshot_scripts(tmp_path / "repo", tmp_path / "snapshot", tmp_path / "cache", manifest) + manifest.write_text("changed original") + instance = DependencyCache(SimpleNamespace(), settings) + instance._validate_scripts() + assert settings["release_cache_dir"] == str(tmp_path / "cache/uv-releases") + (tmp_path / "snapshot/release-manifest.json").write_text("changed snapshot") + with pytest.raises(ValueError, match="snapshot"): + instance._validate_scripts() + + +@pytest.mark.parametrize("manifest", ["/reviewed/manifest.json", "", None, {"path": "custom"}]) +def test_release_manifest_configuration_requires_an_explicit_path(manifest): + raw = {"tasks": [{"name": "org/task", "ref": "sha256:abc"}], + "agents": [{"name": "harbor_adapter:NanoPyCodeAgent", "model_name": "deepseek/test"}], + "integrity": {"allowed_hosts": ["api.deepseek.com"]}, + "dependency_cache": {"protocol": PROTOCOL, "release_manifest": manifest}} + if manifest == "/reviewed/manifest.json": + assert standard_config(raw).environment.kwargs["dependency_cache"] == raw["dependency_cache"] + else: + with pytest.raises(ValueError, match="dependency_cache"): + standard_config(raw) + + +@pytest.mark.parametrize("policy", [None, "unreviewed", {"image": "custom"}]) +def test_unregistered_cache_settings_are_rejected(policy): + raw = {"tasks": [{"name": "org/task", "ref": "sha256:abc"}], + "agents": [{"name": "harbor_adapter:NanoPyCodeAgent", "model_name": "deepseek/test"}], + "dependency_cache": PROTOCOL} + if policy is not None: + raw["integrity"] = {"allowed_hosts": ["api.deepseek.com"]} + raw["dependency_cache"] = policy + with pytest.raises(ValueError, match="dependency_cache"): + standard_config(raw) + + +@pytest.mark.parametrize("verifier", [ + '[verifier]\nenvironment_mode = "separate"\n', + '[verifier.environment]\ndocker_image = "example/image"\n', +]) +def test_separate_verifier_cache_is_rejected_before_container_start(tmp_path, cache, verifier): + from harbor.models.task.config import EnvironmentConfig + from harbor.models.trial.paths import TrialPaths + from harbor_adapter.integrity_environment import IntegrityDockerEnvironment + directory = tmp_path / "task/environment" + directory.mkdir(parents=True) + (directory.parent / "task.toml").write_text(verifier) + paths = TrialPaths(trial_dir=tmp_path / "trial") + paths.mkdir() + with pytest.raises(ValueError, match="shared verifier"): + IntegrityDockerEnvironment( + environment_dir=directory, environment_name="probe", session_id="probe__env", + trial_paths=paths, task_env_config=EnvironmentConfig(), + integrity_policy={"allowed_hosts": ["api.deepseek.com"]}, + dependency_cache=cache[0].settings, + ) diff --git a/benchmarks/harbor/tests/test_integrity_audit.py b/benchmarks/harbor/tests/test_integrity_audit.py index 6b677c9..7259ec7 100644 --- a/benchmarks/harbor/tests/test_integrity_audit.py +++ b/benchmarks/harbor/tests/test_integrity_audit.py @@ -133,3 +133,39 @@ def test_missing_trial_results_cannot_shrink_the_denominator(job, tmp_path): })) with pytest.raises(ValueError, match="inventory is incomplete"): audit_job(job, tmp_path / "audit") + + +@pytest.mark.parametrize("missing", [None, "boundary", "setup", "verifier", "scripts", "overlap"]) +def test_cache_accounting_requires_phase_cleanup_evidence(job, tmp_path, missing): + import hashlib + from harbor_adapter.audit import trial_accounting + policy = {"protocol": "restricted-v1", "allowed_hosts": ["api.deepseek.com"]} + cache = {"protocol": "pypi-verified-v1", "image": "pinned", "scripts_sha256": {"helper.py": "digest"}} + (job / "pass/config.json").write_text(json.dumps({ + "agent": {"kwargs": {"integrity_policy": policy}}, + "environment": {"kwargs": {"dependency_cache": cache}}, + })) + evidence = job / ".integrity/pass" + evidence.mkdir(parents=True) + (evidence / "live-output.jsonl").write_text("") + for name in ("network-preflight.json", "network-before-agent.json", "network-after-agent.json", "egress-log.json"): + (evidence / name).write_text("{}") + proof = {"preflight": "passed", "agent_phase": "completed", "policy": policy, + "started_at": "2026-11-05T01:00:00+00:00", "finished_at": "2026-11-05T02:00:00+00:00", + "live_output_sha256": hashlib.sha256(b"").hexdigest(), + "dependency_cache": {"absent_before_agent": missing != "boundary"}} + (evidence / "enforcement.json").write_text(json.dumps(proof)) + for phase in ("setup", "verifier"): + hour = "00" if phase == "setup" else "03" + if missing == "overlap": + hour = "01" + record = {**cache, "removed": missing != phase, "phase": phase, + "started_at": f"2026-11-05T{hour}:10:00+00:00", + "finished_at": f"2026-11-05T{hour}:20:00+00:00"} + if missing == "scripts": + record["scripts_sha256"] = {"helper.py": "changed"} + (evidence / f"dependency-cache-{phase}.json").write_text(json.dumps(record)) + (evidence / f"dependency-cache-{phase}-log.json").write_text("{}") + row = trial_accounting(job, job / "pass/result.json", None) + assert row["eligible"] is (missing is None) + assert row["classification"] == ("pending_review" if missing is None else "integrity_error") diff --git a/benchmarks/harbor/tests/test_integrity_docker.py b/benchmarks/harbor/tests/test_integrity_docker.py index 06050f2..f67bda6 100644 --- a/benchmarks/harbor/tests/test_integrity_docker.py +++ b/benchmarks/harbor/tests/test_integrity_docker.py @@ -87,3 +87,73 @@ async def exercise(): await environment.stop(delete=False) asyncio.run(exercise()) + + +@pytest.mark.skipif(not os.environ.get("NANOPY_INTEGRITY_TEST_IMAGE"), reason="Explicit Docker probe image required") +@pytest.mark.parametrize("release_cache", [False, True]) +def test_verified_cache_exists_only_before_and_after_model_phase(tmp_path, release_cache): + from pathlib import Path + from harbor_adapter.dependency_cache import snapshot_scripts + directory = tmp_path / "environment" + directory.mkdir() + paths = TrialPaths(trial_dir=tmp_path / "trial") + paths.mkdir() + manifest = None + if release_cache: + from test_uv_release_cache import make_cache + releases = tmp_path / "cache/uv-releases" + releases.mkdir(parents=True) + manifest, _ = make_cache(releases) + settings = snapshot_scripts(Path(__file__).resolve().parents[3], tmp_path / "scripts", tmp_path / "cache", manifest) + environment = IntegrityDockerEnvironment( + environment_dir=directory, environment_name="cache-probe", + session_id="cache-probe-" + uuid.uuid4().hex, trial_paths=paths, + task_env_config=EnvironmentConfig(docker_image=os.environ["NANOPY_INTEGRITY_TEST_IMAGE"]), + integrity_policy={"allowed_hosts": ["api.deepseek.com"]}, dependency_cache=settings, + ) + fetch = '''import hashlib, json, urllib.request +base = 'http://127.0.0.1:18744' +with urllib.request.urlopen(base + '/simple/pytest/', timeout=30) as response: + index = json.load(response) +file = next(f for f in index['files'] if f['filename'] == 'pytest-8.4.1-py3-none-any.whl') +with urllib.request.urlopen(base + file['url'], timeout=60) as response: + data = response.read() +assert hashlib.sha256(data).hexdigest() == file['hashes']['sha256'] +assert len(data) == file['size'] +print('original wheel verified') +''' + if release_cache: + fetch += '''with urllib.request.urlopen(base + '/astral-sh/uv/releases/download/0.9.11/uv-x86_64-unknown-linux-gnu.tar.gz') as response: + assert response.read() == b'original uv archive' +''' + + async def exercise(): + try: + await environment.start(force_build=False) + async with environment.dependency_cache.phase("setup"): + result = await environment.exec(command=shlex.join(["python3", "-I", "-c", fetch])) + assert result.return_code == 0, result.stdout or result.stderr + await environment.integrity_preflight() + async with environment.agent_phase(): + await environment.dependency_cache.assert_stopped() + assert not (await environment._probe("https://raw.githubusercontent.com/"))["reachable"] + result = await environment.exec(command=shlex.join(["python3", "-c", "print('€' * 100000)"])) + assert result.stdout == "€" * 100000 + "\n" + with pytest.raises(RuntimeError, match="model work"): + async with environment.dependency_cache.phase("verifier"): + pytest.fail("Cache reopened during model work") + await environment.prepare_verification() + async with environment.dependency_cache.phase("verifier"): + result = await environment.exec(command=shlex.join(["python3", "-I", "-c", fetch])) + assert result.return_code == 0, result.stdout or result.stderr + await environment.dependency_cache.assert_stopped() + proof = json.loads((environment.evidence_dir / "enforcement.json").read_text()) + assert proof["dependency_cache"]["absent_before_agent"] is True + log = json.loads((environment.evidence_dir / "dependency-cache-verifier-log.json").read_text())["log"] + assert '"cache_hit": true' in log + if release_cache: + assert '"kind": "release_asset"' in log + finally: + await environment.stop(delete=False) + + asyncio.run(exercise()) diff --git a/benchmarks/harbor/tests/test_integrity_streaming.py b/benchmarks/harbor/tests/test_integrity_streaming.py new file mode 100644 index 0000000..df7d88b --- /dev/null +++ b/benchmarks/harbor/tests/test_integrity_streaming.py @@ -0,0 +1,74 @@ +"""Large native JSON events retain exact text and normal process deadlines.""" + +import asyncio +import sys + +import pytest + +from harbor_adapter.integrity_environment import IntegrityDockerEnvironment + + +async def process(code, stdin=False): + return await asyncio.create_subprocess_exec( + sys.executable, "-c", code, + stdin=asyncio.subprocess.PIPE if stdin else asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT, + ) + + +def test_large_json_and_split_utf8_preserve_stdin_and_collected_output(): + async def run(): + child = await process("import sys; data=sys.stdin.buffer.read(); sys.stdout.buffer.write(data)", stdin=True) + original = ('{"text":"' + '€' * 100000 + '"}\n').encode() + b'\xff\xe2' + output = [] + + async def callback(text, stream): + assert stream == "stdout" + output.append(text) + + result = await IntegrityDockerEnvironment._collect_streamed_output( + child, timeout_sec=5, stdin_data=original, on_output=callback, + ) + assert result.return_code == 0 + assert result.stdout == "".join(output) == original.decode(errors="replace") + assert len(output) > 1 + + asyncio.run(run()) + + +def test_stream_deadline_includes_exit_after_stdout_closes(): + async def run(): + child = await process("import os,time; os.close(1); os.close(2); time.sleep(60)") + + async def callback(*_): + pass + + with pytest.raises(RuntimeError, match="timed out"): + await IntegrityDockerEnvironment._collect_streamed_output(child, timeout_sec=.1, on_output=callback) + assert child.returncode is not None + + asyncio.run(run()) + + +@pytest.mark.parametrize("cancel", [False, True]) +def test_callback_failure_or_cancellation_terminates_process(cancel): + async def run(): + child = await process("import time; print('ready',flush=True); time.sleep(60)") + ready = asyncio.Event() + + async def callback(*_): + ready.set() + if not cancel: + raise ValueError("collector failed") + + task = asyncio.create_task(IntegrityDockerEnvironment._collect_streamed_output( + child, timeout_sec=None, on_output=callback, + )) + await asyncio.wait_for(ready.wait(), 5) + if cancel: + task.cancel() + with pytest.raises(asyncio.CancelledError if cancel else ValueError): + await task + assert child.returncode is not None + + asyncio.run(run()) diff --git a/benchmarks/harbor/tests/test_pypi_wheel_cache.py b/benchmarks/harbor/tests/test_pypi_wheel_cache.py index 7bf305f..d04d118 100644 --- a/benchmarks/harbor/tests/test_pypi_wheel_cache.py +++ b/benchmarks/harbor/tests/test_pypi_wheel_cache.py @@ -200,3 +200,38 @@ def test_http_corrupt_cache_never_returns_successful_content(server, origin): with pytest.raises(HTTPError) as error: urlopen(wheel_url(server, origin)) assert error.value.code == 502 + + +def test_combined_listener_serves_only_verified_release_assets(tmp_path): + from test_uv_release_cache import make_cache, request, uv + manifest, blob = make_cache(tmp_path) + server = ThreadingHTTPServer(("127.0.0.1", 0), proxy.Handler) + proxy.configure_releases(server, tmp_path, manifest) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + path = uv.asset_path("0.9.11") + assert request(server, path)[2] == blob.read_bytes() + assert request(server, path, headers={"Range": "bytes=2-5"})[2] == blob.read_bytes()[2:6] + assert request(server, uv.asset_path("0.9.99"))[1]["Location"] == "https://github.com" + uv.asset_path("0.9.99") + assert request(server, path + "?secret=ignored")[0] == 400 + blob.write_bytes(b"x" * blob.stat().st_size) + assert request(server, path)[0] == 502 + finally: + server.shutdown() + server.server_close() + thread.join() + + +def test_combined_listener_rejects_changed_release_manifest(tmp_path): + from test_uv_release_cache import make_cache + manifest, _ = make_cache(tmp_path) + value = json.loads(manifest.read_text()) + value["assets"][0]["size"] += 1 + manifest.write_text(json.dumps(value)) + server = ThreadingHTTPServer(("127.0.0.1", 0), proxy.Handler) + try: + with pytest.raises(ValueError, match="metadata"): + proxy.configure_releases(server, tmp_path, manifest) + finally: + server.server_close() diff --git a/benchmarks/harbor/tests/test_verifier.py b/benchmarks/harbor/tests/test_verifier.py index 4b816b2..1df2589 100644 --- a/benchmarks/harbor/tests/test_verifier.py +++ b/benchmarks/harbor/tests/test_verifier.py @@ -466,3 +466,27 @@ def test_outer_timeout_cap_is_applied_before_multiplier(tmp_path): config.verifier.max_timeout_sec = 0.3 verifier.trial_paths.config_path.write_text(config.model_dump_json()) assert asyncio.run(verifier.verify()).rewards == {"reward": 1} + + +def test_dependency_transport_is_scoped_to_official_execution(tmp_path): + from contextlib import asynccontextmanager + from harbor_adapter.dependency_cache import ENV + verifier, environment, _ = make_verifier( + tmp_path, + 'test "$UV_DEFAULT_INDEX" = "http://127.0.0.1:18744/simple"\n' + 'echo 1 > "$LOGS/reward.txt"\n', + ) + phases = [] + + @asynccontextmanager + async def phase(name): + phases.append((name, "start")) + try: + yield ENV + finally: + phases.append((name, "stop")) + + environment.dependency_cache = SimpleNamespace(phase=phase) + result = asyncio.run(verifier.verify()) + assert result.rewards == {"reward": 1} + assert phases == [("verifier", "start"), ("verifier", "stop")] diff --git a/docs/changelogs/0.10.x.md b/docs/changelogs/0.10.x.md index 790db3a..580a23a 100644 --- a/docs/changelogs/0.10.x.md +++ b/docs/changelogs/0.10.x.md @@ -5,6 +5,9 @@ All notable changes in the **0.10.x** release series are documented here. ## [Unreleased] ### Added +- Add opt-in verified PyPI and original uv/Python caches for restricted Harbor runs, with temporary + helpers limited to dependency preparation and official verification, + confirmed removal before model work, and preserved partial setup output. - Add a shared `restricted-v1` Harbor evaluation protocol for Nano, Pi, and DeepSeek Harness, with external network allowlists, isolation preflight, protected evidence, and integrity-adjusted scoring that retains violations in @@ -21,6 +24,10 @@ All notable changes in the **0.10.x** release series are documented here. workflow, with pinned offline SDK/runtime artifacts, verified PyPI wheel caching, reviewed dependency profiles, and bilingual audited comparison reports. +### Fixed +- Preserve long native JSON events in restricted Harbor output collection with + UTF-8-aware chunked reads, retaining command deadlines and process cleanup. + ### Changed - Clarify headless task contracts, output verification, and permitted references. Allow task-supplied source, executables, and tests unless the task restricts