Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,23 @@ Do not label a document as `hand-written` unless its contents are actually
authored and maintained by humans. For agent-authored source-of-truth documents,
use wording such as `Chinese source` instead.

### Bilingual benchmark reports

Harbor benchmark reports MUST be generated and maintained in both languages:

- `benchmarks/harbor/reports/en/` contains the English reports.
- `benchmarks/harbor/reports/zh-CN/` contains complete Simplified Chinese
translations with the same filenames as their English counterparts.
- When creating or updating a report, generate or refresh both versions in the
same change. Preserve all results, caveats, tables, evidence references,
identifiers, commands, and hashes across the pair; do not substitute a summary
for a full translation.
- Keep relative links valid after moving or translating reports. English
documents should link to English reports, and Chinese documents to Chinese
reports; links between reports should stay in the current language.
- Before opening or landing a PR that changes reports, verify that both
language versions are present and in sync, and report that status to the user.

### Bilingual research notes

Research notes are bilingual:
Expand Down
8 changes: 8 additions & 0 deletions benchmarks/harbor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,14 @@ valid scored failures stop immediately. Exhausted dependency failures raise
canonical reward. The default remains timeout-only; record the authorization
and selected policy in the experiment plan.

## Benchmark reports

Reports are available in [English](reports/en/) and
[Simplified Chinese](reports/zh-CN/), with matching filenames. Create or update
both full versions together, preserving results and evidence references. Raw
run evidence stays in Git-ignored `jobs/`; machine-readable results remain in
`results/`.

## Test the adapter

```bash
Expand Down
6 changes: 6 additions & 0 deletions benchmarks/harbor/README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,12 @@ Harbor 原生的 provider 凭证和已经配置的 base URL 也会转换为 nano
评分也会立即停止。依赖错误耗尽尝试后抛出 `VerifierDependencyError`,保留每次
证据并清除当前评分。默认仍只重试超时;实验计划应记录授权和实际选用的策略。

## Benchmark 报告

报告提供[英文](reports/en/)和[简体中文](reports/zh-CN/)两个版本,文件名一一对应。
新增或更新报告时,应同时生成或刷新两个完整版本,保留全部结果和证据引用。
原始运行证据保留在由 Git 忽略的 `jobs/`,机器可读结果仍放在 `results/`。

## 测试 adapter

```bash
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
**Completed:** sampler passed all nine official tests on the first verification;
pipeline failed two of four cases on the second verification after its first
attempt timed out during dependency preparation. No third attempt was started.
See the [report](../../reports/deepseek-v41-flash-1017-recovery-20261018.md) and
See the [report](../../reports/en/deepseek-v41-flash-1017-recovery-20261018.md) and
[audited results](../../results/deepseek-v41-flash-1017-recovery-20261018.json).

The user requested another attempt on `adaptive-rejection-sampler` and
`torch-pipeline-parallelism`, with at most three attempts. The initial
[six-trial run](../../reports/deepseek-v41-flash-1017-rerun-20261018.md) left
[six-trial run](../../reports/en/deepseek-v41-flash-1017-rerun-20261018.md) left
these two tasks unscored because verifier dependencies failed to download.
The original containers were deleted and their artifact manifests contain no
`/app` outputs, so this is a **new model trial per task**, not a regrade of the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@

**Completed:** three passes, one scored failure, and two unscored verifier
dependency-download errors. See the
[final report](../../reports/deepseek-v41-flash-1017-rerun-20261018.md) and
[final report](../../reports/en/deepseek-v41-flash-1017-rerun-20261018.md) and
[audited results](../../results/deepseek-v41-flash-1017-rerun-20261018.json).
The sampler and pipeline native zeros are retained as raw evidence; their
analytical scores are null because official tests never started.

This experiment selects all six failed tasks from the latest
[34-task baseline](../../reports/dev34-main-baseline-20261012.md):
[34-task baseline](../../reports/en/dev34-main-baseline-20261012.md):
`adaptive-rejection-sampler`, `db-wal-recovery`, `kv-store-grpc`,
`query-optimize`, `torch-pipeline-parallelism`, and `regex-chess`.
The [selection](selection.json) binds the previous result file and trial IDs.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The other 55 tasks are left unopened by this experiment.

Completed result: **10/12 passed**, with two scored correctness failures and no
final setup or verifier errors. See the
[baseline report](../../reports/dev12-main-baseline-20261011.md) and
[baseline report](../../reports/en/dev12-main-baseline-20261011.md) and
[machine-readable evidence](../../results/dev12-main-baseline-20261011.json).

The runtime is main at `2eb99b608dfb39464d17d109a4a8f11afda607a9`, isolated in
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ none of their model outputs or rewards count toward this run.

**Completed:** 28/34 passed (82.35%), with six scored failures, zero model-phase
setup errors, zero verifier errors, and no verifier retries. See the
[final report](../../reports/dev34-main-baseline-20261012.md) and
[final report](../../reports/en/dev34-main-baseline-20261012.md) and
[audited results](../../results/dev34-main-baseline-20261012.json).
One passing tensor-parallelism trial accessed external benchmark revision
metadata; its raw score is retained with a separate exposure flag.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ the common eligible result is **18/19 versus 17/19**, with one improvement
(`regex-chess`), no observed regression, and one shared failure (pipeline).
Both arms have zero unscored trials. Estimated model cost rises 30.3% and summed
agent time rises 21.6%. This is one adaptive exploratory run per task and arm.
See the [full report](../../reports/headless-p0-guarded-20261007.md) and
See the [full report](../../reports/en/headless-p0-guarded-20261007.md) and
[structured evidence](../../results/tb21-headless-p0-guarded-20261007.json).

This [revised plan](experiment.json) evaluates the P0 contract/verification
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Bounded truncation recovery experiment

The [predeclared plan](experiment.json) selects the two failures from the
[v0.9.0 pilot20 baseline](../../reports/v0.9.0-baseline-20261005.md), with their
[v0.9.0 pilot20 baseline](../../reports/en/v0.9.0-baseline-20261005.md), with their
original revisions and budgets. Each task receives one fresh model run; no
score replaces a trial in the released baseline.

Expand Down Expand Up @@ -35,6 +35,6 @@ uv run --project benchmarks/harbor python -m harbor_adapter run \

Record both task scores and actual recovery events. A task that passes without
reaching `max_tokens` does not demonstrate recovery effectiveness. The
[experiment report](../../reports/truncation-recovery-20261005.md) and
[experiment report](../../reports/en/truncation-recovery-20261005.md) and
[structured results](../../results/tb21-truncation-recovery-20261005.json)
preserve that distinction.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Three additional runs per task

This [predeclared plan](experiment.json) follows the earlier
[two-task experiment](../../reports/truncation-recovery-20261005.md), which
[two-task experiment](../../reports/en/truncation-recovery-20261005.md), which
passed both tasks without exercising truncation recovery. Run each task three
more times from scratch, preserving all six results. These are additional
samples, not replacements for earlier scores.
Expand Down Expand Up @@ -41,5 +41,5 @@ tool execution, terminal outcome, score, and accounting completeness per trial.
A passing trial without truncation does not demonstrate recovery effectiveness.

The recorded run used checkout `382cb38413d7d5368ad1be97e85142f280172a5b`.
See the [six-run report](../../reports/truncation-recovery-repeat3-20261005.md)
See the [six-run report](../../reports/en/truncation-recovery-repeat3-20261005.md)
and [structured results](../../results/tb21-truncation-recovery-repeat3-20261005.json).
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ infrastructure failures rather than model failures.

The complete configuration, per-trial usage, token-estimated cost, and balance
snapshots are in the
[public results](../results/tb21-deepseek-official-pilot20-20260930.json). Raw
[public results](../../results/tb21-deepseek-official-pilot20-20260930.json). Raw
evidence remains in the recorded Git-ignored `jobs/` directories.

## Configuration
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@ original containers had been deleted and their artifact manifests held no
It does not regrade or prove correctness of the original unscored submissions.
Those results and frozen artifacts remain unchanged.

The [registered plan](../configs/deepseek-v41-flash-1017-recovery-20261018/experiment.json)
The [registered plan](../../configs/deepseek-v41-flash-1017-recovery-20261018/experiment.json)
has SHA256
`7ba0d4083cbc856a364417bc51025a8bd84434eb39319927664bfb4c018fba9d`.
The [audited results](../results/deepseek-v41-flash-1017-recovery-20261018.json)
The [audited results](../../results/deepseek-v41-flash-1017-recovery-20261018.json)
retain trial IDs, all verifier attempts, actual pytest and native CTRF counts,
source and wheel hashes, preserved-output hashes, authorization, and preflight
and exposure findings. The
[configuration README](../configs/deepseek-v41-flash-1017-recovery-20261018/README.md)
[configuration README](../../configs/deepseek-v41-flash-1017-recovery-20261018/README.md)
documents execution and local evidence paths.

Both fresh model trials used `deepseek/deepseek-flash` against the official
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ are not counted as model failures.
These are fresh, single trials of all six failures from the
[October 12 baseline](dev34-main-baseline-20261012.md), using the same agent and
adapter source, prompt, task revisions, resources, and work budgets. The
[machine-readable results](../results/deepseek-v41-flash-1017-rerun-20261018.json)
[machine-readable results](../../results/deepseek-v41-flash-1017-rerun-20261018.json)
contain trial IDs, raw rewards, analytical scores, timing, source and artifact
hashes, preflight history, and audit findings. The
[final registration](../configs/deepseek-v41-flash-1017-rerun-20261018/experiment-v2.json)
[final registration](../../configs/deepseek-v41-flash-1017-rerun-20261018/experiment-v2.json)
has SHA256
`9c9306f3eb17c3044987587a7329671160ee902a43c2b67be6de4f857e4c96c2`.

Expand Down Expand Up @@ -88,7 +88,7 @@ The archive's upstream size, filename, and SHA256 were verified with the
repository's existing cache utility. All six verifier dependency profiles were
explicitly **`not_configured`**; successful agent installation did not certify
their verifier dependencies. The
[configuration README](../configs/deepseek-v41-flash-1017-rerun-20261018/README.md)
[configuration README](../../configs/deepseek-v41-flash-1017-rerun-20261018/README.md)
documents both plans and the retained interruption evidence. Original images,
package sources, official test scripts, resources, and verifier time limits
were preserved.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ shares its unproven assumption. More self-tests alone would not address either.
setup has a separate 1,800-second allowance. Original verifier limits remain
unchanged, with at most three attempts and only timeouts permitting a retry.

[Selection and configuration](../configs/dev12-baseline-20261011/README.md),
[active plan](../configs/dev12-baseline-20261011/experiment-v3.json), and
[machine-readable results](../results/dev12-main-baseline-20261011.json) contain
[Selection and configuration](../../configs/dev12-baseline-20261011/README.md),
[active plan](../../configs/dev12-baseline-20261011/experiment-v3.json), and
[machine-readable results](../../results/dev12-main-baseline-20261011.json) contain
the revisions, budgets, artifact hashes, metrics, and post-run diagnosis.

## Results
Expand Down Expand Up @@ -137,8 +137,8 @@ choice. Label accuracy should not substitute for behavioral equivalence.
The original install-only preflight had five successes, six image-pull/start
errors, and one APT setup timeout. The second had eleven successes and one
partial uv download. Both complete histories remain in the
[first](../configs/dev12-baseline-20261011/preflight-v1-evidence.json) and
[second](../configs/dev12-baseline-20261011/preflight-v2-evidence.json) evidence
[first](../../configs/dev12-baseline-20261011/preflight-v1-evidence.json) and
[second](../../configs/dev12-baseline-20261011/preflight-v2-evidence.json) evidence
files. These eight preflight failures are separate from the twelve scored
trials.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ revision metadata. Its official pass is retained and the exposure is reported
separately below. This is an already exposed development pool, not a held-out
benchmark or a claim of evaluation without external benchmark-material access.

The [machine-readable results](../results/dev34-main-baseline-20261012.json)
The [machine-readable results](../../results/dev34-main-baseline-20261012.json)
contain all trial identifiers, source/task/configuration provenance, native
scores, test counts, timing, partial usage, failure diagnoses, and evidence
hashes. Raw trajectories and logs remain local because they can contain secrets.
Expand Down Expand Up @@ -38,7 +38,7 @@ that main commit. No agent or prompt optimization was included.
| Agent work | Native task timeout minus 180 seconds |
| Verifier | Original per-attempt limit; timeout-only retries allowed, none used |

The final [v5 registration](../configs/dev34-main-baseline-20261012/experiment-v5.json)
The final [v5 registration](../../configs/dev34-main-baseline-20261012/experiment-v5.json)
has SHA256 `f7bc7b329767f4b8007956462f6cdbd4407d386af493524df27ba313c0a4e4c3`.
All 34 jobs installed the same wheel, SHA256
`f4dcc18758e584696f955ae57fd18c751599b70e8b497ca2db7ef9e2b16e9350`.
Expand Down Expand Up @@ -234,7 +234,7 @@ Each task had separate cache sidecars in its own network namespace, no published
ports, no task filesystem or credential mounts, and separate CPU/memory limits.
All 34 model trials used the final v5 transport. Details, original failure
evidence, manifests, and network integration probes are linked from the
[run configuration README](../configs/dev34-main-baseline-20261012/README.md).
[run configuration README](../../configs/dev34-main-baseline-20261012/README.md).
The supporting Harbor workflow suite passed **167 tests** before model work;
the corresponding branch commit also passed all three CI checks.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ and PyTorch pipeline numerical correctness failed their independent verifiers;
normal finalization did not imply task success.

The complete configuration, metrics, and all retained trial rows are in the
[public results](../results/tb21-dualbudget-fixed-provider-20260929.json), including
[public results](../../results/tb21-dualbudget-fixed-provider-20260929.json), including
the four superseded treatment trials. Raw evidence remains in the recorded
Git-ignored `jobs/` directories.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ without selection, and all twenty guarded-candidate trials were fresh. There are
no held-out tasks or repeat samples. A score difference on this cohort does not
establish a stable or general capability improvement.

The [structured results](../results/tb21-headless-p0-guarded-20261007.json) preserve
The [structured results](../../results/tb21-headless-p0-guarded-20261007.json) preserve
every trial, paired outcome, accounting limitation, integrity finding, and artifact
hash. See the [guarded plan](../configs/headless-p0-guarded-20261007/experiment.json)
and [reproduction instructions](../configs/headless-p0-guarded-20261007/README.md).
hash. See the [guarded plan](../../configs/headless-p0-guarded-20261007/experiment.json)
and [reproduction instructions](../../configs/headless-p0-guarded-20261007/README.md).

## Candidate and frozen inputs

Expand Down Expand Up @@ -223,8 +223,8 @@ reference. The candidate arm was interrupted at 15:06 UTC. All five started
trials are preserved: **three passes, one installation failure before model work,
and one contaminated trial cancelled during verification**. Fifteen candidate
trials never started. These records are not replaced by or combined with the
new candidate's outcomes. See the [original-candidate audit](../results/headless-p0-original-candidate-audit-20261007.json)
and [original plan](../configs/headless-p0-20261007/experiment.json).
new candidate's outcomes. See the [original-candidate audit](../../results/headless-p0-original-candidate-audit-20261007.json)
and [original plan](../../configs/headless-p0-20261007/experiment.json).
The abandoned arm's additional known estimated model cost is **$0.231970**;
the installation failure made no model calls and therefore has zero inference
cost. This extra cost is outside the comparison table above.
Expand All @@ -234,7 +234,7 @@ v0.9.0 baseline's passing pipeline trial: it downloaded the task repository,
read the solution and evaluation tests, and ran the copied tests. Its old raw
reward remains unchanged, but that pass is ineligible as independent-capability
evidence. The historical **18/20 is a preserved raw score**, not a clean causal
control for this experiment. See the [historical audit](../results/v0.9.0-pipeline-integrity-audit-20261007.json)
control for this experiment. See the [historical audit](../../results/v0.9.0-pipeline-integrity-audit-20261007.json)
and the annotation in the [original report](v0.9.0-baseline-20261005.md).
A targeted scan of the other nineteen historical trajectories found no matching
retrieval pattern; this is not an exhaustive retrospective audit.
Expand Down Expand Up @@ -269,10 +269,10 @@ No task name, filename, algorithm, answer, or verifier formula entered the promp

The current prompt has 385 words, SHA256
`37c4d9d44640c0051dc5a20e5954a067b98e8c51be07a48f080aa059e725c55d`.
Its [single-task plan](../configs/headless-workspace-20261007/experiment.json)
Its [single-task plan](../../configs/headless-workspace-20261007/experiment.json)
was registered before one fresh path-tracing run from an isolated checkout.
The run waited for a full-comparison arm to finish so concurrency stayed at most
two. See the [follow-up evidence](../results/headless-workspace-followup-20261007.json).
two. See the [follow-up evidence](../../results/headless-workspace-followup-20261007.json).

The clarified candidate **passed** the official verifier on its first attempt.
It disassembled the supplied executable (ATIF step 29 and subsequent reads),
Expand Down Expand Up @@ -315,7 +315,7 @@ then waits and kills the command session at its timeout. The guarded batching
scheduler also ran `timeout 1500 python3 run4.py ...` and was interrupted at the
actual 120-second tool limit.

The [scaled local diagnostic](../results/headless-tool-limit-probe-20261007.json)
The [scaled local diagnostic](../../results/headless-tool-limit-probe-20261007.json)
uses the unchanged Bash implementation with a 0.25-second tool limit and a
2-second workload. The problematic conditional-background form times out;
redirecting the whole group returns immediately and its completion marker is
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,9 +129,9 @@ the retry branch.

## Evidence locations

- [September 12 results](../results/tb21-generation-budget-65536-20260912.json)
- [September 14 results](../results/tb21-tool-input-recovery-20260914.json)
- [Journal v3 protocol](../../../docs/dev_docs/en/event-journal-protocol-v3.md)
- [September 12 results](../../results/tb21-generation-budget-65536-20260912.json)
- [September 14 results](../../results/tb21-tool-input-recovery-20260914.json)
- [Journal v3 protocol](../../../../docs/dev_docs/en/event-journal-protocol-v3.md)
- Local record: `jobs/tb21-stream-recovery-targeted3-20260915-record/`
- Local offline replay: `offline_replay.py` and `offline-replay.json` in that record

Expand Down
Loading
Loading