Skip to content

feat(benchmarks): enforce shared evaluation integrity policy - #57

Merged
minixalpha merged 3 commits into
mainfrom
feat/harbor-integrity-policy
Nov 4, 2026
Merged

minixalpha merged 3 commits into
mainfrom
feat/harbor-integrity-policy

Conversation

@minixalpha

@minixalpha minixalpha commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

New standard Harbor runs require a shared restricted-v1 policy for Nano, Pi, and DeepSeek Harness. Exact-host network allowlists are enforced outside the agent and checked before model work. All three adapters use process supervision, and collected output and network evidence remain on the host. Resource overrides are rejected, and non-root shared verifiers retain writable log directories.

Completed outputs are frozen before evidence-bound review. Original rewards and errors remain intact; confirmed violations count as zero without leaving the denominator. Pending reviews prevent publication of adjusted pass rates. Historical reproductions require an explicit --legacy-open-network opt-out.

The initial scope is single-step Linux Dockerfile/image tasks. Task images, official tests, and native budgets are preserved. Shared verifiers remain shared; the protocol does not establish complete protection against malicious submissions or reference exposure through allowed services.

Validation:

  • Harbor regression suite: 369 passed, 1 skipped (the opt-in Docker test).
  • Docker integration test run separately: 1 passed, including network restrictions, packet-mark bypass rejection, and UID 65534 writing verifier logs, rewards, and retry archives.
  • No model calls or scored benchmark reruns were performed.

The English and Simplified Chinese Harbor READMEs are synchronized, and the 0.10.x changelog records the protocol under Unreleased. No development notes, research notes, or benchmark reports changed.

@minixalpha
minixalpha merged commit 924d0ee into main Nov 4, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant