Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions HARNESS_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

A harness may have several independent interventions and PRs. A historical `AWAITING_EXTERNAL` batch state **remains historical** even when a separate publication ledger later confirms an upstream merge. All external submission/acceptance claims below come from the linked owning source; `SUBMITTED_UNVERIFIED_LIVE` means **no current PR-state claim**.

**Scope:** 153 harnesses · 168 interventions · 39 unique external PR URLs · 25 assessment batches · 32 monitored PR targets.
**Scope:** 153 harnesses · 169 interventions · 40 unique external PR URLs · 25 assessment batches · 33 monitored PR targets.

## Harness overview

Expand Down Expand Up @@ -81,7 +81,7 @@ A harness may have several independent interventions and PRs. A historical `AWAI
| [k4yt3x/meka](https://github.com/k4yt3x/meka) | 1 | 0 | 0 | 0 | 0 |
| [khanuzeb/axiom](https://github.com/khanuzeb/axiom) | 1 | 0 | 0 | 0 | 0 |
| [knowflow-ai/KnowWork](https://github.com/knowflow-ai/KnowWork) | 2 | 1 | 0 | 0 | 0 |
| [kstawiski/evidence-gated-scientific-agent](https://github.com/kstawiski/evidence-gated-scientific-agent) | 1 | 1 | 0 | 0 | 0 |
| [kstawiski/evidence-gated-scientific-agent](https://github.com/kstawiski/evidence-gated-scientific-agent) | 2 | 1 | 0 | 0 | 0 |
| [KumihoIO/Revka](https://github.com/KumihoIO/Revka) | 1 | 1 | 0 | 0 | 1 |
| [kushagrathisside/bridle](https://github.com/kushagrathisside/bridle) | 1 | 0 | 0 | 0 | 0 |
| [lbdremy/phax](https://github.com/lbdremy/phax) | 2 | 1 | 0 | 0 | 0 |
Expand Down Expand Up @@ -245,6 +245,7 @@ A harness may have several independent interventions and PRs. A historical `AWAI
| knowflow-ai/KnowWork | [assessment:knowwork-sandbox-multiprocess-ownership](data/assessment-candidate-batches/2026-11-10-25.json) | S2 | AWAITING_EXTERNAL | NOT_SUBMITTED | — | assessment-upstream-2026-11-10-25 | Refresh source; check external publication grant |
| knowflow-ai/KnowWork | [monitor:knowflow-ai/knowwork#4](data/pr-maintenance/targets.json) | — | NOT_TRACKED | WATCHLIST_ONLY | [#4](https://github.com/knowflow-ai/KnowWork/pull/4) | monitor_only | Read PR maintenance inbox; no technical inference |
| kstawiski/evidence-gated-scientific-agent | [assessment:evidence-bench-native-evidence-admission](data/assessment-candidate-batches/2026-10-23-11.json) | S1 | AWAITING_EXTERNAL | NOT_SUBMITTED | — | assessment-upstream-2026-10-23-11 | Refresh source; check external publication grant |
| kstawiski/evidence-gated-scientific-agent | [monitor:kstawiski/evidence-gated-scientific-agent#9](data/pr-maintenance/targets.json) | — | NOT_TRACKED | WATCHLIST_ONLY | [#9](https://github.com/kstawiski/evidence-gated-scientific-agent/pull/9) | monitor_only | Read PR maintenance inbox; no technical inference |
| KumihoIO/Revka | [assessment:revka-workflow-validation-contract](data/assessment-candidate-batches/2026-10-23-10.json) | S1 | AWAITING_EXTERNAL | BLOCKED_PUBLICATION | — | assessment-upstream-2026-10-23-10 | Resolve documented publication gate |
| kushagrathisside/bridle | [assessment:bridle-run-admission-crash-recovery](data/assessment-candidate-batches/2026-11-09-19.json) | S2 | CLOSED_NO_CHANGE | NONE | — | assessment-upstream-2026-11-09-19 | No code change justified |
| lbdremy/phax | [assessment:phax-cross-run-overlap-reconciliation](data/assessment-candidate-batches/2026-11-06-13.json) | S2 | AWAITING_EXTERNAL | NOT_SUBMITTED | — | assessment-upstream-2026-11-06-13 | Refresh source; check external publication grant |
Expand Down
23 changes: 23 additions & 0 deletions data/external-pr-publication-readbacks/2026-11-12.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"schema_version": 1,
"observed_at": "2026-11-12",
"source": "Authenticated dev-alpha-2 GitHub REST POST and exact external/fork-local GET readback; effective standing publication mandate",
"authenticated_actor": "xLagerFeuer",
"metric_boundary": "Publication evidence only; no maintainer merge, verified adoption or VSM regrade",
"cases": [
{
"id": "evidence-bench-native-evidence-admission",
"repository": "kstawiski/evidence-gated-scientific-agent",
"public_fork": "opensiro-contributions/evidence-gated-scientific-agent",
"fork_local_pr_url": "https://github.com/opensiro-contributions/evidence-gated-scientific-agent/pull/1",
"pr_url": "https://github.com/kstawiski/evidence-gated-scientific-agent/pull/9",
"published_at": "2026-11-12T00:11:59Z",
"base_sha": "85b396a824dfad6ec0880cbe07d86d3e24920ae8",
"head_sha": "9710d908adb05bc5effc6a557de321bff8cda142",
"pr_commits": 1,
"publication_state": "OPEN_READY",
"approved_exact_title_body": true,
"approval_boundary": "ACTIVE standing external PR mandate; authenticated xLagerFeuer dev-alpha-2 REST POST and exact title/body/head/base/two-file external readback; 16/16 Python policy tests PASS and full Python non-live test command exited 0; npm ci not run (Node/npm unavailable), disclosed in external PR. No legal-person CLA/DCO or maintainer acceptance claim."
}
]
}
5 changes: 5 additions & 0 deletions data/pr-maintenance/targets.json
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,11 @@
"number": 4,
"url": "https://github.com/knowflow-ai/KnowWork/pull/4"
},
{
"repository": "kstawiski/evidence-gated-scientific-agent",
"number": 9,
"url": "https://github.com/kstawiski/evidence-gated-scientific-agent/pull/9"
},
{
"repository": "lbdremy/phax",
"number": 205,
Expand Down
15 changes: 14 additions & 1 deletion docs/PR_MAINTENANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ This is a construction/operating contract. The read-only detector and alert tran

The watcher uses Python standard library only and authenticates using the GitHub Actions token. Each observation is a live **GET** from the GitHub repository that owns it; unexpected API errors, missing identities, unknown states and truncated pagination **fail closed**. Third-party message **bodies are deliberately not stored, executed, or trusted as instructions**: only event IDs, author, status and GitHub URLs are used. Humans/agents inspect the exact review on the external source before repairing.

Current pinned targets are **32** reviewed external PR identities for contributor `xLagerFeuer` (18 previously retained, seven 2026-11-10 publication readbacks, [KnowWork #4](https://github.com/knowflow-ai/KnowWork/pull/4), [Runloop #1](https://github.com/murilo-bassan/durable-agent-runtime/pull/1), [CodeAct #26](https://github.com/xingyaoww/code-act/pull/26), [Clouds-Coder #50](https://github.com/FonaTech/Clouds-Coder/pull/50), [yoagent #450](https://github.com/yologdev/yoagent/pull/450), and [phax #205](https://github.com/lbdremy/phax/pull/205) and [Fara #96](https://github.com/microsoft/fara/pull/96) on 2026-11-11), including Headcount #47 (previously verified but returning GitHub 404 on 2026-11-05), closed-unmerged StarNet #56, and six earlier Corrective PRs: [Workflow One #173](https://github.com/chumingjun/dsh-harness-one/pull/173) (merged 2026-10-29), [Reef/Polyp #19](https://github.com/Coral-Bricks-AI/reef/pull/19), [Spartan #1](https://github.com/CorticalComputer/Spartan/pull/1), [BESSER #205](https://github.com/BESSER-PEARL/BESSER-Agentic-Framework/pull/205), [BabyAI #128](https://github.com/mila-iqia/babyai/pull/128), and [Phoenix #1](https://github.com/Phoenixrr2113/agent-harness/pull/1). This list is a versioned scope snapshot and is **not** dynamically widened by an unreviewed search result. Adding/removing targets requires a reviewable manifest change and explicit checkpoint-migration handling. Closed/merged PRs remain in the snapshot to preserve lifecycle evidence until a deliberate archival change.
Current pinned targets are **33** reviewed external PR identities for contributor `xLagerFeuer` (18 previously retained, seven 2026-11-10 publication readbacks, [KnowWork #4](https://github.com/knowflow-ai/KnowWork/pull/4), [Runloop #1](https://github.com/murilo-bassan/durable-agent-runtime/pull/1), [CodeAct #26](https://github.com/xingyaoww/code-act/pull/26), [Clouds-Coder #50](https://github.com/FonaTech/Clouds-Coder/pull/50), [yoagent #450](https://github.com/yologdev/yoagent/pull/450), and [phax #205](https://github.com/lbdremy/phax/pull/205) and [Fara #96](https://github.com/microsoft/fara/pull/96) on 2026-11-11), including Headcount #47 (previously verified but returning GitHub 404 on 2026-11-05), closed-unmerged StarNet #56, and six earlier Corrective PRs: [Workflow One #173](https://github.com/chumingjun/dsh-harness-one/pull/173) (merged 2026-10-29), [Reef/Polyp #19](https://github.com/Coral-Bricks-AI/reef/pull/19), [Spartan #1](https://github.com/CorticalComputer/Spartan/pull/1), [BESSER #205](https://github.com/BESSER-PEARL/BESSER-Agentic-Framework/pull/205), [BabyAI #128](https://github.com/mila-iqia/babyai/pull/128), and [Phoenix #1](https://github.com/Phoenixrr2113/agent-harness/pull/1). This list is a versioned scope snapshot and is **not** dynamically widened by an unreviewed search result. Adding/removing targets requires a reviewable manifest change and explicit checkpoint-migration handling. Closed/merged PRs remain in the snapshot to preserve lifecycle evidence until a deliberate archival change.

**Reviewed availability-only exception (Upstream #139 / PR #140):** Headcount #47 is retained as the eighteenth identity, with the sole allowlisted `monitoring: verify_404_only` manifest mode. Every run MUST GET that exact source and confirm HTTP 404. It may then report **17 fully observed / 1 explicitly UNVERIFIABLE of 18**, continue reading the other 17, and store a distinct *availability marker* instead of stale PR fields. The first transition emits one inbox `SOURCE_UNVERIFIABLE` alert, never a fabricated close/merge/review. Recovery (HTTP 200), a different status, any additional unavailable URL, or a malformed source **fails closed** pending source review and removal of the exception. The state is never silently `OPEN` or `CLOSED_UNMERGED`. This is a reviewed change to monitoring *coverage*, not a removal of the historical cohort, or permission to claim an 18-of-18 event snapshot. A returned count of 17 is prominently labelled in the observer and inbox output.

Expand Down Expand Up @@ -83,3 +83,16 @@ were not executed. This exact allowlisted addition permits a 31 to 32
maintenance-target migration without replaying historical comments or
inventing maintainer activity. Publication is not third-party acceptance,
measured downstream benefit or a VSM regrade.

**Checkpoint migration for Evidence Bench #9 (2026-11-12 UTC, standing publication mandate):**
External [Evidence Bench #9](https://github.com/kstawiski/evidence-gated-scientific-agent/pull/9)
was published Open/Ready from [fork-local #1](https://github.com/opensiro-contributions/evidence-gated-scientific-agent/pull/1),
on original main at 85b396a824dfad6ec0880cbe07d86d3e24920ae8, source-native
branch head 9710d908adb05bc5effc6a557de321bff8cda142; two changed files.
Native Python policy regression 16/16 PASS, full Python non-live suite command
exited 0 after uv sync frozen/dev/analysis; no test total separately recorded.
The contributor's documented npm ci step was not performed, with Node/npm
absent, disclosed in external body. This single allowlisted addition enables
32 to 33 maintenance targets, preserving earlier checkpoint state and recording
only NEW_TRACKED_PR informational transitions. No maintainer merge, real-world
benefit, or VSM-grade change is inferred.
2 changes: 2 additions & 0 deletions scripts/pr_maintenance_watch.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@
"https://github.com/FonaTech/Clouds-Coder/pull/50",
# yoagent standing-mandate publication, native all-features validated on 2026-11-11.
"https://github.com/yologdev/yoagent/pull/450",
# Evidence Bench policy 16/16 and full non-live exit 0; external #9 readback 2026-11-12.
"https://github.com/kstawiski/evidence-gated-scientific-agent/pull/9",
# Fara focused 3/3 native tests and exact external #96 readback 2026-11-11.
"https://github.com/microsoft/fara/pull/96",
# phax source-native full-gate PASS and standing publication 2026-11-11.
Expand Down
6 changes: 3 additions & 3 deletions tests/test_portfolio_reconcile.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,9 @@ def pages(self, path):
class PortfolioReconcileTests(unittest.TestCase):
def test_frozen_source_union_exact_size_and_identity(self):
entries = portfolio.cohort()
self.assertEqual(len(entries), 39)
self.assertEqual(sum("watchlist" in row["groups"] for row in entries), 32)
self.assertEqual(len({row["url"] for row in entries}), 39)
self.assertEqual(len(entries), 40)
self.assertEqual(sum("watchlist" in row["groups"] for row in entries), 33)
self.assertEqual(len({row["url"] for row in entries}), 40)
self.assertEqual(len([row for row in entries if "casebook" in row["groups"]]), 9)
self.assertEqual(len([row for row in entries if "registry" in row["groups"]]), 1)
self.assertEqual(
Expand Down
10 changes: 7 additions & 3 deletions tests/test_pr_maintenance_watch.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,10 +145,10 @@ def write_targets(self, root):
"targets": [{"repository": "Example/project", "number": 3, "url": URL}]}))
return p

def test_exact_manifest_has_thirty_two_targets_and_no_duplicates(self):
def test_exact_manifest_has_thirty_three_targets_and_no_duplicates(self):
data = watch.validate_targets()
self.assertEqual(len(data["targets"]), 32)
self.assertEqual(len({t["url"] for t in data["targets"]}), 32)
self.assertEqual(len(data["targets"]), 33)
self.assertEqual(len({t["url"] for t in data["targets"]}), 33)
self.assertIn("https://github.com/knowflow-ai/KnowWork/pull/4",
{t["url"] for t in data["targets"]})
self.assertIn("https://github.com/knowflow-ai/KnowWork/pull/4",
Expand All @@ -165,6 +165,10 @@ def test_exact_manifest_has_thirty_two_targets_and_no_duplicates(self):
self.assertIn("https://github.com/yologdev/yoagent/pull/450", watch.APPROVED_CHECKPOINT_ADDITIONS)
self.assertIn("https://github.com/FonaTech/Clouds-Coder/pull/50", {t["url"] for t in data["targets"]})
self.assertIn("https://github.com/FonaTech/Clouds-Coder/pull/50", watch.APPROVED_CHECKPOINT_ADDITIONS)
self.assertIn("https://github.com/kstawiski/evidence-gated-scientific-agent/pull/9",
{t["url"] for t in data["targets"]})
self.assertIn("https://github.com/kstawiski/evidence-gated-scientific-agent/pull/9",
watch.APPROVED_CHECKPOINT_ADDITIONS)
self.assertIn("https://github.com/microsoft/fara/pull/96",
{t["url"] for t in data["targets"]})
self.assertIn("https://github.com/microsoft/fara/pull/96",
Expand Down
2 changes: 1 addition & 1 deletion tests/test_resolved_pr_acceptance.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ def test_multiple_bot_report_comments_fail_before_write(self):
self.assertFalse(api.writes)

def test_only_reviewed_watchlist_and_no_unreviewed_expansion(self):
self.assertEqual(len(metric.validate_targets()["targets"]), 32)
self.assertEqual(len(metric.validate_targets()["targets"]), 33)
self.assertEqual(metric.INBOX_REPO, "opensiro/upstream")
self.assertEqual(metric.INBOX_NUMBER, 125)

Expand Down
4 changes: 2 additions & 2 deletions tests/test_verified_external_outcomes.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,12 +87,12 @@ class VerifiedExternalOutcomeTests(unittest.TestCase):
def test_owner_ledgers_cover_exact_portfolio_with_one_replacement_chain(self):
items, blocked = outcomes.cohort()
self.assertEqual((len(items), sum(len(x["pr_urls"]) for x in items), blocked),
(38, 39, 21))
(39, 40, 21))
self.assertEqual(len([x for x in items if x["kind"] == "improvement"]), 2)
self.assertEqual(len([x for x in items if x["kind"] == "corrective"]), 13)
self.assertEqual(len([x for x in items if x["kind"] == "casebook"]), 8)
self.assertEqual(len([x for x in items if x["kind"] == "registry"]), 1)
self.assertEqual(len([x for x in items if x["kind"] == "publication"]), 14)
self.assertEqual(len([x for x in items if x["kind"] == "publication"]), 15)
self.assertIn("publication:knowwork-sandbox-start-race",
{x["id"] for x in items})
self.assertIn("publication:runloop-budget-finalization-idempotency",
Expand Down
Loading