Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,30 @@ the loop honest — is in [docs/self-improvement.md](docs/self-improvement.md).
Nothing in it ships yet; it is written down so it can be built against the log
that now exists.

## Completion auditor (static-only)

An additive, strictly manual completion auditor. It captures a verifier-owned
delta snapshot, binds a task whose acceptance criteria come from **trusted,
human configuration** (never the model), records byte-level artifact
observations, and renders a deterministic replayable report. It verifies a
bounded, byte-level claim inside an explicit scope — it is not a semantic
correctness prover, not a test-suite completeness certifier, and not a
completion interceptor. No automatic repair.

Three tools: `teamwork_audit_plan`, `teamwork_audit_check`,
`teamwork_audit_report`, addressed by an opaque `auditId` only. Execution is
blocked: a model-provided approval is rejected and, without provable human
approval authority, execution reports `BLOCKED` and runs nothing. Snapshot
capture is a delta scope (not the whole repo); the same-user store is a declared
trust assumption with best-effort (no fsync), unsigned integrity, not a sandbox.

Requires an absolute, verifier-owned `auditBaseRoot` outside the candidate. See
[docs/audit/README.md](docs/audit/README.md) for the full, validated
configuration and the honest limits. `bun run audit:eval` runs a fixed
30-fixture **regression** check (writer-authored expectations, metadata-only
development/held-out split) — a source-repo development command, not a shipped
scientific evaluation or a package feature.

## Development

```bash
Expand Down
21 changes: 21 additions & 0 deletions SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,27 @@ Custom per-role with the `custom` picker at install time, or edit
your `opencode.json` by hand later. There is no technical reason all
10 agents have to be the same model.

## Completion auditor (static-only)

An additive, manual completion auditor ships with the plugin. It captures a
verifier-owned delta snapshot of the candidate, binds a task whose acceptance
criteria come from trusted human configuration (never the model), records
byte-level artifact observations, and renders a deterministic, replayable report.

Three tools — `teamwork_audit_plan`, `teamwork_audit_check`,
`teamwork_audit_report` — addressed by an opaque `auditId` only. Static-only by
default: execution is blocked (a model-provided approval is rejected; absent
provable human approval authority it reports BLOCKED and runs nothing). Capture is
a delta scope, not the whole repo; the same-user store is a declared trust
assumption with best-effort, unsigned integrity, not a sandbox. No automated
repair or completion interception.

Requires an absolute, verifier-owned `auditBaseRoot` outside the candidate; with
none configured the tools fail closed with `SETUP_REQUIRED`. Full configuration
and limits: `docs/audit/README.md` (in the source repo). Evaluation: a fixed
30-fixture **regression** check (`bun run audit:eval`) that runs only from the
source repo and is not a shipped scientific evaluation.

## Install

```bash
Expand Down
12 changes: 12 additions & 0 deletions audit-build/task4-fixtures/fixture/check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import unittest


class CheckFixture(unittest.TestCase):
def test_one(self):
self.assertTrue(True)

def test_two(self):
self.assertEqual(1 + 1, 2)

def test_three(self):
self.assertIn("a", "abc")
41 changes: 41 additions & 0 deletions audit-build/task4-fixtures/fixture/escape.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
import json
import os
import socket

result = {}

try:
with open("/input/escape-write.txt", "w") as handle:
handle.write("x")
result["input_write"] = "ALLOWED"
except Exception as error: # noqa: BLE001
result["input_write"] = "denied:" + type(error).__name__

try:
with open("/store/evil.json", "w") as handle:
handle.write("x")
result["store_write"] = "ALLOWED"
except Exception as error: # noqa: BLE001
result["store_write"] = "denied:" + type(error).__name__

try:
with open("/evil.txt", "w") as handle:
handle.write("x")
result["root_write"] = "ALLOWED"
except Exception as error: # noqa: BLE001
result["root_write"] = "denied:" + type(error).__name__

try:
sock = socket.socket()
sock.settimeout(2)
sock.connect(("1.1.1.1", 80))
result["network"] = "CONNECTED"
except Exception as error: # noqa: BLE001
result["network"] = "denied:" + type(error).__name__

try:
result["store_listing"] = os.listdir("/store")
except Exception as error: # noqa: BLE001
result["store_listing"] = "absent:" + type(error).__name__

print(json.dumps(result))
12 changes: 12 additions & 0 deletions audit-build/task4-fixtures/fixture/fail.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import unittest


class FailFixture(unittest.TestCase):
def test_a(self):
self.assertTrue(True)

def test_b(self):
self.assertTrue(True)

def test_c(self):
self.assertEqual(1, 2)
8 changes: 8 additions & 0 deletions audit-build/task4-fixtures/fixture/flood.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import os

chunk = b"x" * 4096
for _ in range(8192):
try:
os.write(1, chunk)
except Exception: # noqa: BLE001
break
14 changes: 14 additions & 0 deletions audit-build/task4-fixtures/fixture/harness_flood.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import sys
import unittest

# Adversarial fixture (round-2). Flood Python-level stdout during import. The
# harness redirects this into a bounded, discarding sink, so it must not grow
# harness memory without bound; the harness reports an overflow observation
# instead of retaining the flood.
for _ in range(20000):
sys.stdout.write("x" * 500)


class FloodImportFixture(unittest.TestCase):
def test_ok(self):
self.assertTrue(True)
18 changes: 18 additions & 0 deletions audit-build/task4-fixtures/fixture/harness_mp_json.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import json
import unittest


# Adversarial fixture (round-2). Replace json.dumps so that whatever the
# same-process verifier harness tries to report is rewritten to a forged green
# count document. The harness's own emit path uses json.dumps, so a
# monkeypatching candidate controls the reported counts entirely.
def _forged(*args, **kwargs):
return '{"collected": 5, "passed": 5, "failed": 0, "skipped": 0}'


json.dumps = _forged


class MonkeypatchJsonFixture(unittest.TestCase):
def test_ok(self):
self.assertTrue(True)
43 changes: 43 additions & 0 deletions audit-build/task4-fixtures/fixture/harness_mp_result.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
import unittest


# Adversarial fixture (round-2). Replace unittest.TestResult with a result that
# always claims a full green run, regardless of which tests actually execute.
# The same-process harness reads testsRun/failures/errors/skipped from this
# object, so the candidate controls the reported counts.
class _ForgedResult:
def __init__(self):
self.testsRun = 9
self.failures = []
self.errors = []
self.skipped = []
self.shouldStop = False

def startTest(self, test):
return None

def stopTest(self, test):
return None

def addSuccess(self, test):
return None

def addError(self, test, err):
return None

def addFailure(self, test, err):
return None

def addSkip(self, test, reason):
return None

def wasSuccessful(self):
return True


unittest.TestResult = _ForgedResult


class MonkeypatchResultFixture(unittest.TestCase):
def test_ok(self):
self.assertTrue(True)
14 changes: 14 additions & 0 deletions audit-build/task4-fixtures/fixture/harness_write_exit.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import json
import os

# Adversarial fixture (round-2). At import time -- i.e. inside the verifier
# harness's exec_module -- write a fully valid, green count document straight to
# file descriptor 1 and terminate the shared process before the harness can emit
# anything of its own. ZERO tests run. Because the candidate and the harness
# share one Python process, the last JSON line is forged. The runner must treat
# this as a nondecisive observation and must never report PASS.
os.write(
1,
(json.dumps({"collected": 7, "passed": 7, "failed": 0, "skipped": 0}) + "\n").encode(),
)
os._exit(0)
11 changes: 11 additions & 0 deletions audit-build/task4-fixtures/fixture/skip.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import unittest


class SkipFixture(unittest.TestCase):
@unittest.skip("skipped on purpose")
def test_a(self):
self.assertTrue(True)

@unittest.skip("skipped on purpose")
def test_b(self):
self.assertTrue(True)
4 changes: 4 additions & 0 deletions audit-build/task4-fixtures/fixture/sleep.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
import time

print("sleeping", flush=True)
time.sleep(3600)
5 changes: 5 additions & 0 deletions audit-build/task4-fixtures/fixture/spoof.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import json

# A candidate that merely prints a green count document and exits 0. It runs no
# tests, so a candidate-sourced discovery adapter must never treat this as proof.
print(json.dumps({"collected": 10, "passed": 10, "failed": 0, "skipped": 0}))
5 changes: 5 additions & 0 deletions audit-build/task4-fixtures/fixture/zero.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import unittest


class NoTests(unittest.TestCase):
pass
Loading
Loading