From 6952350edfb480f7a6a3ef5e7ccb8ea41dd425c0 Mon Sep 17 00:00:00 2001 From: Alok Subbarao Date: Sun, 18 Oct 2026 17:30:29 -0700 Subject: [PATCH 1/4] verification is one query whose rows are the answer; no parquet round trip The template runs the proposal and the DuckDB version on the same inputs and returns one row per input with both answers and a differs flag. Logging is the agent-log COPY wrapped around that same query, never read back to get the answer. Removes the read_parquet readback entry from useful_queries. Verified on dev. Co-Authored-By: Claude Opus 5.5 --- skills/agent-dispatch/SKILL.md | 64 +++++++++------------- skills/query/references/useful_queries.sql | 19 ------- 2 files changed, 25 insertions(+), 58 deletions(-) diff --git a/skills/agent-dispatch/SKILL.md b/skills/agent-dispatch/SKILL.md index fd9331d1..3dd2dbb8 100644 --- a/skills/agent-dispatch/SKILL.md +++ b/skills/agent-dispatch/SKILL.md @@ -152,48 +152,34 @@ ran it, and the lead passed it on. When run, it mistyped week dates and `+HH:MM` crashed on year-1 offsets. The same claim had already been disproved on 10-15: the Python version needed a regex and grew to 74 lines. -**The verification is one self-dispatched agent-log COPY that runs both sides.** The inputs and -the signature are declared once, in the first CTE. The dispatched statement is the agent-log COPY, -so both sides run as typed columns inside it and land as signed rows. The receipt says only -whether the write happened, and the readback reads the rows as columns (compendium: "each -statement writes them and the next stage reads the files"). +**The verification is one query, and its rows are the answer.** It runs the proposal and the DuckDB +version on the same inputs and returns one row per input with both answers side by side: ```sql -WITH verification AS ( -- 1. plain values, declared once - SELECT [] AS raws, to_json(raws)::VARCHAR AS raws_json, - '' AS agent, '' AS session_id, '' AS skill, - $n$$n$ AS notes, '' AS log_dir -), sides AS ( -- 2. the program under test, rendered once as a column - SELECT *, tera_render($tera$ UNION ALL BY NAME $tera$, v::JSON, autoescape := false) AS both_sides - FROM verification AS v -), rendered AS ( -- 3. the agent-log COPY: both_sides stored as text and run as a subquery - SELECT tera_render($tera$COPY (SELECT uuidv7() AS row_id, 'verification' AS type, $v${{ agent }}$v$ AS agent, - $v${{ session_id }}$v$ AS session_id, $v${{ agent }}-{{ session_id }}$v$ AS agent_signature, - $v${{ skill }}$v$ AS skill, $query${{ both_sides }}$query$ AS query_was_ran, r.*, - $n${{ notes }}$n$ AS markdown_notes, now() AS ts - FROM ({{ both_sides }}) AS r) - TO $v${{ log_dir }}$v$ (FORMAT parquet, PARTITION_BY (type, agent_signature), OVERWRITE_OR_IGNORE true, FILENAME_PATTERN '{uuid}')$tera$, - s::JSON, autoescape := false) AS sql - FROM sides AS s -), dispatched AS ( -- 4. post it - SELECT array_agg(selfdispatch(sql)) AS receipts FROM rendered +WITH python_side AS ( -- the inputs live here once; the proposal runs on them + SELECT * FROM read_json($cmd$/opt/homebrew/bin/uv run --quiet --no-project python - <<'PY' +import json +for raw in []: + try: + print(json.dumps(dict(raw=raw, =(raw), error=None))) + except Exception as e: + print(json.dumps(dict(raw=raw, =None, error=type(e).__name__))) +PY +|$cmd$, format := 'newline_delimited') +), duckdb_side AS ( -- the same inputs, taken from the Python rows + SELECT raw, AS FROM python_side ) -SELECT receipt.* FROM dispatched CROSS JOIN UNNEST(receipts) AS t(receipt); +SELECT raw, duckdb_row. AS duckdb_answer, python_row. AS python_answer, + python_row.error AS python_error, + duckdb_row. IS DISTINCT FROM python_row. AS differs +FROM duckdb_side AS duckdb_row JOIN python_side AS python_row USING (raw) +ORDER BY differs DESC, raw ``` -`` is ` UNION ALL BY NAME ` over the same declared list: -- **DuckDB side:** `SELECT 'duckdb' AS side, raw, AS - FROM (SELECT unnest($raws${{ raws_json }}$raws$::JSON::VARCHAR[]) AS raw)`. -- **Python side:** `SELECT 'python' AS side, * FROM read_json($cmd$/opt/homebrew/bin/uv run --quiet - --no-project python - <<'PY' … for raw in {{ raws_json }}: … PY |$cmd$, format := 'newline_delimited')`. - The script prints one JSON line per input, and for an input that raises it prints the exception - type in `error`, so every input is a row on both sides. The reader infers the columns. - -The program is rendered once, so the stored `query_was_ran` is exactly what ran, and the signature -is built from the declared agent and session id. Values render inside dollar quotes (`$v$`, -`$raws$`, `$n$`), so inputs and notes may contain a single quote. A worker testing ponytail's or -Thermos's proposal writes it as the Python (or bash) side. Worked example (verified on dev -2026-10-18, an input and the notes containing `'`): `skill = 'T4-check-dispatched-v3'` under -`~/.duck/agent_log/skill_trial_2026_10_17/type=verification`. -Readback: `query/references/useful_queries.sql`, "Verification: both sides". +An input the proposal raises on still prints its row, with the exception type in `error`. To keep +the run as a signed record, wrap this same query in the agent-log COPY (`$query$…$query$` stored +and executed); nothing reads that record back to get the answer, the query already returned it. +A worker testing ponytail's or Thermos's proposal writes it as the Python (or bash) side. Verified +on dev 2026-10-18 (T4 proposal: the `+05:00` midnight, the year-1 `OverflowError` and an input +containing `'` all show as `differs`). ## 4. Evidence is false-first diff --git a/skills/query/references/useful_queries.sql b/skills/query/references/useful_queries.sql index 929a81fa..56cefa11 100644 --- a/skills/query/references/useful_queries.sql +++ b/skills/query/references/useful_queries.sql @@ -241,22 +241,3 @@ WITH c AS ( SELECT array_agg(selfdispatch(sql)) AS receipts FROM rendered ) SELECT receipt.* FROM dispatched CROSS JOIN UNNEST(receipts) AS t(receipt); - --- ## Verification: both sides of a DuckDB-vs-Python claim, joined on the input (Opus 5.5, 2026-10-17) --- when: reading back an agent-log `type=verification` row whose query was UNION ALL BY NAME (agent-dispatch section 3). --- uses: read_parquet (core). --- returns: one row per input: raw, duckdb_row and python_row (each side's whole logged row as a STRUCT, query_was_ran and markdown_notes included), missing_duckdb, missing_python, differs BOOLEAN; a missing_* row is an input only one side produced. is the output both sides print (value_type in the example). --- tested: T4 fromisoformat proposal vs the INF-1535 classifier, 19 inputs, 3 disagreements (all Python errors). -WITH logged AS ( - SELECT * FROM read_parquet('/type=verification/**/*.parquet', hive_partitioning := true, union_by_name := true) - WHERE skill = '' -), duckdb_side AS ( - SELECT * FROM logged WHERE side = 'duckdb' -), python_side AS ( - SELECT * FROM logged WHERE side = 'python' -) -SELECT raw, duckdb_row, python_row, - duckdb_row.side IS NULL AS missing_duckdb, python_row.side IS NULL AS missing_python, - duckdb_row. IS DISTINCT FROM python_row. AS differs -FROM duckdb_side AS duckdb_row FULL JOIN python_side AS python_row USING (raw) -ORDER BY differs DESC, raw; From 56558c106c7f8fcc7a6fb89891eec04c3feeb98c Mon Sep 17 00:00:00 2001 From: Alok Subbarao Date: Sun, 18 Oct 2026 17:39:49 -0700 Subject: [PATCH 2/4] verification template: compute the DuckDB answer on the Python rows, no self-join Review on #69: duckdb_side was built from python_side and joined back to it; the answer is now one more column on the same row, so a repeated input cannot multiply rows. Same result on dev. Co-Authored-By: Claude Opus 5.5 --- skills/agent-dispatch/SKILL.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/skills/agent-dispatch/SKILL.md b/skills/agent-dispatch/SKILL.md index 3dd2dbb8..bacc4991 100644 --- a/skills/agent-dispatch/SKILL.md +++ b/skills/agent-dispatch/SKILL.md @@ -165,16 +165,16 @@ for raw in []: print(json.dumps(dict(raw=raw, =None, error=type(e).__name__))) PY |$cmd$, format := 'newline_delimited') -), duckdb_side AS ( -- the same inputs, taken from the Python rows - SELECT raw, AS FROM python_side ) -SELECT raw, duckdb_row. AS duckdb_answer, python_row. AS python_answer, - python_row.error AS python_error, - duckdb_row. IS DISTINCT FROM python_row. AS differs -FROM duckdb_side AS duckdb_row JOIN python_side AS python_row USING (raw) +SELECT raw, AS duckdb_answer, AS python_answer, + error AS python_error, duckdb_answer IS DISTINCT FROM python_answer AS differs +FROM python_side ORDER BY differs DESC, raw ``` -An input the proposal raises on still prints its row, with the exception type in `error`. To keep +The DuckDB answer is computed on the same row as the Python answer, so there is no second side to +join. An input the proposal raises on still prints its row, with the exception type in `error`. +The inputs stay in the script: a self-dispatched Python side returns receipts (`status`, `body` +as text), not the script's columns. To keep the run as a signed record, wrap this same query in the agent-log COPY (`$query$…$query$` stored and executed); nothing reads that record back to get the answer, the query already returned it. A worker testing ponytail's or Thermos's proposal writes it as the Python (or bash) side. Verified From 6e4d7cdf72489dd06e9365f5a49b6ed00e1859ab Mon Sep 17 00:00:00 2001 From: Alok Subbarao Date: Mon, 19 Oct 2026 09:44:46 -0700 Subject: [PATCH 3/4] verification: per-input self-dispatch beside the DuckDB answer, no loop or try The inputs are rows in the first CTE; each renders a one-input Python statement that is self-dispatched beside its duckdb_answer, and the receipts unnest back into columns. An input the proposal raises on is its own failed receipt and counts as differs (the old loop's NULL-vs-NULL compare marked raising inputs as agreeing). The example runs on dev as written. Record: COPY the whole query, as in PR #60; nothing reads it back. Co-Authored-By: Claude Opus 5.5 --- skills/agent-dispatch/SKILL.md | 48 ++++++++++++++++++---------------- 1 file changed, 25 insertions(+), 23 deletions(-) diff --git a/skills/agent-dispatch/SKILL.md b/skills/agent-dispatch/SKILL.md index bacc4991..3ffa20bb 100644 --- a/skills/agent-dispatch/SKILL.md +++ b/skills/agent-dispatch/SKILL.md @@ -152,34 +152,36 @@ ran it, and the lead passed it on. When run, it mistyped week dates and `+HH:MM` crashed on year-1 offsets. The same claim had already been disproved on 10-15: the Python version needed a regex and grew to 74 lines. -**The verification is one query, and its rows are the answer.** It runs the proposal and the DuckDB -version on the same inputs and returns one row per input with both answers side by side: +**The verification is one self-dispatched query, and its rows are the answer.** The inputs are rows +in the first CTE. Each one gets its DuckDB answer and renders a one-input Python statement, which is +self-dispatched beside it; the receipts come back as columns, one row per input: ```sql -WITH python_side AS ( -- the inputs live here once; the proposal runs on them - SELECT * FROM read_json($cmd$/opt/homebrew/bin/uv run --quiet --no-project python - <<'PY' -import json -for raw in []: - try: - print(json.dumps(dict(raw=raw, =(raw), error=None))) - except Exception as e: - print(json.dumps(dict(raw=raw, =None, error=type(e).__name__))) +WITH intent AS ( -- the inputs, declared once as rows + SELECT unnest(['2026-10-18', '2026-02-30', $v$O'Brien$v$]) AS raw +), rendered AS ( -- per input: the DuckDB answer, and the proposal as a one-input statement + SELECT raw, try_strptime(raw, '%Y-%m-%d')::DATE::VARCHAR AS duckdb_answer, + concat($py$SELECT * FROM read_json($cmd$/opt/homebrew/bin/uv run --quiet --no-project python - <<'PY' +import json, datetime +print(json.dumps(dict(python_answer=datetime.date.fromisoformat($py$, to_json(raw), $py$).isoformat()))) PY -|$cmd$, format := 'newline_delimited') +|$cmd$, format := 'newline_delimited')$py$) AS sql + FROM intent +), dispatched AS ( -- each input keeps its receipt + SELECT raw, duckdb_answer, array_agg(selfdispatch(sql)) AS receipts FROM rendered GROUP BY ALL ) -SELECT raw, AS duckdb_answer, AS python_answer, - error AS python_error, duckdb_answer IS DISTINCT FROM python_answer AS differs -FROM python_side +SELECT raw, duckdb_answer, receipt.body::JSON ->> '$[*].python_answer' AS python_answers, + [duckdb_answer] IS DISTINCT FROM python_answers AS differs, receipt.status, receipt.body +FROM dispatched CROSS JOIN UNNEST(receipts) AS t(receipt) ORDER BY differs DESC, raw ``` -The DuckDB answer is computed on the same row as the Python answer, so there is no second side to -join. An input the proposal raises on still prints its row, with the exception type in `error`. -The inputs stay in the script: a self-dispatched Python side returns receipts (`status`, `body` -as text), not the script's columns. To keep -the run as a signed record, wrap this same query in the agent-log COPY (`$query$…$query$` stored -and executed); nothing reads that record back to get the answer, the query already returned it. -A worker testing ponytail's or Thermos's proposal writes it as the Python (or bash) side. Verified -on dev 2026-10-18 (T4 proposal: the `+05:00` midnight, the year-1 `OverflowError` and an input -containing `'` all show as `differs`). +Swap in the real inputs, the DuckDB expression and the proposal (`fromisoformat` here). There is no +loop and no `try`: an input the proposal raises on comes back as its own failed receipt (`status` +500, the error in `body`, `python_answers` empty) and counts as `differs`. Verified on dev +2026-10-19: three rows; `2026-10-18` agrees, `2026-02-30` and `O'Brien` differ (Python raised). +To keep a record, wrap the whole query in +`COPY (…) TO '' (FORMAT parquet, PARTITION_BY (type, agent_signature), OVERWRITE_OR_IGNORE true, FILENAME_PATTERN '{uuid}')`; +nothing reads it back to get the answer. A worker testing ponytail's or Thermos's proposal writes +it as the Python (or bash) statement. ## 4. Evidence is false-first From 0f4eb2066ae79a5fb90c1b97e66e17fa811a12e5 Mon Sep 17 00:00:00 2001 From: Alok Subbarao Date: Mon, 19 Oct 2026 10:11:02 -0700 Subject: [PATCH 4/4] verification: result_rows as in PR #60, one ->> per field, no COPY record The body rows unnest once (coalesce(try_cast(body AS JSON[]), [NULL::JSON])), so a failed dispatch stays as one row, and python_answer is read with a single ->> instead of a $[*] path. The COPY-to-agent-log sentence is gone: the query's rows are the answer. Same three rows as the previous version (table_diff_summary pk raw: 3 identical). Co-Authored-By: Claude Opus 5.5 --- skills/agent-dispatch/SKILL.md | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/skills/agent-dispatch/SKILL.md b/skills/agent-dispatch/SKILL.md index 3ffa20bb..28a62981 100644 --- a/skills/agent-dispatch/SKILL.md +++ b/skills/agent-dispatch/SKILL.md @@ -152,9 +152,12 @@ ran it, and the lead passed it on. When run, it mistyped week dates and `+HH:MM` crashed on year-1 offsets. The same claim had already been disproved on 10-15: the Python version needed a regex and grew to 74 lines. -**The verification is one self-dispatched query, and its rows are the answer.** The inputs are rows -in the first CTE. Each one gets its DuckDB answer and renders a one-input Python statement, which is -self-dispatched beside it; the receipts come back as columns, one row per input: +**The verification is one self-dispatched query that runs both sides.** The inputs are declared +once, as rows in the first CTE. Each input gets its DuckDB answer and renders the proposal as a +one-input Python statement, dispatched beside it, so both sides come back in the receipts, nothing +written to read back. Each receipt stays whole (`receipt.*`, body included), each row it returned is +one `result_row`, and a failed dispatch stays as one row with a NULL `result_row`. The last SELECT +names the compared field with one `->>` (RULES.md 5). ```sql WITH intent AS ( -- the inputs, declared once as rows SELECT unnest(['2026-10-18', '2026-02-30', $v$O'Brien$v$]) AS raw @@ -166,22 +169,24 @@ print(json.dumps(dict(python_answer=datetime.date.fromisoformat($py$, to_json(ra PY |$cmd$, format := 'newline_delimited')$py$) AS sql FROM intent -), dispatched AS ( -- each input keeps its receipt +), dispatched AS ( -- run them, each receipt beside its input SELECT raw, duckdb_answer, array_agg(selfdispatch(sql)) AS receipts FROM rendered GROUP BY ALL +), result_rows AS ( -- every receipt, and each row it returned + SELECT * EXCLUDE (receipts, receipt), receipt.* + FROM dispatched + CROSS JOIN UNNEST(receipts) AS t(receipt) + CROSS JOIN UNNEST(coalesce(try_cast(receipt.body AS JSON[]), [NULL::JSON])) AS r(result_row) ) -SELECT raw, duckdb_answer, receipt.body::JSON ->> '$[*].python_answer' AS python_answers, - [duckdb_answer] IS DISTINCT FROM python_answers AS differs, receipt.status, receipt.body -FROM dispatched CROSS JOIN UNNEST(receipts) AS t(receipt) +SELECT raw, duckdb_answer, result_row ->> 'python_answer' AS python_answer, status, body, + row(duckdb_answer, 200) IS DISTINCT FROM row(python_answer, status) AS differs +FROM result_rows ORDER BY differs DESC, raw ``` Swap in the real inputs, the DuckDB expression and the proposal (`fromisoformat` here). There is no loop and no `try`: an input the proposal raises on comes back as its own failed receipt (`status` -500, the error in `body`, `python_answers` empty) and counts as `differs`. Verified on dev -2026-10-19: three rows; `2026-10-18` agrees, `2026-02-30` and `O'Brien` differ (Python raised). -To keep a record, wrap the whole query in -`COPY (…) TO '' (FORMAT parquet, PARTITION_BY (type, agent_signature), OVERWRITE_OR_IGNORE true, FILENAME_PATTERN '{uuid}')`; -nothing reads it back to get the answer. A worker testing ponytail's or Thermos's proposal writes -it as the Python (or bash) statement. +500, the error in `body`) and counts as `differs`. A worker testing ponytail's or Thermos's proposal +writes it as the Python (or bash) statement. Verified on dev 2026-10-19: three rows; `2026-10-18` +agrees, `2026-02-30` and `O'Brien` differ (Python raised). ## 4. Evidence is false-first