Skip to content

Verification is one query whose rows are the answer - #69

Open
asubbarao wants to merge 4 commits into
mainfrom
verification-one-query
Open

asubbarao wants to merge 4 commits into
mainfrom
verification-one-query

Conversation

@asubbarao

Copy link
Copy Markdown
Owner

#57 merged a verification template that wrote both sides to the agent log as parquet and read them back with read_parquet. The comparison doesn't need that round trip: it is one query.

  • agent-dispatch section 3: the template runs the proposal (one shellfs uv run --no-project python - reader, which holds the inputs once) and the DuckDB version over the same rows. It returns one row per input: duckdb_answer, python_answer, python_error, differs. Logging the run is the agent-log COPY wrapped around this same query, and nothing reads it back to get the answer.
  • useful_queries.sql: removes the read_parquet "Verification: both sides" readback entry.

Verified on dev 2026-10-18: T4's proposal against a DuckDB expression on 4 inputs. The +05:00 midnight, the year-1 OverflowError and an input containing ' all come back as differs = true in the query result.

🤖 Generated with Claude Code

@asubbarao

Copy link
Copy Markdown
Owner Author

duckstack/rules: success

  • warn sql_file_too_long in skills/query/references/useful_queries.sql: Keep a job or loader file to 30-80 lines and one concern; split it, or state the reason and line count in the PR.
  • warn sql_vague_cte_names in skills/query/references/useful_queries.sql: Name every CTE for what it holds (content_first_100, self_dispatch), not t / x / sd / tmp.

duckstack review queue, rule gate (hooks/hook_checks.sql)

@asubbarao
asubbarao enabled auto-merge October 19, 2026 00:31
@asubbarao

Copy link
Copy Markdown
Owner Author

duckstack/review: error

The reviewer gave no verdict. Its answer, cut at 3000 characters:

Opus 5.5, duckstack review queue (duckstack-review skill)

asubbarao added a commit that referenced this pull request Oct 19, 2026
…no self-join

Review on #69: duckdb_side was built from python_side and joined back to it; the answer is now one
more column on the same row, so a repeated input cannot multiply rows. Same result on dev.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@asubbarao

Copy link
Copy Markdown
Owner Author

duckstack/rules: success

  • warn sql_vague_cte_names in skills/query/references/useful_queries.sql: Name every CTE for what it holds (content_first_100, self_dispatch), not t / x / sd / tmp.
  • warn sql_file_too_long in skills/query/references/useful_queries.sql: Keep a job or loader file to 30-80 lines and one concern; split it, or state the reason and line count in the PR.

duckstack review queue, rule gate (hooks/hook_checks.sql)

@asubbarao

Copy link
Copy Markdown
Owner Author

duckstack/review: failure

  • skills/duckstack-review/queue.sql:0 review input (How to run it: the review runs on the lines the PR adds): The headless review prompt contained the skill, RULES.md and SQL-MANIFESTO.md but none of the PR's added lines, and the session had no file, shell or MCP tools to fetch them. With no code to read, none of checks 1–10 could run, and a clean verdict would have nothing behind it. Line 0 means no line number could be given.
    Rewrite: Have queue.sql put the PR's added lines in the prompt it sends to the headless review, as the receipt of a gh pr diff <n> read through shellfs (one dispatched statement per queued PR), placed after the skill text. Then run this review again on the same PR head.

Opus 5.5, duckstack review queue (duckstack-review skill)

@asubbarao

Copy link
Copy Markdown
Owner Author

duckstack/review: failure

  • skills/agent-dispatch/SKILL.md:176 RULES.md 3 and 5 (author's reason does not hold): The stated reason for keeping the inputs in the script ("a self-dispatched Python side returns receipts (status, body as text), not the script's columns") contradicts RULES.md 5: what you build in the self-dispatch comes back as a column when you unnest the array. RULES.md 3 also says self-dispatch is the escape hatch for exactly this column-into-reader binder limit, so the reason does not justify the pattern.
    Rewrite: Replace lines 174–177 with: "The inputs are rows in the first CTE. Each one renders a one-input Python statement, which is self-dispatched beside its duckdb_answer, and the receipts are unnested back into columns (r.receipt.*). An input the proposal raises on comes back as its own receipt with the error."
  • skills/agent-dispatch/SKILL.md:161 check 1 / RULES.md 6 and 4: A Python for loop with a per-row try/except runs the proposal once per input. That is a loop in a script doing per-row work that should be one dispatched statement per row, where a failing input comes back as its own receipt with its error kept.
    Rewrite: Make the Python a one-input program: print(json.dumps(dict(python_answer=<proposal>({{ raw | json_encode() }})))), rendered per intent row and sent with array_agg(selfdispatch(sql)). Do not write a for raw in [...] loop or a try/except that builds error rows. An input the proposal raises on comes back as a failed receipt that carries the exception, joined to its raw and duckdb_answer in the same struct.
  • skills/agent-dispatch/SKILL.md:158 RULES.md 7 / check 4 (no intent CTE): The first CTE is python_side, and the inputs are hidden as [<inputs>] inside a Python heredoc. A reader of the SQL cannot see what is being compared, and the DuckDB side only gets its inputs back from whatever Python re-serialises as raw.
    Rewrite: WITH intent AS ( -- the inputs, declared once as rows
    SELECT unnest(['', '', '']) AS raw
    ), rendered AS (
    SELECT raw, AS duckdb_answer,
    tera_render($tera$
    SELECT * FROM read_json($cmd$/opt/homebrew/bin/uv run --quiet --no-project python - <<'PY'
    import json
    print(json.dumps(dict(python_answer=({{ raw | json_encode() }}))))
    PY
    |$cmd$, format := 'newline_delimited')
    $tera$, i::JSON, autoescape := false) AS sql
    FROM intent AS i
    ), dispatched AS (
    SELECT array_agg({raw: raw, duckdb_answer: duckdb_answer, receipt: selfdispatch(sql)}) AS receipts FROM rendered
    )
    SELECT r.raw, r.duckdb_answer, r.receipt.*
    FROM dispatched CROSS JOIN UNNEST(receipts) AS t(r)

Opus 5.5, duckstack review queue (duckstack-review skill)

asubbarao added a commit that referenced this pull request Oct 19, 2026
A dispatched read of a review's result file that fails to reach dev
(status 0, empty body) was posted as "the reviewer gave no verdict", as
on #69 at 53b2cbf. Now it posts nothing and the next tick reads the file
again; dev's own error about a bad file still posts `error` with its text.
Tested by aiming the result reads at a dead port: status 0, no posts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
asubbarao and others added 3 commits October 19, 2026 09:43
… trip

The template runs the proposal and the DuckDB version on the same inputs and returns one row per
input with both answers and a differs flag. Logging is the agent-log COPY wrapped around that same
query, never read back to get the answer. Removes the read_parquet readback entry from
useful_queries. Verified on dev.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…no self-join

Review on #69: duckdb_side was built from python_side and joined back to it; the answer is now one
more column on the same row, so a repeated input cannot multiply rows. Same result on dev.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…op or try

The inputs are rows in the first CTE; each renders a one-input Python statement that
is self-dispatched beside its duckdb_answer, and the receipts unnest back into columns.
An input the proposal raises on is its own failed receipt and counts as differs (the
old loop's NULL-vs-NULL compare marked raising inputs as agreeing). The example runs
on dev as written. Record: COPY the whole query, as in PR #60; nothing reads it back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@asubbarao
asubbarao force-pushed the verification-one-query branch from 285cdcb to 6e4d7cd Compare October 19, 2026 16:44
…cord

The body rows unnest once (coalesce(try_cast(body AS JSON[]), [NULL::JSON])), so a
failed dispatch stays as one row, and python_answer is read with a single ->> instead
of a $[*] path. The COPY-to-agent-log sentence is gone: the query's rows are the answer.
Same three rows as the previous version (table_diff_summary pk raw: 3 identical).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
asubbarao added a commit that referenced this pull request Oct 19, 2026
#69 rewrites the same block (one dispatched statement per input, no COPY, no readback); this PR keeps the checklist and compendium fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@asubbarao

Copy link
Copy Markdown
Owner Author

duckstack/rules: success

  • existing sql_file_too_long in skills/query/references/useful_queries.sql: Keep a job or loader file to 30-80 lines and one concern; split it, or state the reason and line count in the PR.
  • existing sql_vague_cte_names in skills/query/references/useful_queries.sql: Name every CTE for what it holds (content_first_100, self_dispatch), not t / x / sd / tmp.

duckstack review queue, rule gate (hooks/hook_checks.sql)

@asubbarao

Copy link
Copy Markdown
Owner Author

duckstack/review: failure

  • skills/agent-dispatch/SKILL.md:166 RULES.md 7 / check 4 (intent CTE), RULES.md 16-17: The proposal is the one thing line 185 tells the reader to swap in, but it is not declared in the first CTE. It is buried in the third quoting layer: $py$ wraps $cmd$, which wraps a heredoc, which wraps Python. To change it, the reader has to edit inside nested quotes instead of changing a value in intent.
    Rewrite: WITH intent AS ( -- the inputs and the proposal under test, declared once
    SELECT unnest(['2026-10-18', '2026-02-30', $v$O'Brien$v$]) AS raw,
    'datetime.date.fromisoformat(value).isoformat()' AS proposal
    ), rendered AS ( -- per input: the DuckDB answer, and the proposal as a one-input statement
    SELECT raw, try_strptime(raw, '%Y-%m-%d')::DATE::VARCHAR AS duckdb_answer,
    tera_render($tera$SELECT * FROM read_json($cmd$/opt/homebrew/bin/uv run --quiet --no-project python - <<'PY'
    import json, datetime
    value = {{ raw | json_encode() }}
    print(json.dumps(dict(python_answer={{ proposal }})))
    PY
    |$cmd$, format := 'newline_delimited')$tera$, i::JSON, autoescape := false) AS sql
    FROM intent AS i
    ) -- 9 lines before, 10 after: the template only substitutes; the swap is one value in intent

Opus 5.5, duckstack review queue (duckstack-review skill)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant