Skip to content

feat(cli): add sqlmesh history to inspect a plan's query engine history [EXPERIMENT: DO NOT MERGE]#5898

Open
sungchun12 wants to merge 2 commits into
SQLMesh:mainfrom
sungchun12:sung/explain-more
Open

feat(cli): add sqlmesh history to inspect a plan's query engine history [EXPERIMENT: DO NOT MERGE]#5898
sungchun12 wants to merge 2 commits into
SQLMesh:mainfrom
sungchun12:sung/explain-more

Conversation

@sungchun12

Copy link
Copy Markdown
Contributor

What

Adds sqlmesh history [PLAN_ID] — a read-only debugging command that shows the query engine history of everything SQLMesh ran for a given plan (success/failure, duration, bytes), or exports the executed SQL with -o. The goal is to debug "something looks off with how SQLMesh behaved" without context-switching into the warehouse's query history or SQLMesh state.

How

Every query SQLMesh runs is already tagged with a correlation id (CorrelationId). history looks that up in the engine's native history:

  • BigQuery (v1): INFORMATION_SCHEMA.JOBS, matched on the sqlmesh_plan job label; the region is resolved dynamically from the project; a Forbidden becomes a clear error with a gcloud remediation command and a docs link.
  • Other engines: a clear "not supported" message (see Scope).

No PLAN_ID → an interactive numbered menu of plans from state (environment, plan id, applied time, model count). Or pass a plan id directly. -o/--output-file exports the statements to a replayable .sql file.

                    History · plan 3f9a2c1e · 41 queries · 38 ✓  2 ✗  1 running
 Time      Status   Operation       Duration     Bytes/Rows
 14:03:11  ✓        CREATE TABLE    1200 ms     1,048,576 bytes
 14:03:29  ✗        INSERT          420 ms                   -
            Syntax error: unexpected identifier 'PRICE' at [1:34]
2 failed · re-run with `-o failures.sql` to export the SQL.

Scope / notes

  • BigQuery only in this version. DuckDB/Postgres/etc. have no queryable history; supporting them would require SQLMesh to persist its own executed queries (a follow-up).
  • Plans only — scheduled runs are not correlated yet (JobType.RUN is currently unused).
  • Requires bigquery.jobs.listAll; history views have ingestion latency + finite retention; unlabeled dataframe/seed load jobs won't appear. All documented.

Tests

  • Unit: SQL generation (label filter), status mapping, permission error (including that the remediation + docs surface via str(exc)), dynamic region resolution, unsupported-engine, console rendering + plan menu, and Context.plan_history paths.
  • Integration: test_query_history in test_integration_bigquery.py exercises the real BigQuery path — empty-tolerant (JOBS ingestion latency) and skips if the service account lacks jobs.listAll.

Docs

  • docs/reference/cli.md ## history entry + a new docs/guides/history.md walkthrough with example output.

🤖 Generated with Claude Code

…tory

Adds a read-only `sqlmesh history [PLAN_ID]` command that reconstructs
everything SQLMesh ran for a plan by querying the query engine's native
query history (matched on the correlation id SQLMesh attaches to every
query), rendering it chronologically with success/failure, duration, and
bytes -- or exporting the SQL with `-o`. This removes the need to context
-switch into the warehouse's query history or SQLMesh state when debugging.

- BigQuery (v1): INFORMATION_SCHEMA.JOBS, matched on the `sqlmesh_plan`
  job label; region resolved dynamically from the project; a clear,
  actionable permission error (gcloud remediation + docs) on jobs.listAll.
- Interactive numbered plan menu sourced from state, or pass a plan id.
- Other engines raise a clear "not supported" message.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sungchun12 sungchun12 changed the title feat(cli): add sqlmesh history to inspect a plan's query engine history feat(cli): add sqlmesh history to inspect a plan's query engine history [EXPERIMENT: DO NOT MERGE] Jul 10, 2026
Follow-up to `sqlmesh history` addressing how interwoven Python models,
hooks, and signals appear:

- Add a `Target` column showing the physical table each job wrote to
  (from BigQuery's INFORMATION_SCHEMA.JOBS destination_table). This makes
  interleaved Python/SQL model steps and pre/post-hook statements legible
  instead of an anonymous stream of INSERT/CREATE. Anonymous SELECT result
  tables are omitted.
- Label BigQuery load jobs (LoadJobConfig) with the correlation id, so
  Python-model dataframe loads and seed loads appear as `LOAD` rows instead
  of being invisible. Factored the label out into `_correlation_labels`,
  shared by query and load jobs.
- Document that signals (Python readiness checks) and pure-Python model
  logic run no SQL and don't appear; point to `check_intervals`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant