feat(cli): add sqlmesh history to inspect a plan's query engine history [EXPERIMENT: DO NOT MERGE]#5898
Open
sungchun12 wants to merge 2 commits into
Open
feat(cli): add sqlmesh history to inspect a plan's query engine history [EXPERIMENT: DO NOT MERGE]#5898sungchun12 wants to merge 2 commits into
sqlmesh history to inspect a plan's query engine history [EXPERIMENT: DO NOT MERGE]#5898sungchun12 wants to merge 2 commits into
Conversation
…tory Adds a read-only `sqlmesh history [PLAN_ID]` command that reconstructs everything SQLMesh ran for a plan by querying the query engine's native query history (matched on the correlation id SQLMesh attaches to every query), rendering it chronologically with success/failure, duration, and bytes -- or exporting the SQL with `-o`. This removes the need to context -switch into the warehouse's query history or SQLMesh state when debugging. - BigQuery (v1): INFORMATION_SCHEMA.JOBS, matched on the `sqlmesh_plan` job label; region resolved dynamically from the project; a clear, actionable permission error (gcloud remediation + docs) on jobs.listAll. - Interactive numbered plan menu sourced from state, or pass a plan id. - Other engines raise a clear "not supported" message. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sqlmesh history to inspect a plan's query engine historysqlmesh history to inspect a plan's query engine history [EXPERIMENT: DO NOT MERGE]
Follow-up to `sqlmesh history` addressing how interwoven Python models, hooks, and signals appear: - Add a `Target` column showing the physical table each job wrote to (from BigQuery's INFORMATION_SCHEMA.JOBS destination_table). This makes interleaved Python/SQL model steps and pre/post-hook statements legible instead of an anonymous stream of INSERT/CREATE. Anonymous SELECT result tables are omitted. - Label BigQuery load jobs (LoadJobConfig) with the correlation id, so Python-model dataframe loads and seed loads appear as `LOAD` rows instead of being invisible. Factored the label out into `_correlation_labels`, shared by query and load jobs. - Document that signals (Python readiness checks) and pure-Python model logic run no SQL and don't appear; point to `check_intervals`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
sqlmesh history [PLAN_ID]— a read-only debugging command that shows the query engine history of everything SQLMesh ran for a given plan (success/failure, duration, bytes), or exports the executed SQL with-o. The goal is to debug "something looks off with how SQLMesh behaved" without context-switching into the warehouse's query history or SQLMesh state.How
Every query SQLMesh runs is already tagged with a correlation id (
CorrelationId).historylooks that up in the engine's native history:INFORMATION_SCHEMA.JOBS, matched on thesqlmesh_planjob label; the region is resolved dynamically from the project; aForbiddenbecomes a clear error with agcloudremediation command and a docs link.No
PLAN_ID→ an interactive numbered menu of plans from state (environment, plan id, applied time, model count). Or pass a plan id directly.-o/--output-fileexports the statements to a replayable.sqlfile.Scope / notes
runs are not correlated yet (JobType.RUNis currently unused).bigquery.jobs.listAll; history views have ingestion latency + finite retention; unlabeled dataframe/seed load jobs won't appear. All documented.Tests
str(exc)), dynamic region resolution, unsupported-engine, console rendering + plan menu, andContext.plan_historypaths.test_query_historyintest_integration_bigquery.pyexercises the real BigQuery path — empty-tolerant (JOBS ingestion latency) and skips if the service account lacksjobs.listAll.Docs
docs/reference/cli.md## historyentry + a newdocs/guides/history.mdwalkthrough with example output.🤖 Generated with Claude Code