The ClickHouse edge answers system.*, DESCRIBE, SHOW and EXISTS from an emulated catalog (T-482, T-483) - #352
Merged
chasers merged 2 commits intoSep 20, 2026
Conversation
chasers
added this pull request to stack #354
September 20, 2026 00:57
…an emulated catalog (T-482, T-483) A ClickHouse client reads the catalog before anything a user typed. HyperDX asks system.settings before its first query and fails every query after if that one fails; it reads columns with DESCRIBE and the sorting key from system.tables. clickhouse-go sends DESCRIBE TABLE before each batch. All of it answered "unknown table", and SHOW TABLES answered DuckDB's own empty catalog, which is a wrong answer rather than a failure. SmolqueryClickHouse.SystemCatalog is built the way SmolqueryPg.PgCatalog is: a GenServer per edge owning a private engine. system.databases, tables and columns are rebuilt from Smolquery.Catalog at most once a second; settings and data_skipping_indices are empty; table_engines and one are static. A table's engine is MergeTree, its sorting_key and primary_key the clustering key, a column's type the ClickHouse type an insert reads it as. - DESCRIBE [TABLE], SHOW DATABASES, SHOW TABLES [FROM] and EXISTS [TABLE] are read at the edge and answered from those tables. - A SELECT is the catalog's only when DuckDB's parse says every table it reads is a system one. system.x is written system_x first, and a bare "table", a reserved word to the engine, is quoted. - A system table that is not emulated is code 60 by its ClickHouse name; a catalog that cannot be read is a retryable 503, never an empty answer. - The edge runtime gains :catalog, resolved as the Postgres edge's is. serialize, base_tables and listed_tables move out of PgCatalog into Smolquery.CatalogEmulation, since both edges now need them and ex_dna would rightly refuse a second copy. Tested with the statements HyperDX sends, verbatim, parameters included. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r, caps its answer, outlives a slow statement, and is not asked about a literal
The edge's emulated catalog runs a client's SELECT in its own DuckDB
engine, and that engine was started bare. The review read a host file
through it:
GET /?query=SELECT * FROM system.one, read_text('/etc/hostname')
The only gate looked at BASE_TABLE nodes, so a table function beside a
system table passed, and it passed the GET read-only check too. The job
engines that run a user's SQL are locked down by the query service; this
one was not.
- The engine has external access off and its configuration locked as soon
as its tables exist. No statement reads a file or a URL, whatever reaches
it. A test asks the engine directly.
- A statement that names a table function anywhere in its AST, or a
RECURSIVE table expression, is not the catalog's. It has nothing to read
here but a generator, and range(100000000000) has no end.
- An answer is cut at 10,000 rows and a statement is given 10 seconds, in a
task. A statement that outlives it answers a retryable 503 and leaves the
server standing. Before, Engine.frame's call exit crashed the GenServer,
and rest_for_one restarted admission and the listener under it, dropping
every request in flight.
- mentions_catalog? read the raw statement, so WHERE MetricName =
'system.cpu.utilization' paid a call to the server and three parses to
learn it was not a catalog statement, and answered 503 when the server
was busy. It reads code tokens now.
Smolquery.CatalogEmulation gets the mirrored test file AGENTS.md asks for,
and the runtime's three new names are tested.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
chasers
force-pushed
the
t-482-clickhouse-system-catalog
branch
from
September 20, 2026 03:11
363b61f to
f2a6575
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR: The ClickHouse edge answers
system.*,DESCRIBE,SHOWandEXISTSfrom an emulated catalog, built the way the Postgres edge buildspg_catalog.Tracker: T-482, T-483. Plan PL-66. Stacked on #351 (T-493).
Why
system.settingsbefore its first query. If that fails, every query after it fails.DESCRIBEand the sorting key fromsystem.tables.SHOW TABLESwas worse: it listed DuckDB's own empty catalog. A wrong answer, not an error.What changed
SmolqueryClickHouse.SystemCatalog: a GenServer per edge with a private engine.:catalog, resolved as the Postgres edge's is.serialize,base_tables,listed_tablesmove fromPgCatalogtoSmolquery.CatalogEmulation. Both edges use them.What answers
system.databases,tables,columnssystem.settings,data_skipping_indicessystem.table_engines,system.oneDESCRIBE [TABLE] db.tSHOW DATABASES,SHOW TABLES [FROM db],EXISTS [TABLE] db.tengineisMergeTree. Itssorting_keyandprimary_keyare the clustering key.How it decides
SELECTis the catalog's only when DuckDB's parse says every table it reads is asystemone.system.xis writtensystem_xfirst. A baretablecolumn is quoted: it is a reserved word to DuckDB.Watch out
systemtable that is not emulated is code 60, by its ClickHouse name.system.numbers(needs a table function),SHOW CREATE TABLE.How to review
system_catalog.ex— the moduledoc table, thenread/2andclassified/2.catalog_emulation.ex— moved code, no change.system_catalog_test.exs— the statements are HyperDX's, verbatim.Checks
mix precommitmix cimix dialyzerReview fixes
SELECT * FROM system.one, read_text('/etc/hostname')answered the file. The catalog's engine now has external access off and its configuration locked.RECURSIVEtable expression, is not the catalog's.'system.cpu'no longer asks the catalog server anything.Smolquery.CatalogEmulation.🤖 Generated with Claude Code