The ClickHouse edge writes down the statements it could not answer, with the client that sent them (T-480) - #360
Merged
chasers merged 3 commits intoSep 20, 2026
Conversation
chasers
added this pull request to stack #354
September 20, 2026 02:16
…ith the client that sent them (T-480) What a ClickHouse client sends is known only from what it has sent. The dialect work is ordered by a corpus, and until now the corpus came from reading HyperDX's source. A first run of a real HyperDX will send things the source did not show, and they have to land somewhere. SmolqueryClickHouse.Unanswered logs a statement the edge refused for a reason that is the dialect's — 62 SYNTAX_ERROR, 46 UNKNOWN_FUNCTION, 47 UNKNOWN_IDENTIFIER, 60 UNKNOWN_TABLE, 73 UNKNOWN_FORMAT, 456 UNKNOWN_QUERY_PARAMETER, 36 BAD_ARGUMENTS — as the client sent it, before any rewrite, on one line, with its user-agent, and counts it in smolquery_clickhouse_unanswered_total by code. A timeout, a full node and a result past its cap say nothing about the dialect and are not logged. A statement's literals are a user's search terms, so by default each string literal is written '?' and the engine's error is cut to its first line, since the lines after it quote the statement. Parameter values are in the URL and are never logged. SMOLQUERY_CLICKHOUSE_UNANSWERED_LOG=verbatim keeps the literals, for an operator reproducing a client's failure, and off logs nothing while the counter still counts. Seen working against HyperDX's client: an ARRAY JOIN arrived in the log as user_agent="clickhouse-js/1.23.0-head... (lv:nodejs/v22; os:linux)". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tring and for a comment
The default log mode replaces a statement's string literals, since they are
a user's search terms. The review found three ways a value still reached
the log:
- The error is logged beside the statement, and Params' BAD_ARGUMENTS
message quotes the value it could not parse: SELECT {id:UInt64} with
alice@example.com logged the address, though the moduledoc said a
parameter's value is never logged.
- A parser error quotes the token it stopped at, which can be a literal:
syntax error at or near "'secret2'".
- A $$dollar-quoted$$ string and a comment were left as written.
Under :redacted, whatever the error quotes is replaced unless it is a bare
word: a keyword or a name the parser stopped at is what the corpus needs,
and a value or a literal is a user's. Dollar-quoted strings and comments
are replaced with the literals. :verbatim is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…O setting, catalog failures that reach the log, and docs that agree The smaller findings of the stack's review, none of them a layer's alone: - The engine's error text was classified twice, in Query and in the emulated catalog, and the catalog's copy answered everything but an unknown table as 1002. A failing system.* probe such as currentDatabase() was therefore never logged or counted by T-480. Errors.engine_failure/1 is the one reading now: 46 for an unknown function wherever the statement ran. - A statement error that is the server's — HTTP Error, Connection Error — had become a 400 when run-time errors were classified. It is a 500 again, which a client may retry. - SETTINGS date_time_output_format = 'iso' in a statement was read for the timeout's sake and ignored for the answer; only the URL chose the style. The merged settings now choose both. - The macro module's doc said the Top-N probe runs a statement's macros. It does not: Top-N refuses a statement that calls one, so HyperDX's searches lose that pruning. The doc says so and T-504 tracks it. - docs/clickhouse-sql-gaps.md listed groupArray and isNull as missing in one table and working in another, and its HyperDX row still listed what T-496 closed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
chasers
force-pushed
the
t-480-clickhouse-unanswered-log
branch
from
September 20, 2026 03:11
0e5b51a to
6b78454
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR: The ClickHouse edge now logs and counts the statements it could not answer, with the client's
user-agent. A first real HyperDX run leaves the list of what to build next.Tracker: T-480. Plans PL-65 (D1) and PL-66. Stacked on #359 (T-496).
Why
What changed
SmolqueryClickHouse.Unanswered.smolquery_clickhouse_unanswered_total, by ClickHouse error code.SMOLQUERY_CLICKHOUSE_UNANSWERED_LOG=redacted(default),verbatim,off.What gets logged
SYNTAX_ERROR, 46UNKNOWN_FUNCTION, 47UNKNOWN_IDENTIFIERUNKNOWN_TABLE, 73UNKNOWN_FORMATUNKNOWN_QUERY_PARAMETER, 36BAD_ARGUMENTSPrivacy
redacted(default): every string literal becomes'?'. The engine's error is cut to its first line, because the lines after it quote the statement.verbatimkeeps literals. Use it for a first HyperDX run, or to reproduce a failure.offlogs nothing. The counter still counts.Watch out
@moduletag :capture_log, so refusals they provoke stay out of the suite output.Checks
mix precommitmix cimix dialyzerARRAY JOINarrived in the log withuser_agent="clickhouse-js/1.23.0-head… (lv:nodejs/v22…)".Review fixes
system.*probe reaches this log.SETTINGS date_time_output_formatis honoured, transient engine errors are a 500 again, and the gap doc no longer contradicts itself.🤖 Generated with Claude Code