Skip to content

HyperDX's own code runs against the edge, 17 of 17: parametric aggregates, LIKE's escape, arrays, and the words DuckDB keeps (T-496) - #357

Merged
chasers merged 2 commits into
t-500-hyperdx-docfrom
t-496-hyperdx-sidebar-syntax
Sep 20, 2026
Merged

chasers merged 2 commits into
t-500-hyperdx-docfrom
t-496-hyperdx-sidebar-syntax

Conversation

@chasers

@chasers chasers commented Sep 20, 2026

Copy link
Copy Markdown
Owner

TL;DR: HyperDX's own code now runs against the edge, 17 of 17 steps. This PR fixes the 3 failures and 1 wrong answer that run found.

Tracker: T-496. Plan PL-66. Stacked on #356 (T-500).

How it was found

  • Earlier layers were checked against SQL copied from HyperDX's source.
  • This time HyperDX's code ran: its SQL generator, metadata reader and client wrapper (hyperdxio/hyperdx @ c42dda8), over the real @clickhouse/client.
  • Target: a local smolquery node with 300 rows.

What the run found

Step Result before Cause
Search (11 forms), histogram, columns, table metadata ✅ passed
Field list, map keys, sidebar values ❌ syntax error groupUniqArray(20)(x), groupUniqArrayArray(1000)(keys)
Search for user_id ❌ 0 rows of 75 LIKE lower('%user\_id%'): DuckDB's LIKE has no default escape character
Hand-written FROM default.otel_logs ❌ syntax error default is a reserved word in DuckDB

What changed

  • Rewrite: a parametric aggregate f(params)(args) becomes f(args, params). quantile is quantile_cont.
  • Rewrite: a LIKE pattern holding a backslash gets ESCAPE '\'.
  • Rewrite: a bare default. is quoted. isNull, isNotNull, any are renamed: they are the parser's own words.
  • Macros: groupUniqArray, groupUniqArrayArray, groupArray, their -If forms, quantileIf, getSubcolumn, lowCardinalityKeys.
  • Format: a list answers as Array(Nullable(T)). It was a String holding JSON. HyperDX reads keysArr as an array.

Watch out

  • ⚠️ Behavior change: a list column in a result is now Array(...), in every format. Before it was JSON text typed String.
  • ⚠️ An unaliased expression is named as DuckDB writes it (quantile_cont(...)), not as ClickHouse does. Only count() is renamed.
  • ⚠️ Still not run: HyperDX's UI and API server. No container runs on the dev box.
  • ⚠️ Still missing: WITH (expr) AS alias, JSONExtract(s, 'Map(...)'), toJSONString, CSV output.

How to repeat the run

  • scripts/hyperdx-probe/probe.ts. Its header has the setup: clone HyperDX, npm install, run with tsx. Node only, no container.

Tests

  • ✅ The three failing statements joined the fixture; the integration test asserts them.
  • ✅ Rewrite, macro and array-format unit tests.
  • ✅ Five chart configurations through HyperDX's chart builder answer (quantile by group, avg, max, filtered count and sum, count(DISTINCT)).

Checks

  • mix precommit
  • mix ci
  • mix dialyzer

Review fixes

  • x = ANY(SELECT ...) keeps its ANY. Only ClickHouse's aggregate any(x) is renamed.
  • ✅ A list of structs answers as JSON text again, not as the text of an Elixir term.

🤖 Generated with Claude Code

@chasers
chasers added this pull request to stack #354 September 20, 2026 01:21
@gitguardian

gitguardian Bot commented Sep 20, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
37463487 Triggered Generic Password 4eb5520 scripts/hyperdx-probe/probe.ts View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Chase Granberry and others added 2 commits September 20, 2026 02:54
…ates, LIKE's escape, arrays, and the words DuckDB keeps (T-496)

Until now the edge was checked against SQL copied from HyperDX's source.
This time HyperDX's code ran: its SQL generator, metadata reader and client
wrapper (hyperdxio/hyperdx @ c42dda8) over the real @clickhouse/client,
against a smolquery node with 300 rows. Search and the histogram passed.
Three things failed and one answered wrongly:

- getAllFields, getMapKeys and getKeyValues: a syntax error on
  groupUniqArrayArray(1000)(keys) and groupUniqArray(20)(x).
- A search for user_id found 0 rows of 75. HyperDX writes
  LIKE lower('%user\_id%'), and the engine's LIKE has no escape character
  unless told, so the backslash matched itself.
- A hand-written FROM default.otel_logs did not parse: default is a word
  the engine reserves.

SmolqueryClickHouse.Rewrite now moves a parametric aggregate's parameters
behind its arguments (quantile is quantile_cont, since ClickHouse's
interpolates), adds ESCAPE '\' after a LIKE pattern holding a backslash,
quotes a bare default database, and renames isNull, isNotNull and any,
which are the parser's own words. The macros gain groupUniqArray,
groupUniqArrayArray, groupArray and their -If forms, quantileIf,
getSubcolumn and lowCardinalityKeys.

HyperDX reads keysArr as an array, and a list answered as a String holding
JSON. Format now answers a list as Array(Nullable(T)): a JSON array,
['a','b'] in a tab-separated row, a length and elements in RowBinary.

scripts/hyperdx-probe/probe.ts is the run, with how to repeat it. The three
statements join the fixture, so the integration test keeps them working.
Five chart configurations through HyperDX's chart builder answer too.

Still not run: HyperDX's UI and API server. No container runs on the dev
box, and docs/clickstack.md says so.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…swers as JSON again, not as an Elixir term

- any( was renamed to any_value( wherever it appeared, which is right for
  ClickHouse's aggregate and wrong for the quantifier both dialects have.
  WHERE x = ANY(SELECT y FROM u) became x = any_value(SELECT ...), a parse
  error, and the statement had run before this branch. An ANY( after a
  comparison operator is now left alone. "ANY (" with a space never
  matched, so the behavior had also depended on whitespace.
- Answering a list as Array(...) caught every {:list, _} dtype, and a map
  whose values are not strings crosses the frame as a list of key/value
  structs. SELECT MAP {1: 2} answered the text of an Elixir term,
  [%{"key" => 1, "value" => 2}], in a tab-separated row, where it had
  answered JSON; and inside an array RowBinary wrote it with no Nullable
  marker under a type that promised one. A list of structs is a String
  holding JSON again, Nullable(String) as an array's element, and every
  format writes what its type says.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@chasers
chasers force-pushed the t-496-hyperdx-sidebar-syntax branch from 970c7ea to 59bb6a9 Compare September 20, 2026 03:11
@chasers
chasers merged commit 457f1c9 into main Sep 20, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant