Skip to content

feat: dialect aware sketch decomposition - #2581

Merged
betodealmeida merged 17 commits into
mainfrom
bdealmeida/dialect-aware-sketch-decomposition
Sep 29, 2026
Merged

betodealmeida merged 17 commits into
mainfrom
bdealmeida/dialect-aware-sketch-decomposition

Conversation

@betodealmeida

Copy link
Copy Markdown
Member

Summary

Rebase of #2559, since I feel bad that GitHub messed up @robinld's PRs. 😆

Test Plan

  • PR has an associated issue: #
  • make check passes
  • make test shows 100% unit test coverage

Deployment Plan

@betodealmeida betodealmeida changed the title Bdealmeida/dialect aware sketch decomposition feat: dialect aware sketch decomposition Sep 24, 2026
@betodealmeida
betodealmeida marked this pull request as ready for review September 24, 2026 16:38
@betodealmeida
betodealmeida added this pull request to stack #2583 September 24, 2026 17:25
@netlify

netlify Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
🔨 Latest commit a98c086
🔍 Latest deploy log https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/6abbdec3e89c2900086ed362

@betodealmeida
betodealmeida force-pushed the bdealmeida/dialect-aware-sketch-decomposition branch from 52437d2 to 33dffa9 Compare September 24, 2026 17:37

@shangyian shangyian left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the new family-decomposition lookup doesn't check which aggregate call it's being asked about?

  1. get_decomposition() applies a metric's sketch family to every aggregate in the formula
  2. then _attach_reaggregate_params broadcasts sketch tuning params onto every component

I think that means it adds reaggregate.params onto every component with aggregation and merge set... so AVG which decomposes into SUM + COUNT will both get the sketch's params attached.

Separately should we add merge_args/serialize etc to MetricComponent so that GraphQL is equivalent with REST?

@robinld
robinld requested a review from shangyian September 26, 2026 06:55
@greptile-apps

greptile-apps Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 3/5

[High risk] Adds dialect-aware sketch decomposition to metric building.

The PR is not safe to merge until Druid cube expressions use the destination's combiner shape and existing parameterized multi-component metrics remain usable.

Findings

  1. P1 Druid cube gets Spark combiner ▶
  2. P1 Multi-component metrics now fail ▶

Summary

This PR adds family-selected, dialect-aware metric decomposition, target-specific sketch serialization, merge arguments, and sketch metadata in REST, GraphQL, and the edit form.

  • Druid cube materialization still decomposes metrics for Spark, leaving dialect-specific cube expressions incorrect.
  • A new parameter guard rejects previously supported multi-component metrics.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Metric with sketch family] --> B[Decompose using build dialect]
  B --> C[Build Spark measures SQL]
  C --> D[Serialize sketches for Druid target]
  B --> E[Store metric combiner for Druid cube]
Loading

Reviews (1) · Last reviewed commit: "Fix sketch decomposition review findings"

Comment thread datajunction-server/datajunction_server/api/cubes.py
Comment thread datajunction-server/datajunction_server/sql/decompose.py
Comment thread datajunction-server/datajunction_server/construction/build_v3/decomposition.py Outdated
Comment thread datajunction-server/datajunction_server/construction/build_v3/decomposition.py Outdated
@betodealmeida
betodealmeida force-pushed the bdealmeida/dialect-aware-sketch-decomposition branch from 74acedf to 568e91a Compare September 28, 2026 21:49
Base automatically changed from bdealmeida/reaggregate-params to main September 28, 2026 22:39
robinld and others added 15 commits September 28, 2026 18:39
Pass the output dialect and source aggregation call through metric decomposition so dialect-specific combiners can be rendered correctly. Add family-based registration so sketch decompositions can be selected explicitly through a metric's reaggregation spec.
Allow components to serialize accumulated values for specific materialization targets and declare the resulting type. Add multi-argument type inference and fixed merge arguments so t-digest measures retain the correct representation and compression when written to Druid.
@betodealmeida
betodealmeida force-pushed the bdealmeida/dialect-aware-sketch-decomposition branch from 4e64fd8 to 8731269 Compare September 28, 2026 22:39
@betodealmeida
betodealmeida merged commit 6ae1462 into main Sep 29, 2026
27 checks passed
@betodealmeida
betodealmeida deleted the bdealmeida/dialect-aware-sketch-decomposition branch September 29, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants