Skip to content

fixed_grain for level metrics - #2532

Open
robinld wants to merge 27 commits into
DataJunction:semi-additive-metricsfrom
robinld:robind/fixed-grain
Open

robinld wants to merge 27 commits into
DataJunction:semi-additive-metricsfrom
robinld:robind/fixed-grain

Conversation

@robinld

@robinld robinld commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Implements the fixed_grain half of #2245. A base metric can declare the grain at which its aggregate is computed, and DJ broadcasts that value when the query requests a finer grain.

This PR is stacked on #2502, which implements the complementary reaggregate direction. It can be retargeted to main after #2502 lands.

- name: revenue
  query: SELECT SUM(revenue) FROM sales_f

- name: total_revenue
  query: SELECT SUM(revenue) FROM sales_f
  fixed_grain: []

- name: pct_of_total
  query: SELECT revenue / total_revenue

fixed_grain has three forms:

  • omitted: compute at the query grain, preserving current behavior
  • []: compute at the global grain
  • [dimension, ...]: compute at the declared partition grain

For example, querying total_revenue at a finer grain generates the equivalent of:

SUM(SUM(sales_f_0.revenue_sum)) OVER () AS total_revenue

The inner aggregate computes the query-group contribution. The outer window re-merges those contributions within the declared fixed grain. A non-empty grain produces OVER (PARTITION BY ...) instead.

Implementation

  • sql/decompose.py validates that the declaration belongs to one freely aggregatable base component, propagates it through decomposition, and rejects a later window over a broadcast value.
  • The broadcast is written into the combiner during decomposition, naming partition dimensions in unresolved form. Each consumer resolves them as it does any dimension reference — a CTE-qualified column for a query, a bare column for a cube — so the window reaches both query SQL and the SQL stored on a materialized cube.
  • construction/build_v3/cube_matcher.py skips a cube that does not materialize a metric's partition dimension, mirroring validate_cube_covers_reaggregate_dimensions. A cube that does carry it is used normally.
  • The declaration is carried through storage, deployment specs, namespace export/copy, REST and GraphQL output, loader allowlists, change classification, and semantic fingerprints.

Current constraints

The initial implementation accepts only cases where broadcasting can be expressed without changing the metric's meaning.

Situation Behavior
COUNT(DISTINCT ...), a multi-component metric (AVG, variance), or a non-decomposable aggregate Refused; the result cannot be re-merged as a window. Multi-component is unimplemented rather than impossible — it needs every merge windowed, not just the first
A declared grain dimension is absent from the query grain Joined and grouped on as private grain, then projected away — the same treatment a protected reaggregate dimension gets. The result stays the shape that was requested
A non-empty fixed grain is combined with another fact grain group Refused; the joined SELECT groups over a COALESCE, so a partition naming one group's column is ungrouped
A derived metric declares its own fixed grain Refused; the declaration belongs to the base aggregate
A frame accumulates a fixed-grain metric, directly or transitively Refused; the frame would add the same broadcast value once per row. MAX/AVG over it are allowed, as is referencing it beside a window — e.g. a trailing sum divided by a global total
A cube does not materialize the partition dimension Cube skipped; DJ builds from the underlying graph. A cube that carries the dimension is used normally

Related fixes

  • Normalizes both frozen-measure creation paths so metric-level reaggregate and fixed_grain declarations do not become part of shared measure state or identity.
  • Adds both declarations to the GraphQL extractedMeasures loader's load_only. Reading a field that loader omits raises MissingGreenlet on a cold session, which the batch handler turns into a silent null rather than an error.

Out of scope

  • Relative grain declarations such as {hierarchy, drill_up}
  • The proposed frame primitive
  • UI support. reaggregate gained create/edit fields and node-page display in feat: semi-additive measures #2502; the equivalent for fixed_grain is deferred to a follow-up, so the declaration is currently settable only through deployment specs and the API.

Known gaps

  • Direct POST /nodes/metric validation does not yet reject an unresolvable fixed-grain column, although deployment validation does. The invalid declaration is rejected later when a query attempts to use it.

Testing

Coverage includes global and partitioned broadcasts, derived ratios, declaration round trips, namespace copying, semantic identity, frozen-measure reuse, GraphQL decomposition loading, window refusal, and cube matching and materialization carrying the broadcast.

@robinld
robinld marked this pull request as ready for review September 9, 2026 22:09
@robinld
robinld requested a review from doulam September 9, 2026 22:10
@robinld robinld linked an issue Sep 10, 2026 that may be closed by this pull request
Comment thread datajunction-server/datajunction_server/construction/build_v3/measures.py Outdated
Comment thread datajunction-server/datajunction_server/sql/decompose.py
Comment thread datajunction-server/datajunction_server/sql/decompose.py Outdated
Comment thread datajunction-server/datajunction_server/internal/deployment/validation.py Outdated
Comment thread datajunction-server/tests/construction/build_v3/fixed_grain_test.py
shangyian and others added 5 commits September 11, 2026 16:06
An experimentation metric is a ratio of two unit-grain metrics: the
activity each allocated unit contributed, over the allocated population.
Writing that shape down here records where this branch stops.

Two refusals land on the shape rather than on an edge case. A population
written as COUNT(DISTINCT unit) at unit grain is refused, though at unit
grain the count is one per unit and the merge is a plain sum. And the
ratio always spans two facts, since the population comes from the
allocation table, so the cross-fact refusal applies to every one of
these metrics.

The global grain the refusal suggests does build, and answers a
different question: OVER () broadcasts the total across every group, so
the result divides an all-time total by one month's population.

The last test is xfail rather than an assertion on the generated SQL. A
fixed-grain dimension outside the query grain never reaches the measures
CTE, so the window partitions by a column that does not exist. Pinning
that SQL would lock in something no engine accepts, so it asserts only
the invariant: a column read from a CTE has to be projected by it.
@robinld
robinld requested a review from shangyian September 12, 2026 00:44

@shangyian shangyian left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: Level (and other Derived) Metrics

3 participants