Skip to content

feat: semi-additive measures - #2502

Merged
betodealmeida merged 34 commits into
mainfrom
semi-additive-metrics
Sep 23, 2026
Merged

betodealmeida merged 34 commits into
mainfrom
semi-additive-metrics

Conversation

@betodealmeida

@betodealmeida betodealmeida commented Sep 3, 2026

Copy link
Copy Markdown
Member

Summary

This PR adds metric reaggregation support for semi-additive measures, aligned with the proposal #2245. Metrics can declare protected dimensions, and the aggregation behavior changes when the query grain drops the dimension.

For example, suppose we have this metric:

name: v3.daily_balance
type: metric
query: SELECT SUM(line_total) FROM v3.order_details
reaggregate:
  rules:
    - dimension: v3.date.date_id[order]
      fn: last_value

Meaning:

daily_balance is additive within a day, but not additive across order date. If date disappears from the query grain, collapse with the last date’s value.

If we get a request that includes the protected dimension:

metrics: [v3.daily_balance]
dimensions: [v3.date.date_id[order]]

We generate this SQL:

WITH order_details_0 AS (
  SELECT
    order_date AS date_id_order,
    SUM(line_total) AS line_total_sum
  FROM v3.order_details
  GROUP BY order_date
)
SELECT
  date_id_order,
  SUM(line_total_sum) AS daily_balance
FROM order_details_0
GROUP BY date_id_order

Because date_id_order is still in the output grain, there is no semi-additive collapse. Normal aggregation is safe.

On the other hand, if we don't request the protected dimension we shouldn't aggregate over it:

metrics: [v3.daily_balance]
dimensions: [v3.product.category]

The generated SQL:

WITH order_details_0 AS (
  SELECT
    product_category AS category,
    order_date AS date_id_order,
    SUM(line_total) AS line_total_sum
  FROM v3.order_details
  GROUP BY product_category, order_date
)
SELECT
  category,
  MAX_BY(line_total_sum, date_id_order) AS daily_balance
FROM order_details_0
GROUP BY category

Here, the user asked for category only, but DJ keeps date_id_order as a private inner grain. Then it collapses each category’s daily balances with MAX_BY(value, date), meaning "take the value from the latest date."

Without this, DJ would generate something like:

SELECT
  product_category AS category,
  SUM(line_total) AS daily_balance
FROM v3.order_details
GROUP BY product_category

That would incorrectly sum balances across dates, which is wrong for snapshots or balances, for example.

I've updated the UI to show and allow setting the semi-additive dimension:

Screenshot 2026-09-04 at 2 23 46 PM

When editing:

Screenshot 2026-09-04 at 2 24 22 PM

Valid options:

Screenshot 2026-09-04 at 2 24 27 PM

Test Plan

  • PR has an associated issue: #
  • make check passes
  • make test shows 100% unit test coverage

Deployment Plan

@netlify

netlify Bot commented Sep 3, 2026

Copy link
Copy Markdown

Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
🔨 Latest commit 49d5866
🔍 Latest deploy log https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/6ab3fdd77932e200088b84f8

@betodealmeida betodealmeida changed the title Semi additive metrics feat: semi-additive metrics Sep 3, 2026
@betodealmeida betodealmeida changed the title feat: semi-additive metrics feat: semi-additive measures Sep 3, 2026
@betodealmeida
betodealmeida force-pushed the semi-additive-metrics branch 3 times, most recently from 733896b to 17e0ca9 Compare September 4, 2026 21:13
@betodealmeida
betodealmeida marked this pull request as ready for review September 10, 2026 14:20
@shangyian
shangyian self-requested a review September 14, 2026 16:56
Comment thread datajunction-ui/src/app/pages/AddEditNodePage/ReaggregateFields.jsx
Comment thread datajunction-server/datajunction_server/construction/build_v3/measures.py Outdated
Comment thread datajunction-server/datajunction_server/internal/nodes.py Outdated
Comment thread datajunction-server/datajunction_server/api/cubes.py
Comment thread datajunction-server/datajunction_server/models/reaggregate.py Outdated
Comment thread datajunction-server/datajunction_server/construction/build_v3/metrics.py Outdated
Comment thread datajunction-server/datajunction_server/construction/build_v3/cube_matcher.py Outdated
Comment thread datajunction-query/uv.lock Outdated
Comment thread datajunction-server/datajunction_server/sql/decompose.py

@shangyian shangyian left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@betodealmeida
betodealmeida merged commit a07b417 into main Sep 23, 2026
32 checks passed
@betodealmeida
betodealmeida deleted the semi-additive-metrics branch September 23, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants