Skip to content

perf: precompute org page contributors leaderboard #4501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
epipav wants to merge 1 commit into
base: main
Choose a base branch
from
+106 −35
Open

perf: precompute org page contributors leaderboard #4501

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions services/libs/tinybird/datasources/org_page_contributors_copy_ds.datasource
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
DESCRIPTION >
Precomputed all-time contribution counts per (organizationId, memberId) for the org
page contributors leaderboard. Rebuilt nightly by org_page_contributors_copy_pipe.
One row per (organizationId, memberId) pair (~2M rows total). Used by
org_page_contributors.pipe for cheap request-time lookups instead of scanning the
full activityRelations cleaned bucket union (~744M rows) per request.

SCHEMA >
`organizationId` String,
`memberId` String,
`contributionCount` UInt64,
`computedAt` DateTime

ENGINE ReplacingMergeTree
ENGINE_SORTING_KEY organizationId, memberId
ENGINE_VER computedAt
108 changes: 73 additions & 35 deletions services/libs/tinybird/pipes/org_page_contributors.pipe
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
DESCRIPTION >
Top contributors for a given organization leaderboard.
Returns members sorted by contribution count within the specified date range.
When no startDate/endDate is given (the dominant traffic shape), reads precomputed
per-(organizationId, memberId) counts from org_page_contributors_copy_ds (keyed
lookup) instead of scanning the full activityRelations cleaned bucket union
(~744M rows) per request. Date-filtered requests fall back to the full scan.

TAGS "Organization page"

Expand All @@ -14,49 +18,83 @@ SQL >
NODE org_page_contributors_activity_aggregates
SQL >
%
{% set use_precomputed = 1 %}
{% if defined(startDate) %} {% set use_precomputed = 0 %} {% end %}
{% if defined(endDate) %} {% set use_precomputed = 0 %} {% end %}
{% if Boolean(count, false) %}
SELECT count(distinct memberId)
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %}
AND timestamp
>= {{ DateTime(startDate, description="Filter activity timestamp after") }}
{% end %}
{% if defined(endDate) %}
AND timestamp < {{ DateTime(endDate, description="Filter activity timestamp before") }}
{% end %}
{% if use_precomputed == 1 %}
SELECT count()
FROM org_page_contributors_copy_ds
WHERE organizationId = (SELECT id FROM org_slug_lookup)
{% else %}
SELECT count(distinct memberId)
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %}
AND timestamp
>= {{ DateTime(startDate, description="Filter activity timestamp after") }}
{% end %}
{% if defined(endDate) %}
AND timestamp
< {{ DateTime(endDate, description="Filter activity timestamp before") }}
{% end %}
{% end %}
{% else %}
SELECT
memberId,
count() as "contributionCount",
ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (), 2) as "contributionPercentage"
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %}
AND timestamp
>= {{ DateTime(startDate, description="Filter activity timestamp after") }}
{% end %}
{% if defined(endDate) %}
AND timestamp < {{ DateTime(endDate, description="Filter activity timestamp before") }}
{% end %}
GROUP BY memberId
ORDER BY contributionCount DESC, memberId DESC
LIMIT {{ Int32(limit, 10) }}
OFFSET {{ Int32(offset, 0) }}
{% if use_precomputed == 1 %}
SELECT
memberId,
contributionCount,
ROUND(
contributionCount * 100.0 / SUM(contributionCount) OVER (), 2
) as "contributionPercentage"
FROM org_page_contributors_copy_ds
WHERE organizationId = (SELECT id FROM org_slug_lookup)
ORDER BY contributionCount DESC, memberId DESC
LIMIT {{ Int32(limit, 10) }}
OFFSET {{ Int32(offset, 0) }}
{% else %}
SELECT
memberId,
count() as "contributionCount",
ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (), 2) as "contributionPercentage"
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %}
AND timestamp
>= {{ DateTime(startDate, description="Filter activity timestamp after") }}
{% end %}
{% if defined(endDate) %}
AND timestamp
< {{ DateTime(endDate, description="Filter activity timestamp before") }}
{% end %}
GROUP BY memberId
ORDER BY contributionCount DESC, memberId DESC
LIMIT {{ Int32(limit, 10) }}
OFFSET {{ Int32(offset, 0) }}
{% end %}
{% end %}

NODE org_page_contributors_leaderboard
SQL >
%
{% set use_precomputed = 1 %}
{% if defined(startDate) %} {% set use_precomputed = 0 %} {% end %}
{% if defined(endDate) %} {% set use_precomputed = 0 %} {% end %}
{% if Boolean(count, false) %}
SELECT count(distinct memberId) as count
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %} AND timestamp >= {{ DateTime(startDate) }} {% end %}
{% if defined(endDate) %} AND timestamp < {{ DateTime(endDate) }} {% end %}
{% if use_precomputed == 1 %}
SELECT count() as count
FROM org_page_contributors_copy_ds
WHERE organizationId = (SELECT id FROM org_slug_lookup)
{% else %}
SELECT count(distinct memberId) as count
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %} AND timestamp >= {{ DateTime(startDate) }} {% end %}
{% if defined(endDate) %} AND timestamp < {{ DateTime(endDate) }} {% end %}
{% end %}
{% else %}
SELECT
m.id,
Expand Down
17 changes: 17 additions & 0 deletions services/libs/tinybird/pipes/org_page_contributors_copy_pipe.pipe
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
DESCRIPTION >
Nightly copy pipe that precomputes all-time contribution counts per
(organizationId, memberId) for the org page contributors leaderboard.

TAGS "Organization page"

NODE org_page_contributors_copy_pipe_data
SQL >
SELECT organizationId, memberId, count() AS contributionCount, now() AS computedAt
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE organizationId != ''
GROUP BY organizationId, memberId

TYPE COPY
TARGET_DATASOURCE org_page_contributors_copy_ds
COPY_MODE replace
COPY_SCHEDULE 50 3 * * *

@cursor cursor Bot Aug 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COPY schedule collides overnight

High Severity

COPY_SCHEDULE 50 3 * * * overlaps activityRelations_by_member_copy (starts 35 3, ~22 min over a full activity scan) and spills into the 4 AM collection-bucket band. This full scan of activityRelations_deduplicated_cleaned_bucket_union (~744M rows) can push concurrent copy jobs over the account quota of 12, leaving pipes permanently queued and emptying downstream datasources.

Fix in Cursor Fix in Web

Triggered by learned rule: Tinybird COPY pipe schedules must not collide — account concurrent quota is 12

Reviewed by Cursor Bugbot for commit 0686845. Configure here.

Loading