Skip to content

perf: precompute org page contributors leaderboard - #4501

Open
epipav wants to merge 1 commit into
mainfrom
perf/org-page-contributors-precompute
Open

perf: precompute org page contributors leaderboard#4501
epipav wants to merge 1 commit into
mainfrom
perf/org-page-contributors-precompute

Conversation

@epipav

@epipav epipav commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Optimizes org_page_contributors endpoint pipe with a nightly dedicated precompute copy pipe

Signed-off-by: anilb <epipav@gmail.com>
Copilot AI balanced review requested due to automatic review settings August 24, 2026 07:15
@cursor

cursor Bot commented Aug 24, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Analytics-only Tinybird change, but the copy pipe is a heavy nightly full-table aggregation and all-time leaderboard numbers will be up to ~24h stale until the first successful run.

Overview
Speeds up the default (all-time) org page contributors leaderboard by serving precomputed per-(organization, member) counts instead of scanning the full activityRelations cleaned bucket union (~744M rows) on every request.

Adds a nightly replace-mode copy pipe (03:50 UTC) that writes ~2M aggregated rows into org_page_contributors_copy_ds. org_page_contributors.pipe uses that table when startDate/endDate are omitted; date-filtered queries still scan the live union. All-time counts can lag by up to a day.

Reviewed by Cursor Bugbot for commit 0686845. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

@epipav
epipav requested review from gaspergrom and removed request for Copilot August 24, 2026 07:16
@epipav epipav self-assigned this Aug 24, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0686845. Configure here.

TYPE COPY
TARGET_DATASOURCE org_page_contributors_copy_ds
COPY_MODE replace
COPY_SCHEDULE 50 3 * * *

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COPY schedule collides overnight

High Severity

COPY_SCHEDULE 50 3 * * * overlaps activityRelations_by_member_copy (starts 35 3, ~22 min over a full activity scan) and spills into the 4 AM collection-bucket band. This full scan of activityRelations_deduplicated_cleaned_bucket_union (~744M rows) can push concurrent copy jobs over the account quota of 12, leaving pipes permanently queued and emptying downstream datasources.

Fix in Cursor Fix in Web

Triggered by learned rule: Tinybird COPY pipe schedules must not collide — account concurrent quota is 12

Reviewed by Cursor Bugbot for commit 0686845. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants