feat(webapp,run-engine): queue metrics and health dashboard by ericallam · Pull Request #4131 · triggerdotdev/trigger.dev

ericallam · 2026-07-03T08:23:47Z

Summary

Adds per-queue observability to the Queues page: depth (backlog), throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and the scheduling delay (how long runs wait between becoming eligible and actually starting). Each queue shows health at a glance in the list, plus a per-queue detail page with charts, so you can answer "does this queue have enough concurrency to keep up?".

Both the data collection and the dashboard are off by default and gated independently: metric emission is a global switch, and the dashboard is turned on per organization. With both off, the Queues page is unchanged.

Design

Queue operations emit two kinds of signal. Gauges (depth, running, limit, throttled) are read inside the same Redis script that performs the enqueue or dequeue, so the reading is atomic, and returned on the script's reply for the app to forward. Counters (enqueued, started, completed) are cumulative odometers, so a dropped reading self-heals: the next one restates the running total. Both land on one Redis stream on a dedicated metrics instance (falling back to the run queue's Redis when self-hosting), drain through a consumer into ClickHouse (raw, a 10-second-bucket materialized view, and a 30-day aggregate), and the dashboards read the aggregate. The run queue's own Redis carries no metrics stream.

The one change that is live the moment this deploys, independent of both flags, is the enqueue/dequeue script reply shape: those scripts now return a 2-tuple so the gauge reading can ride back to the app. That path is exercised on every queue op, so it is the part of run-engine worth the closest review.

changeset-bot · 2026-07-03T08:23:52Z

⚠️ No Changeset found

Latest commit: 0dfaec2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

coderabbitai · 2026-07-03T08:25:40Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

This PR adds queue-metrics ingestion, storage, query, and UI support. It introduces a Redis/ClickHouse metrics pipeline package, ClickHouse queue-metrics tables and query helpers, run-queue emission hooks, gap-filling support in TSQL, and new webapp admin, dashboard, list, and detail routes. It also adds environment and feature-flag gating, seed tooling, and tests across the pipeline and query layers.

Related PRs: None found.

Suggested labels: enhancement, area: webapp, area: run-engine, area: internal-packages

Suggested reviewers: ericallam, matt-aitken

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description is missing required template sections, including Closes `#issue`, checklist, Testing, Changelog, and Screenshots.	Add the template sections and fill them out, starting with Closes #, the checklist, testing steps, changelog, and screenshots.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: queue metrics and health dashboard for webapp/run-engine.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/queue-metrics-and-health

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

pkg-pr-new · 2026-07-04T08:31:49Z

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@c6de9d4

trigger.dev

npm i https://pkg.pr.new/trigger.dev@c6de9d4

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@c6de9d4

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@c6de9d4

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@c6de9d4

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@c6de9d4

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@c6de9d4

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@c6de9d4

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@c6de9d4

commit: c6de9d4

devin-ai-integration

Devin Review found 0 new potential issues.

View 1 additional finding in Devin Review.

devin-ai-integration

Devin Review found 0 new potential issues.

…peline

…signals Gauges are read inside the enqueue/dequeue Lua and returned on the script reply as a 2-tuple; counters are cumulative odometers. The run-queue Redis carries no metrics stream of its own.

…witch

The simulator's --reset only cleared the raw and 10s tables, leaving stale rows in the 5m and env rollups. It also force-merges the rollups after seeding so current-value widgets read cleanly.

Counter events now emit per queue and op odometer readings with a seeded zero baseline, matching the production emitter, so throughput and started counts reconstruct from simulated data instead of reading zero. Scenario switches prune the previous scenario's queues, a --project flag seeds each scenario into its own project for side-by-side design review, and a new many-queues scenario covers pagination and relevance ranking with one runaway queue, a busy head, a bursty middle, and a sparse tail. Adds --help.

A --usage flag stages plausible running counts in the local run-queue Redis for the seeded queues, so the list's Running column and the Allocation tab's usage bars have data without the run engine. Staged state is reconciled on every run: present with --usage, cleared without. Local Redis hosts only.

The tail query's exclusion list overwrote the search's name filter via object spread, so searching while sorted by activity showed unrelated queues past the ranked head. Combine the conditions with AND instead.

…ot ready Without a readiness guard, every fire-and-forget emit during a metrics Redis outage queued a command in ioredis's in-memory offline queue until rejection. Metrics are loss-tolerant by design, so drop instead; waitUntilReady() lets embedders await the initial connect.

The allocation view keeps manual limit edits, the review dialog, and bulk apply. The one-shot auto-balance button is removed (and the row locks whose only purpose was protecting queues from it); a policy-driven approach can replace it if rebalancing returns.

deltaSumTimestamp states are kept per queue, and merging them across queues silently returns wrong totals, on the dashboard and the public query API alike. Columns can now declare a mergeGroupKey, and the compiler rejects queries that merge such a column without grouping by that key or pinning it to a single value. The error names the column, explains the failure, and includes a corrected example query.

…e calls Short parameter lists on quantilesMerge and quantilesTDigestMerge do execute (the state layout is parameter independent, verified on both ClickHouse versions we run), but they rely on undocumented leniency and make the result-array indexes mean different quantiles per call site. Every merge now uses the stored four-quantile list with indexes re-pointed accordingly; returned values are unchanged.

The consumer retries a failed batch with the same insert deduplication token, and deduplicate_blocks_in_dependent_materialized_views re-runs the materialized views on a source-deduplicated insert, relying on each target table's dedup window to drop the duplicate. Only the raw table had one, so retries appended extra copies into every aggregate tier, silently inflating sums and quantiles. All three targets now set non_replicated_deduplication_window, with a regression test inserting the same batch three times.

… cluster slots, and stream caps Counter readings for names past the per-env cardinality cap are dropped instead of merging unrelated odometers under the overflow label (gauges still flow). The odometer key now shares the stream's shard hash tag so the INCR plus XADD script stays in one Cluster slot. The counter stream cap defaults lower when the stream shares the run-queue Redis. The per-bucket counter boundary undercount is documented on the delta columns.

…d live per-key breakdown CK queues now emit two extra gauge fields from the CK-path Lua scripts: the number of concurrency keys with queued runs (ZCARD of the ckIndex) and the head-of-line wait of the most-starved key (now minus the oldest ckIndex score). Both flow through the existing stream into new max-aggregated columns on the 10s and 5m tiers, and non-CK scripts keep the 7-field gauge shape. The queue detail page grows a concurrency-keys section for queues with CK activity: charts for backlogged keys and most-starved wait, plus a live per-key table (queued, running, oldest wait) read from the ckIndex zset, most starved first. Per-key history is intentionally not stored: key values are user-controlled, so the per-key dimension stays in Redis where it is bounded by the live backlog. The queue simulator gains a tenant-hotspot scenario that stages the CK gauge columns and a live ckIndex so the per-key table and charts render with data.

…trics history Queues that shard work with concurrencyKey get a per-key history tier. Counter events for CK runs advance a second per-key odometer and carry both readings on ONE stream entry (cum + ck/ckcum), so per-key attribution adds no stream volume; the consumer expands the entry into a base row and a per-key row. A new 10s AggregatingMergeTree tier keyed by (queue, concurrency_key) holds per-key enqueue/started/ack deltas, backlog/running maxes, and wait sums. Rows are activity-bound: a (queue, key, bucket) row exists only when that key had events, so user-controlled key cardinality cannot inflate the table (benchmarked at ~19 bytes per event with reads under 100ms in the worst shapes). The per-queue tiers stay exact: their counter and wait aggregations now consume only base rows, so per-key odometers never merge under one queue_name and waits are never double counted. A per-queue key limiter (default 10k) acts as a safety valve; on overflow the per-key row is dropped while the base row keeps per-queue counts exact. Per-key odometers use a short TTL, which cumulative counters make loss-free. The queue detail page gains a top-keys-by-backlog chart, a key table merging live state (queued, running, oldest wait) with range stats (started, peak backlog, mean delay), and click-through per-key drill-down charts. The new queue_metrics_by_key table is also queryable directly; its delta columns require grouping or pinning BOTH queue and concurrency_key, enforced by the compile-time merge guard which now supports compound keys.

…il page The queue detail page splits into Overview (the existing concurrency, backlog, delay, and throttle charts) and a Concurrency keys tab that holds all per-key content. The tab only appears for queues with key activity. Inside it, the grouped per-key charts now lead (backlog by key plus a new throughput-by-key chart that makes fair-share visible), followed by the two whole-queue health charts retitled to say what they aggregate (keys with queued runs count, most-starved key wait), then the key table and drill-down.

…ery surfaces TableSchema gains a hidden flag: hidden tables stay fully queryable through the engine (the concurrency-keys tab keeps working, tenancy and the merge guard still apply) but are excluded from the query editor autocomplete, the Query page schema and examples panels, the AI query generator context, and the schema API. queue_metrics_by_key is hidden until the per-key surface is settled.

…n re-enqueue The wait metric measured dequeue time minus the original trigger time even when a run re-entered the queue after a waitpoint, checkpoint, or pending version, so the whole wait or checkpoint duration showed up as scheduling delay. Re-enqueues now anchor to the re-enqueue time while first enqueues keep the trigger or delay anchor; queue ordering is unchanged, so re-enqueued runs keep their original position. Nacked runs never left the queue stint and keep the original anchor. Also replaces the :ck: suffix regexes on user-controlled queue names with indexOf slicing (identical semantics) to remove a polynomial regex flagged by code scanning.

Behind the per-org queue metrics UI flag: the task detail page links its queue to the queue detail view and shows live queued/running counts, delay p95, peak backlog, and a queue backlog chart; the run inspector links the queue and concurrency key, and queued, delayed, and pending-version runs get a "Waiting in queue" block with an at-limit callout and per-key counts. Adds optional concurrency-key params to the run engine queue reads.

…ask detail queue chart The run inspector's "Waiting in queue" section now leads with two borderless stat + last-30-minutes sparklines (backlog and scheduling delay), matching the inspector's divider-and-rows layout instead of nested cards. On the task detail page the queue backlog moves behind a tab in the activity card rather than sitting beside the runs-by-status chart, so it does not crowd the agent-task hero charts.

…ersion collision Main added 035_fix_error_display_derivation.sql, which collides with the queue-metrics migration that also landed as 035. goose versions by numeric prefix, so two 035 files break the migrator. Renumber the queue-metrics migration to 036 (it sorts after the error-display fix on a fresh database).

devin-ai-integration

Devin Review found 0 new potential issues.

## Queue pages design overhaul (frontend for the queue-metrics feature) Builds the dashboards on top of the queue-metrics pipeline (base branch). ### Queues list <img width="1440" height="788" alt="Screenshot 2026-07-20 at 01 02 55" src="https://github.com/user-attachments/assets/0b97dd6c-d05b-4dba-99c9-04266ec22cb3" /> ### Queue detail <img width="1440" height="1272" alt="tab-overview" src="https://github.com/user-attachments/assets/22b33f82-522d-4410-a477-fe5d2c2f7adf" /> <img width="1440" height="1532" alt="tab-keys" src="https://github.com/user-attachments/assets/b05619fa-93a8-43ef-8838-a6d7fc5d789b" /> ### Under the hood - Percent-based concurrency overrides (schema + migration + recalc on env limit change, capped at the env limit) — covered by unit tests - All recurring refresh hits ClickHouse only (15s blocks / 60s charts, paused in hidden tabs); Redis/PG serve first paint - Added `MetricsLayout` that is currently used for `/queues + $id`, `/agents + $id`, `/settings/usage` <img width="500" height="auto" alt="Screenshot 2026-07-20 at 01 07 31" src="https://github.com/user-attachments/assets/e8a798e7-deee-455d-be96-414f7d09a06c" />

…d-health # Conflicts: # apps/webapp/app/routes/_app.orgs.$organizationSlug.settings.usage/route.tsx

devin-ai-integration

Devin Review found 0 new potential issues.

The run inspector now shows a mini queue view while a run is waiting: Status (with a paused / at-limit warning), Concurrency and how long the run has waited, a p50/p95 scheduling-delay chart, and a link to the queue.

The four header chart tiles rendered even in the not-success states (engine-version upgrade, no tasks) and when the list filtered to empty, showing four empty 'No activity' cards above the blank state. Gate them on the same queue set the table shows.

devin-ai-integration

Devin Review found 0 new potential issues.

This comment was marked as resolved.

Sign in to view

ericallam marked this pull request as ready for review July 3, 2026 10:26

This comment was marked as resolved.

Sign in to view

ericallam force-pushed the feat/queue-metrics-and-health branch from a892684 to 9412bf5 Compare July 4, 2026 08:30

This comment was marked as resolved.

Sign in to view

ericallam force-pushed the feat/queue-metrics-and-health branch from 6432d9f to 3c67a0c Compare July 4, 2026 22:16

This comment was marked as resolved.

Sign in to view

ericallam closed this Jul 5, 2026

ericallam reopened this Jul 5, 2026

ericallam force-pushed the feat/queue-metrics-and-health branch from 1416ee6 to cbe5444 Compare July 5, 2026 12:44

This comment was marked as resolved.

Sign in to view

devin-ai-integration Bot reviewed Jul 5, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

devin-ai-integration Bot reviewed Jul 5, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

ericallam added 5 commits July 13, 2026 20:38

feat(metrics-pipeline): generic Redis-stream to ClickHouse metrics pi…

3f1731f

…peline

feat(clickhouse): queue metrics tables and read queries

33562f5

feat(run-engine): emit queue depth, throughput, and scheduling-delay …

a3a2434

…signals Gauges are read inside the enqueue/dequeue Lua and returned on the script reply as a 2-tuple; counters are cumulative odometers. The run-queue Redis carries no metrics stream of its own.

feat(webapp): queue metrics ingestion, admin controls, and emission s…

c206e12

…witch

feat(tsql): opt-in gap-fill for time-bucketed series

3581326

ericallam added 19 commits July 13, 2026 20:39

fix(webapp): include rollup tables in the queue metrics simulator reset

9f603ae

The simulator's --reset only cleared the raw and 10s tables, leaving stale rows in the 5m and env rollups. It also force-merges the rollups after seeding so current-value widgets read cleanly.

fix(webapp): keep the search filter on the ranked queue list's tail

9bd77e8

The tail query's exclusion list overwrote the search's name filter via object spread, so searching while sorted by activity showed unrelated queues past the ranked head. Combine the conditions with AND instead.

fix(tsql): satisfy the tenant column type in the merge guard test schema

74f1979

chore(webapp): remove trailing blank line flagged by oxfmt

61931ab

ericallam force-pushed the feat/queue-metrics-and-health branch from 6c0d3da to ddae31f Compare July 13, 2026 19:44

devin-ai-integration Bot reviewed Jul 13, 2026

View reviewed changes

matt-aitken approved these changes Jul 13, 2026

View reviewed changes

kathiekiwi and others added 2 commits July 21, 2026 19:29

Merge remote-tracking branch 'origin/main' into feat/queue-metrics-an…

6716463

…d-health # Conflicts: # apps/webapp/app/routes/_app.orgs.$organizationSlug.settings.usage/route.tsx

devin-ai-integration Bot reviewed Jul 21, 2026

View reviewed changes

feat(webapp): compact queue status block on waiting runs

c6de9d4

The run inspector now shows a mini queue view while a run is waiting: Status (with a paused / at-limit warning), Concurrency and how long the run has waited, a p50/p95 scheduling-delay chart, and a link to the queue.

This comment was marked as resolved.

Sign in to view

devin-ai-integration Bot reviewed Jul 22, 2026

View reviewed changes

Uh oh!

Uh oh!

Conversation

ericallam commented Jul 3, 2026

Summary

Design

Uh oh!

changeset-bot Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

coderabbitai Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

❌ Failed checks (1 warning)

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

pkg-pr-new Bot commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

changeset-bot Bot commented Jul 3, 2026 •

edited

Loading

coderabbitai Bot commented Jul 3, 2026 •

edited

Loading

pkg-pr-new Bot commented Jul 4, 2026 •

edited

Loading