backend/feat: add scheduler job lifecycle counter - #1469
Open
piyushKumar-1 wants to merge 2 commits into
Open
Conversation
Adds `scheduler_job_lifecycle_counter{job_type, status, version}`, a single
counter covering the whole life of a scheduler job.
Statuses: created, picked, completed, failed, rescheduled, retried,
retry_exhausted, duplicate.
Today only three of these points are instrumented, across three separately
named metrics with inconsistent labels (`stream_jobs_counter`,
`stream_jobs_failed_counter`, `scheduler_jobs_fail_counter` -- the last
labelled `scheduler_type` rather than `job_type`). Job creation, retry within
budget, reschedule and duplicate execution are not counted at all, so there is
no way to compare jobs entering the scheduler against jobs leaving it.
This is purely additive -- the existing counters are untouched, so current
dashboards keep working. Call sites land in the nammayatri scheduler lib.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
WalkthroughAdds scheduler job lifecycle and producer-stage Prometheus metrics. Updates the ChangesScheduler metrics
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Adds `scheduler_producer_stage_counter{stage, version}`, counting jobs as they
move through the producer: `picked_from_set`, `inserted_to_stream` and
`stream_insert_failed`.
Takes a count rather than incrementing by one, since the producer moves jobs in
batches. No `job_type` label -- the producer handles opaque encoded job blobs
and does not parse them, so tagging by type would mean decoding every job on
the hot path.
Together with scheduler_job_lifecycle_counter this makes the whole path
observable end to end:
created -> picked_from_set -> inserted_to_stream -> dequeued -> picked
which localises a stall to a specific hop instead of leaving it to be inferred.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
piyushKumar-1
added a commit
to nammayatri/nammayatri
that referenced
this pull request
Aug 11, 2026
Fills in the three hops between a job being scheduled and a job executing, so a
stall can be localised to a hop instead of inferred.
Producer (`Producer/Flow.hs`), via scheduler_producer_stage_counter:
- `picked_from_set` -- jobs read out of the scheduled sorted set
- `inserted_to_stream` -- XADD onto the stream succeeded
- `stream_insert_failed` -- XADD threw
The XADD counting is deliberate: those writes are forked and the caller does
not wait for them, yet it goes on to advance the producer watermark and
zRemRangeByScore the source range. A failing XADD therefore drops the job
permanently. Previously that was silent; now it is counted. (Making the write
synchronous before advancing the watermark is the actual fix and is left for a
separate change.)
Allocator (`Handler.hs`), via scheduler_job_lifecycle_counter:
- `dequeued` -- jobs read off the stream, counted before the blacklist filter
and before any per-job lock
`dequeued - picked` is then jobs lost to lock contention or blacklisting, which
was previously invisible.
Full path: created -> picked_from_set -> inserted_to_stream -> dequeued ->
picked -> terminal.
Depends on nammayatri/shared-kernel#1469.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/mobility-core/src/Kernel/Tools/Metrics/CoreMetrics/Types.hs`:
- Around line 119-126: Add default implementations for
incrementSchedulerJobLifecycleCounter and addSchedulerProducerStageCount in
CoreMetrics so existing external instances remain source-compatible. Ensure each
default is a no-op with the same signature, leaving custom implementations free
to override them.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1a16fe01-907f-4283-9bfc-2274fdf4546e
📒 Files selected for processing (5)
lib/mobility-core/src/Kernel/Mock/App.hslib/mobility-core/src/Kernel/Tools/Metrics/CoreMetrics.hslib/mobility-core/src/Kernel/Tools/Metrics/CoreMetrics/Types.hslib/mobility-core/src/Kernel/Types/Flow.hslib/mobility-core/test/src/APIExceptions.hs
🚧 Files skipped from review as they are similar to previous changes (2)
- lib/mobility-core/test/src/APIExceptions.hs
- lib/mobility-core/src/Kernel/Types/Flow.hs
Comment on lines
+119
to
+126
| -- | @incrementSchedulerJobLifecycleCounter jobType status@ -- record a job | ||
| -- lifecycle transition. See "Lib.Scheduler.Metrics" for the status values. | ||
| incrementSchedulerJobLifecycleCounter :: Text -> Text -> m () | ||
|
|
||
| -- | @addSchedulerProducerStageCount stage n@ -- record that @n@ jobs passed | ||
| -- through a producer pipeline stage. Takes a count because the producer | ||
| -- moves jobs in batches. | ||
| addSchedulerProducerStageCount :: Text -> Int -> m () |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n -P \
'instance\b.*\bCoreMetrics\b|\bCoreMetrics\b.*\bwhere|incrementSchedulerJobLifecycleCounter|addSchedulerProducerStageCount' \
--glob '*.hs' .Repository: nammayatri/shared-kernel
Length of output: 3556
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- CoreMetrics class and methods ---'
sed -n '90,140p' lib/mobility-core/src/Kernel/Tools/Metrics/CoreMetrics/Types.hs
printf '%s\n' '--- All CoreMetrics references in Haskell files ---'
rg -n -C 2 '\b(CoreMetrics|incrementSchedulerJobLifecycleCounter|addSchedulerProducerStageCount)\b' \
--glob '*.hs' lib/mobility-core
printf '%s\n' '--- Package metadata and compatibility wording ---'
rg -n -i 'coremetrics|breaking change|backward compat|compatib|version' \
--glob 'package.yaml' --glob '*.cabal' --glob 'README*' --glob 'CHANGELOG*' --glob '*.md' \
lib/mobility-core .github 2>/dev/null || trueRepository: nammayatri/shared-kernel
Length of output: 50381
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- CoreMetrics module header and class ---'
sed -n '1,135p' lib/mobility-core/src/Kernel/Tools/Metrics/CoreMetrics/Types.hs
printf '%s\n' '--- CoreMetrics instance declarations ---'
rg -n -U -P 'instance(?s:.{0,300}?)\bCoreMetrics\b(?s:.{0,100}?)\bwhere\b' \
--glob '*.hs' lib/mobility-core \
| rg -n '(^|:)instance|CoreMetrics|incrementSchedulerJobLifecycleCounter|addSchedulerProducerStageCount|where'
printf '%s\n' '--- Package metadata files ---'
git ls-files lib/mobility-core | rg '(^|/)(package\.yaml|[^/]+\.cabal|CHANGELOG[^/]*|README[^/]*)$' || true
printf '%s\n' '--- Public module/export references ---'
rg -n -C 2 'Kernel\.Tools\.Metrics\.CoreMetrics\.Types|CoreMetrics' \
lib/mobility-core/package.yaml lib/mobility-core/*.cabal lib/mobility-core/src/Kernel/Tools/Metrics/CoreMetrics.hs \
2>/dev/null || trueRepository: nammayatri/shared-kernel
Length of output: 27849
Preserve compatibility for external CoreMetrics instances. CoreMetrics (..) is publicly exported, and both new methods lack defaults. Downstream instances must implement both methods. Add default implementations, or document the migration and release this as a breaking change.
🧰 Tools
🪛 GitHub Actions: CI / 0_nix-ci.txt
[error] 124-124: treefmt/ormolu formatting check failed because the formatter modified this file by adding a blank line. Run the formatter and commit the resulting change.
🪛 GitHub Actions: CI / nix-ci
[error] 124-124: treefmt/ormolu formatting check failed because the hook modified this file by adding a blank line. Run the formatter and commit the resulting changes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/mobility-core/src/Kernel/Tools/Metrics/CoreMetrics/Types.hs` around lines
119 - 126, Add default implementations for incrementSchedulerJobLifecycleCounter
and addSchedulerProducerStageCount in CoreMetrics so existing external instances
remain source-compatible. Ensure each default is a no-op with the same
signature, leaving custom implementations free to override them.
Source: Coding guidelines
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
scheduler_job_lifecycle_counter{job_type, status, version}— one counter covering the whole life of a scheduler job.Statuses:
created·picked·completed·failed·rescheduled·retried·retry_exhausted·duplicateWhy
Only three of these points are instrumented today, spread across three separately named metrics with inconsistent labels:
stream_jobs_counterstream_jobs_failed_counterscheduler_jobs_fail_counter(labelledscheduler_type, notjob_type)Because job creation isn't counted, there's no way to compare jobs entering the scheduler against jobs leaving it. That comparison is exactly the signal that would have surfaced the 2026-08-11 allocator stall directly, independent of the producer that caused it:
Notes
SendSearchRequestToDriver). The older call sites applyshowto an already-Textvalue, which is why theirjob_typelabels carry embedded quotes (Executor_"SendSearchRequestToDriver"). Not changed here to avoid breaking existing queries.CoreMetricsinstances:FlowR r,MockM e, and theIOinstance in the test suite.Testing
mobility-corelibrary and its test suite build clean.Downstream call sites live in nammayatri (
Lib/Scheduler/{Metrics,Environment,ScheduleJob,Handler}.hs). That branch was validated against this exact patch applied to the currently pinned rev —schedulerlib,driver-offer-allocator(1824 modules) andrider-app+producer(1617 modules) all build and link.Merge this first, then the nammayatri PR bumps
flake.lockto pick it up.🤖 Generated with Claude Code
Summary by CodeRabbit