Skip to content

fix(worker): start background services once per role and run the DistributionScheduler - #1356

Open
Wittig18 wants to merge 1 commit into
RevoraOrg:masterfrom
Wittig18:fix/1297-1298-background-services
Open

Wittig18 wants to merge 1 commit into
RevoraOrg:masterfrom
Wittig18:fix/1297-1298-background-services

Conversation

@Wittig18

Copy link
Copy Markdown

Closes #1297
Closes #1298

Both issues change the same worker boot block in src/index.ts, so they're fixed together: #1298 extracts it, and #1297 adds the scheduler to the extracted version.

#1298: services started twice / in the wrong role

  • Removed the unconditional PayoutDriftDetector that started before the role checks and was never stopped.
  • New src/services/backgroundServices.ts: startBackgroundServices(roleConfig, factories) starts each batch service (audit purge, session compaction, payout drift, reconciliation, distribution scheduler) at most once, only under its own role flag, and returns a stop function for each. index.ts passes real factories, and tests pass spies.
  • Session compaction has its own sessionCompaction flag instead of reusing auditPurge. It's on for batch/all and off for api, so behaviour is the same as before.

#1297: DistributionScheduler never started

  • New distributionScheduler role flag: on for batch/all, off for api.
  • createDistributionSchedulerRuntime({ engine, revenueReportRepo, scheduledDistributionRepo, intervalMs, metrics }) in distributionScheduler.ts:
    • runs catchUpMissedWindows once at boot, then processPendingDistributions + processScheduledDistributions each tick
    • ticks are chained with setTimeout (the next is scheduled only after the previous finishes), so they never overlap
    • stop() cancels the pending timer and resolves after the in-flight tick finishes
    • interval comes from the option, else DISTRIBUTION_SCHEDULER_INTERVAL_MS, else 60 s
  • shutdown() now awaits the background stop functions before closePool(), so an in-flight tick isn't cut off.
  • Batch workers never run createApp, so the snapshot-backed balance provider is now a small exported createSnapshotBalanceProvider(). createApp and the worker build the same DistributionEngine.

Extra fix: index.ts used StellarTransactionVerifierImpl without importing it (tsc TS2304 on master), which would throw when batch/all starts the reconciliation scheduler. Added the import.

Tests

  • src/__tests__/backgroundServices.test.ts (new):
    • api starts zero services
    • batch and all start each exactly once, including exactly one drift detector
    • every started service has a stop function, for all roles
    • async stops are awaited
    • session compaction is gated on its own flag
    • a regression check on index.ts that the drift detector is only created through startBackgroundServices. On master there are two.
  • distributionScheduler.test.ts runtime block, with fake timers:
    • catch-up only once at boot
    • a due scheduled row is executed within one interval
    • a slow tick delays the next one (max concurrency 1)
    • stop() cancels the timer
    • stop() awaits the in-flight tick
    • a throwing tick doesn't stop the loop
  • workerRole.test.ts: the new flags across the role matrix.

Results:

  • backgroundServices + workerRole: 34/34 pass.
  • The runtime block: 6/6 pass.
  • The distributionScheduler.test.ts file has 10 holiday-calendar tests that already fail on master. Those are unchanged.
  • tsc --noEmit reports no new errors; the TS2304 above is gone. ESLint is clean on the new files.

🤖 Generated with Claude Code

…ributionScheduler

The boot block started a PayoutDriftDetector before the role checks and a
second one under roleConfig.payoutDrift, so api replicas ran drift
detection and batch/all ran it twice; the first was never stopped. Move
the batch services into startBackgroundServices(roleConfig, factories),
which starts each at most once under its own flag and returns its stop
function. Session compaction gets its own flag instead of reusing
auditPurge.

Nothing ever started the DistributionScheduler, so scheduled_distributions
rows stayed scheduled forever. Add a distributionScheduler role flag and
createDistributionSchedulerRuntime: catch-up once at boot, then process
pending and scheduled distributions on chained setTimeout ticks that never
overlap; stop() cancels the timer and awaits the in-flight tick, and
shutdown() awaits it before closing the pool. Also import the
StellarTransactionVerifierImpl the reconciliation start already used.

Closes RevoraOrg#1297
Closes RevoraOrg#1298

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant