Skip to content

Adaptive auto-throttle (AIMD) — steer rate by target latency - #4

Merged
NitinKumar004 merged 6 commits into
mainfrom
feature/adaptive-throttle
Aug 9, 2026
Merged

Adaptive auto-throttle (AIMD) — steer rate by target latency#4
NitinKumar004 merged 6 commits into
mainfrom
feature/adaptive-throttle

Conversation

@NitinKumar004

Copy link
Copy Markdown
Contributor

What this adds

Adaptive auto-throttle. Turn it on for a task and Marathon watches the target database's latency and adjusts the rate automatically — backing off when the DB is under load and easing back up when it recovers, using the same additive-increase / multiplicative-decrease law TCP uses for congestion control. Your rate becomes the ceiling.

This is the "polite to production" capability: instead of guessing a safe fixed rate, Marathon finds it and re-finds it as conditions change.

How it works

Before each batch (rate-limited to one probe per 250ms), the worker times a trivial round-trip to the target (SELECT 1) as a health signal. That latency feeds the AIMD controller, which sets the token-bucket rate within [min, ceiling]. When the DB is busy even the probe slows, so the controller reduces the rate; when it's healthy, the rate ramps back toward the ceiling.

  • Works in solo and fleet mode (same controller in both the sequential runner and the lease worker).
  • The AIMD controller was already implemented and unit-tested; this PR wires it into the workers behind a per-task adaptive flag (migration 011).
  • The dashboard speed slider becomes the ceiling for adaptive runs, and still applies live.

Evidence

  • TestIntegration_AdaptiveRunCompletes — an adaptive run applies every row exactly once (the feedback loop never breaks correctness).
  • TestIntegration_AdaptiveBacksOffUnderLatency — with ~80ms probe latency injected, the controller drives the rate well below its ceiling, and the run still completes correctly.
  • Existing AIMD unit tests cover the control law (back-off on spike, gradual recovery, ceiling/floor clamping).

Screenshot

Captured with headless Chrome. The run console shows the AUTO throttle indicator and "ceiling" wording:

Adaptive run console

Notes

  • The probe is a cheap SELECT 1; on a busy DB it slows along with everything else, which is exactly the signal we want. Fleet mode could later use replication lag as an additional signal.
  • Opt-in per task via a checkbox in the new-operation wizard ("Adaptive auto-throttle"). Existing tasks default to off — no behavior change.

@NitinKumar004
NitinKumar004 merged commit 0c866c2 into main Aug 9, 2026
3 checks passed
@NitinKumar004
NitinKumar004 deleted the feature/adaptive-throttle branch August 9, 2026 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant