Adaptive auto-throttle (AIMD) — steer rate by target latency - #4
Merged
Conversation
… when a task opts in
…s, row/scrollbar hover were hardcoded light
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Adaptive auto-throttle. Turn it on for a task and Marathon watches the target database's latency and adjusts the rate automatically — backing off when the DB is under load and easing back up when it recovers, using the same additive-increase / multiplicative-decrease law TCP uses for congestion control. Your rate becomes the ceiling.
This is the "polite to production" capability: instead of guessing a safe fixed rate, Marathon finds it and re-finds it as conditions change.
How it works
Before each batch (rate-limited to one probe per 250ms), the worker times a trivial round-trip to the target (
SELECT 1) as a health signal. That latency feeds the AIMD controller, which sets the token-bucket rate within [min, ceiling]. When the DB is busy even the probe slows, so the controller reduces the rate; when it's healthy, the rate ramps back toward the ceiling.adaptiveflag (migration 011).Evidence
TestIntegration_AdaptiveRunCompletes— an adaptive run applies every row exactly once (the feedback loop never breaks correctness).TestIntegration_AdaptiveBacksOffUnderLatency— with ~80ms probe latency injected, the controller drives the rate well below its ceiling, and the run still completes correctly.Screenshot
Captured with headless Chrome. The run console shows the AUTO throttle indicator and "ceiling" wording:
Notes
SELECT 1; on a busy DB it slows along with everything else, which is exactly the signal we want. Fleet mode could later use replication lag as an additional signal.