Run auto-standby operations on bounded workers

[sjmiller609]

Problem

The auto-standby controller executes standby operations synchronously inside its single Run event loop. A standby pauses the VM and writes its full memory snapshot to disk, which can take fractions of a second per standby.

• standby throughput is capped at one operation at a time per host
• while a standby is running, instance lifecycle events, conntrack events, and other timer fires all queue behind it — idle countdowns for unrelated instances start late
• when many instances go idle together, tail queue wait grows unboundedly, and callers waiting on a Standby state transition can time out on instances that are perfectly healthy

A secondary issue: a standby attempt that fails because the instance was deleted re-arms the idle timer and retries forever until the delete event drains through the (congested) loop.

Change

• handleStandbyTimer now marks the instance standby-requested and hands the StandbyInstance call to a goroutine gated by a semaphore, so the event loop never blocks on snapshot IO. Follows the same shape as the existing firecracker_max_concurrent_restores cap on the restore side.
• New config auto_standby.max_concurrent (default 16) bounds concurrent standby operations per host. Demand above the cap queues; max_concurrent: 1 reproduces the old serialized throughput while still keeping the loop unblocked.
• Inbound activity observed while an attempt is queued clears standbyRequested, and the worker re-validates state after acquiring a slot, cancelling the attempt. The same flag deduplicates repeat timer fires for one instance.
• The store adapter translates instances.ErrNotFound into autostandby.ErrInstanceNotFound; the controller drops tracking state for such instances instead of re-arming the timer.
• Run cancels and waits for in-flight workers on exit.
• New gauge hypeman_auto_standby_standby_in_flight for observing concurrency; startup log includes the configured cap.

Testing

• go test -race ./lib/autostandby/ ./lib/providers/ ./cmd/api/config/ passes. New tests cover: concurrency never exceeds the cap, queued attempts cancelled by inbound activity, duplicate timer fires deduplicate, and not-found errors drop state without re-arming.
• End-to-end: TestAutoStandbyCloudHypervisorActiveInboundTCP (the manual-only e2e that CI skips, HYPEMAN_RUN_AUTO_STANDBY_E2E=1) passes on this branch on a KVM-capable Linux host: a real VM stays Running while an inbound TCP connection is open, and reaches Standby via the new async worker path ~1.3s after the idle timer fires. Note it runs a single instance, so the semaphore is never contended — concurrency behavior is covered by the unit tests above.
• go build ./... passes; wire regenerated via make generate-wire (includes a cosmetic identifier rename from the current wire version).
• The cmd/api/api integration suite fails in my environment on an image-pull timeout; it fails identically on unmodified main, so it is unrelated to this change.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes core VM standby timing and concurrency on the control plane; behavior is well-tested but mis-tuned max_concurrent or race fixes could affect when instances enter Standby or how load spikes hit disk.

Overview
Auto-standby no longer blocks its event loop on memory snapshot IO. When an idle timer fires, the controller marks the instance standby-requested and dispatches StandbyInstance to background workers gated by a semaphore (standbySlots), so conntrack, lifecycle events, and other timers keep processing.

New server config auto_standby.max_concurrent (default 16, validated in config load) wires through ProvideAutoStandbyController into ControllerOptions.MaxConcurrentStandbys. Extra demand queues until a slot frees; startup logging and docs describe the tradeoff between snapshot IO pressure and standby latency.

Standby lifecycle and races are tightened: standbyExecuting prevents duplicate workers across resyncs; inbound traffic or reconcile while queued clears standbyRequested and workers no-op after acquiring a slot; failed standbys skip re-arming the idle timer when connections became active mid-flight. instances.ErrNotFound maps to autostandby.ErrInstanceNotFound so deleted instances drop tracking instead of retrying forever. Run cancels worker context on exit and waits on standbyWG; persist errors during idle refresh log warnings but still arm timers.

Observability: gauges hypeman_auto_standby_standby_in_flight and hypeman_auto_standby_standby_queued plus expanded unit tests for concurrency, queue cancel, shutdown, and edge cases.

^{Reviewed by Cursor Bugbot for commit 90e7cfd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[yummybomb]

One thing to verify:

lib/autostandby/controller.go:729-740 — On StandbyInstance failure, the worker always sets a fresh idleSince, arms the idle timer, and persists idle runtime without checking activeInbound. With the loop unblocked, conntrack NEW can land during the snapshot; a late failure then overwrites that active state and arms a countdown while connections exist. That also persists a false IdleSince, which can shorten the effective idle window across restart. Sync path couldn’t hit this because the loop was blocked inside standby.

[hiroTamada]

reviewed the full diff — solid PR. the dispatch/execute split is careful, the standbyExecuting guard closes the resync double-dispatch race, and the tests are deterministic (channel-gated, no sleeps). nothing blocking, a few things worth a look:

Questions

• lib/autostandby/controller.go:192-196 — on shutdown, Run waits for in-flight StandbyInstance calls; is the snapshot write ctx-aware, or can Run hang in standbyWG.Wait() behind a slow snapshot?
• lib/autostandby/controller.go:703 — inbound activity arriving after standbyExecuting is set is now processed concurrently instead of queueing behind the standby; confirm pausing a VM with a just-established connection is safe (restore-on-traffic covers it?)

Suggestions

• lib/autostandby/metrics.go:73-79 — the in-flight gauge tops out at max_concurrent so it can't show queue pressure; consider also observing queued attempts so saturation is visible

Nits

• lib/autostandby/controller.go:697 — consider clearing standbyRequested when executeStandby bails on active inbound, so the invariant doesn't depend on every activeInbound writer also clearing the flag
• lib/autostandby/controller_test.go — no test covers the shutdown path (ctx cancelled while a worker waits on the semaphore / Run's cancel-then-wait ordering)

[sjmiller609]

@yummybomb good catch, fixed in b1d6dee — the failure path now checks activeInbound first: if inbound activity arrived during the attempt it only clears standbyRequested and leaves the countdown to the reconcile/destroy flow, so no timer is armed and no false IdleSince is persisted while connections exist. Covered by TestStandbyFailureWithMidFlightActivityDoesNotRearmIdle (asserts nil idleSince/nextStandbyAt and nil persisted IdleSince after a mid-flight NEW event + failure).

[sjmiller609]

@hiroTamada thanks — answers on the questions, and the suggestion + nits are in b1d6dee:

Shutdown hang in standbyWG.Wait(): cancelWorkers() runs before the wait (defer LIFO), the ctx flows into StandbyInstance, and the hypervisor snapshot calls underneath are ctx-aware HTTP, so cancellation propagates. Worst case is the tail of one in-flight snapshot — the same exposure the old synchronous loop had, where Run was blocked inside that very call. b1d6dee also adds a guard so a worker that wins a freed slot during teardown exits instead of starting a new snapshot, plus TestShutdownCancelsQueuedStandbyAndDrainsWorkers covering the cancel-then-wait ordering.

Pausing a VM with a just-established connection: unchanged semantics from the old code — once StandbyInstance starts there was never an abort path; the old loop just deferred the conntrack events until after the snapshot with the same outcome. The decision window up to the call start is re-checked, and a client that connects mid-snapshot is handled by restore-on-traffic at the ingress layer.

Queue-pressure gauge: added — hypeman_auto_standby_standby_queued counts dispatched attempts waiting for a slot (standbyRequested && !standbyExecuting), so saturation is visible alongside the in-flight gauge.

Bail-path flag clear: done — executeStandby now clears standbyRequested when it bails on active inbound, so the invariant no longer depends on every activeInbound writer also clearing it.

Shutdown test: added as above; ran 20x under -race to check for flakes.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit b1d6dee. Configure here.}