server: reserve cluster capacity for HA failover (HA admission control) - #14042
Open
nagaboinaramgopal wants to merge 1 commit into
Open
server: reserve cluster capacity for HA failover (HA admission control)#14042nagaboinaramgopal wants to merge 1 commit into
nagaboinaramgopal wants to merge 1 commit into
Conversation
Adds cluster.ha.failover.capacity.reservethreshold (default 1.0 = disabled). When set below 1.0, FirstFitPlanner also excludes any cluster whose allocated+requested cpu/memory would cross the reserve from new deployments, so headroom stays free for HA-triggered restarts. The threshold is per-cluster scope aware.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds optional HA admission control: an operator can reserve a fraction of each cluster's CPU/memory so new deployments leave headroom for HA-triggered restarts when a host fails.
A new cluster-scoped setting
cluster.ha.failover.capacity.reservethreshold(Float, default 1.0 = disabled) controls it. Below 1.0, FirstFitPlanner excludes any cluster whose allocated + requested CPU or memory would cross that fraction from new deployments, using the samelistClustersCrossingThresholdpath the capacity disable threshold already uses. HA restarts are not subject to the reserve, so the headroom is available exactly when failover needs it. Off by default, so existing behaviour is unchanged until an operator opts in per cluster.For example, 0.8 keeps about 20% of a cluster's CPU and memory free for failover. It is meant to be set below the corresponding
cluster.*.allocated.capacity.disablethreshold.Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
How Has This Been Tested?
Added unit tests in
FirstFitPlannerTest:checkHAFailoverReserveDisabledByDefault: at the default 1.0 the cluster selection is unchanged.checkHAFailoverReserveExcludesClusterOnDeploy: a cluster whose allocation would cross the reserve is excluded from a new deployment.Also built the standard packages and deployed on a KVM advanced zone.