Cap prod app node pool at 3 and stop memory-based autoscaling - #1065
Merged
Conversation
Prod EKS raised an IP-exhaustion health alert (issue #1063). Root cause was the main-app HPA scaling on memory: main-app memory grows with pod uptime rather than load, so the HPA kept adding replicas chasing it, which added nodes, which reserved subnet IPs and exhausted the /24 in us-east-1b. Changes: - app-hpa: remove the memory metric (scale on CPU only) and set replicas to min 2 / max 4 so the pod count can't overrun the node pool. - terraform.tfvars: cap ng_prod_v2 at min 2 / max 3 (was max 8) so the node pool can't scale up unbounded. Agreed with the platform owner. Memory request/limit left unchanged so the memory growth stays visible and reproducible. The underlying app memory work is tracked separately by the application team.
arielr-lt
force-pushed
the
fix/prod-main-app-mem-rightsizing
branch
from
August 14, 2026 14:30
04186c6 to
5cfa10e
Compare
arielr-lt
changed the title
rohit-joy
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Prod EKS cluster
ce-registry-eksraised an IP-exhaustion health alert (#1063). Root cause was a feedback loop, not a networking capacity problem.Root cause
The
main-appHPA scaled on memory utilization. Sincemain-appmemory grows with pod uptime rather than with load, the HPA kept adding replicas chasing that growth → cluster-autoscaler added nodes → each node reserved a VPC-CNI warm pool of IPs → the/24private subnet inus-east-1bdropped to ~8 free addresses → EKS health alert.Changes
app-hpa.yamlminReplicas 2,maxReplicas 4terraform.tfvarsng_prod_v2capmin 2 / max 3.Memory request/limit are left unchanged (3560Mi / 8Gi) so the app's memory growth stays visible and reproducible rather than being masked. That app-side work is tracked separately by the application team.
Scope
Stabilization to keep the cluster patchable ahead of the 2026-08-19 EKS platform patch. Relates to #1063.