feat: support topologySpreadConstraints for vector aggregators - #273
feat: support topologySpreadConstraints for vector aggregators#273stigglor wants to merge 2 commits into
Conversation
Aggregator replicas need to spread across availability zones, for failure isolation and to keep cross zone traffic down, and spec.affinity cannot express that. Preferred pod anti-affinity is only a scheduling score, so under a provisioner like Karpenter, which adds nodes to fit pending pods rather than to satisfy preferences, replicas still pack onto one node. Required anti-affinity has no count parameter, so it allows one pod per zone and leaves every replica above the zone count Pending, which rules it out for HPA driven scaling. VectorAggregator and ClusterVectorAggregator now take an optional spec.topologySpreadConstraints, passed to the aggregator pod template next to affinity, tolerations and schedulerName. With maxSkew and whenUnsatisfiable of ScheduleAnyway that expresses "spread evenly, but never block scheduling". The field lives on the aggregator spec rather than the shared one, since the agent is a DaemonSet that already places one pod per node. Both the Deployment and the StatefulSet paths are covered because they share one pod template, and the event collector inherits it the same way it already inherits affinity. Unset means no constraints, so existing aggregators are untouched. Closes kaasops#270
aa1ex
left a comment
There was a problem hiding this comment.
The passthrough is clean and it reaches everywhere it should: the Deployment, the StatefulSet and the event collector. An aggregator that leaves the field unset renders byte for byte as before, so the upgrade rolls nothing.
Two notes on the docs, left inline.
| template. Use it to keep replicas out of a single zone or node, for failure isolation and to | ||
| avoid paying for cross zone traffic on every event. |
There was a problem hiding this comment.
Does the cross zone half hold on its own? The aggregator Service is a plain ClusterIP one, and the operator sets neither spec.trafficDistribution nor the service.kubernetes.io/topology-mode annotation on it, so kube-proxy still picks an endpoint anywhere in the cluster regardless of zone. That makes an even spread a precondition for zone local routing rather than something that reduces cross zone traffic by itself. Does this assume topology aware routing is enabled separately? If so it may be worth stating as a prerequisite; otherwise the line could be narrowed to failure isolation.
There was a problem hiding this comment.
You are right, it does not hold on its own. The services are plain ClusterIP with no trafficDistribution and no topology-mode annotation, so nothing here makes routing zone local. Narrowed the opening to failure isolation and added a paragraph stating the routing prerequisite, since anyone reaching for zone spreading is probably chasing the traffic bill too.
spec.annotations is applied to the aggregator services, so service.kubernetes.io/topology-mode: Auto is settable today. spec.trafficDistribution is not, which would need a field of its own. Happy to open an issue for that if it looks worth having.
Setting any constraint drops the scheduler's built in defaults rather than adding to them, so a zone only example quietly gives up the node level spreading that came for free. Carry a hostname constraint alongside the zone one, and say why. Also narrow the cross zone traffic claim. The aggregator services are plain ClusterIP, so an even spread is a precondition for zone local routing rather than something that cuts cross zone traffic on its own, and the annotation that turns routing on is worth naming.
Adds an optional
spec.topologySpreadConstraintstoVectorAggregatorandClusterVectorAggregator, passed through to the aggregator pod template.Aggregator replicas need to spread across availability zones, for failure isolation and
to keep cross zone traffic down, and
spec.affinitycannot express that. Preferred podanti-affinity is only a scheduling score, so under a provisioner like Karpenter, which
adds nodes to fit pending pods rather than to satisfy preferences, replicas still pack
onto one node. Required anti-affinity has no count parameter, so it allows one pod per
zone and leaves every replica above the zone count Pending, which rules it out for HPA
driven scaling.
maxSkewwithwhenUnsatisfiable: ScheduleAnywayis the only way to say"spread evenly, but never block scheduling".
The field sits on the aggregator spec rather than the shared one, since the agent is a
DaemonSet that already places one pod per node. Both the Deployment and the StatefulSet
paths are covered because they share one pod template, and the event collector inherits
it the same way it already inherits
affinity. Unset means no constraints, so existingaggregators are untouched and no RBAC changes.
Note the selector needs
component: Aggregatoralongside the instance label. The eventcollector shares the instance label, so instance alone would count that pod toward the
per zone totals.
Closes #270