Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/guides/milvus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Milvus is an open-source vector database built to power embedding similarity sea
| External object storage support | ✓ |
| KubeDB-managed or externally managed etcd metadata storage | ✓ |
| Custom Configuration | ✓ |
| Monitoring using Prometheus and Grafana | ✓ |
| Monitoring using Prometheus | ✓ |
| Authentication & Authorization (TLS) | ✓ |
| Externally manageable Auth Secret (Rotate Auth) | ✓ |
| Persistent volume | ✓ |
Expand All @@ -44,7 +44,10 @@ KubeDB supports the following Milvus Versions.
- `2.6.7`
- `2.6.9`
- `2.6.11`
- `2.6.15`
- `2.6.15-gpu`
- `3.0.0`
- `3.0.0-gpu`

## User Guide

Expand Down
91 changes: 71 additions & 20 deletions docs/guides/milvus/autoscaler/compute/guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,21 +23,32 @@ This guide will show you how to use the `KubeDB` Autoscaler operator to autoscal
- [MilvusAutoscaler](/docs/guides/milvus/concepts/milvusautoscaler.md)
- [Compute Autoscaling Overview](/docs/guides/milvus/autoscaler/compute/overview.md)

- Install the **KubeDB Autoscaler** operator and a **metrics server** in your cluster — the VPA recommender needs metrics to produce recommendations.
- Install the **KubeDB Autoscaler** operator and a **metrics server** in your cluster — the autoscaler's built-in recommender needs metrics to produce recommendations.

```bash
$ kubectl get deploy metrics-server -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
metrics-server 1/1 1 1 5m
```

- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus.
- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`).

> Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/autoscaler/compute/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/autoscaler/compute/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs).

## Compute Autoscaling — Standalone Milvus

Deploy a standalone Milvus and wait until it is `Ready`. Then create a `MilvusAutoscaler` targeting the standalone `node`:
Deploy the [standalone quickstart](/docs/guides/milvus/quickstart/standalone.md) manifest and wait until it is `Ready`:

> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`.

```bash
$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/standalone.yaml
milvus.kubedb.com/milvus-standalone created

$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w
```

Then create a `MilvusAutoscaler` targeting the standalone `node`:

`compute-standalone.yaml`

Expand All @@ -53,60 +64,76 @@ spec:
compute:
node:
trigger: "On"
podLifeTimeThreshold: 1m
podLifeTimeThreshold: 10m
minAllowed:
cpu: 100m
memory: 256Mi
memory: 1Gi
maxAllowed:
cpu: 1000m
memory: 2Gi
resourceDiffPercentage: 10
resourceDiffPercentage: 20
controlledResources: ["cpu", "memory"]
opsRequestOptions:
apply: IfReady
timeout: 5m
```

Here,

- `minAllowed.memory: 1Gi` keeps the autoscaler from shrinking Milvus below what it needs. The generated OpsRequest sets the memory limit equal to the request, so a lower floor (for example `256Mi`) leaves no headroom and Milvus is OOM-killed under real load.
- `podLifeTimeThreshold: 10m` and `resourceDiffPercentage: 20` damp the autoscaler. Every OpsRequest it creates restarts the pods, and a freshly started Milvus briefly uses much more CPU. With a short `podLifeTimeThreshold` and a small `resourceDiffPercentage`, that startup spike immediately triggers another re-scale (and another restart). Waiting 10 minutes after a pod starts and ignoring differences below 20% prevents this.

```bash
$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/autoscaler/compute/yamls/compute-standalone.yaml
milvusautoscaler.autoscaling.kubedb.com/milvus-standalone-compute-autoscaler created
```

The autoscaler creates a `VerticalPodAutoscaler` (VPA) object. Once the VPA recommender produces a recommendation that differs from the current resources by more than `resourceDiffPercentage`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest`.
The autoscaler runs its own built-in recommender (it does not create a `VerticalPodAutoscaler` object). Once the recommender produces a recommendation that differs from the current resources by more than `resourceDiffPercentage`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest`.

> **Each autoscaler-created OpsRequest restarts the pods** of the role it scales (it uses `Restart` mode). For a standalone Milvus, that means a short outage every time the autoscaler re-scales.

```bash
$ kubectl get milvusautoscaler -n demo
NAME AGE
milvus-standalone-compute-autoscaler 59s
```

The autoscaler runs a VPA recommender (fed by the metrics server) and records the recommendation in its status. Once enough samples are collected, the `RecommendationProvided` condition becomes `True` and a target resource set is published:
The recommender (fed by the metrics server) records its recommendation in the autoscaler status. Once enough samples are collected, the `RecommendationProvided` condition becomes `True` and a target resource set is published. The status has this shape (values and the large `checkpoints` array elided):

```bash
$ kubectl get milvusautoscaler milvus-standalone-compute-autoscaler -n demo -o jsonpath='{.status}' | jq .
{
"checkpoints": [ ... ],
"conditions": [ ... ],
"vpas": [
{
"conditions": [
{ "type": "RecommendationProvided", "status": "True", "lastTransitionTime": "2026-06-30T18:40:17Z" }
{ "type": "RecommendationProvided", "status": "True", ... }
],
"recommendation": {
"containerRecommendations": [
{
"containerName": "milvus",
"lowerBound": { "cpu": "100m", "memory": "256Mi" },
"target": { "cpu": "143m", "memory": "256Mi" },
"uncappedTarget": { "cpu": "143m", "memory": "262144k" }
"lowerBound": { "cpu": ..., "memory": ... },
"target": { "cpu": ..., "memory": ... },
"uncappedTarget": { "cpu": ..., "memory": ... },
...
}
]
},
"ref": { "containerName": "milvus", "vpaObjectName": "milvus-standalone" }
"vpaName": "milvus-standalone"
}
]
}
```

Here the recommended `target` is `cpu: 143m` / `memory: 256Mi` (the standalone idles well below its `500m` request). Because the recommendation differs from the current request by more than `resourceDiffPercentage` (10%) and stays within `minAllowed`/`maxAllowed`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest`. This is recorded in the autoscaler status as a `CreateOpsRequest` condition:
To see just the recommended target:

```bash
$ kubectl get milvusautoscaler milvus-standalone-compute-autoscaler -n demo -o jsonpath='{.status.vpas[0].recommendation.containerRecommendations[0].target}'
```

An idle standalone uses far less CPU than its `500m` request, so the CPU target is lower, while the memory target is held at `minAllowed` (`1Gi`). When the recommendation differs from the current request by more than `resourceDiffPercentage` (20%) and stays within `minAllowed`/`maxAllowed`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest`. This is recorded in the autoscaler status as a `CreateOpsRequest` condition:

```bash
$ kubectl get milvusautoscaler milvus-standalone-compute-autoscaler -n demo \
Expand All @@ -122,7 +149,16 @@ The Ops-manager then applies the vertical scaling exactly as in the [vertical sc

## Compute Autoscaling — Distributed Milvus

For a distributed Milvus, the autoscaler is keyed by role (`proxy`, `mixcoord`, `datanode`, `querynode`, `streamingnode`). The sample below enables compute autoscaling for all five roles:
For a distributed Milvus, the autoscaler is keyed by role (`proxy`, `mixcoord`, `datanode`, `querynode`, `streamingnode`). The sample below enables compute autoscaling for all five roles.

Deploy the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md) manifest and wait until it is `Ready`:

```bash
$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/distributed.yaml
milvus.kubedb.com/milvus-cluster created

$ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w
```

`compute-distributed.yaml`

Expand All @@ -138,14 +174,14 @@ spec:
compute:
proxy:
trigger: "On"
podLifeTimeThreshold: 1m
podLifeTimeThreshold: 10m
minAllowed:
cpu: 100m
memory: 256Mi
memory: 1Gi
maxAllowed:
cpu: 1000m
memory: 2Gi
resourceDiffPercentage: 10
resourceDiffPercentage: 20
controlledResources: ["cpu", "memory"]
mixcoord: { ... }
datanode: { ... }
Expand All @@ -158,13 +194,28 @@ spec:

(Each role block is identical to the `proxy` block above; see the full file in the `yamls` folder.)

The behavior is identical to standalone, except a VPA object and resource recommendation are produced **per role**. When any role's recommendation differs from its current resources by more than `resourceDiffPercentage`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest` scoped to that role (see the [vertical scaling guide](/docs/guides/milvus/scaling/vertical-scaling/guide.md) for what that ops request looks like).
```bash
$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/autoscaler/compute/yamls/compute-distributed.yaml
milvusautoscaler.autoscaling.kubedb.com/milvus-compute-autoscaler created
```

The behavior is identical to standalone, except a resource recommendation is produced **per role** (one `status.vpas[]` entry per role, for example `vpaName: milvus-cluster-proxy`). When any role's recommendation differs from its current resources by more than `resourceDiffPercentage`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest` scoped to that role (see the [vertical scaling guide](/docs/guides/milvus/scaling/vertical-scaling/guide.md) for what that ops request looks like). The OpsRequests run one at a time, and each restarts the pods of its role.

## Cleaning up

The OpsRequests created by a `MilvusAutoscaler` are owned by it, so deleting the autoscaler also deletes them, **including one that is still running**. An OpsRequest deleted mid-flight can leave the database paused or with pods that don't match its spec. Before deleting the autoscaler, wait until none of its OpsRequests is `Progressing`:

```bash
$ kubectl get milvusopsrequest -n demo
```

Then clean up:

```bash
$ kubectl delete milvusautoscaler -n demo --all
$ kubectl delete milvus.kubedb.com -n demo milvus-standalone
$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge"
$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge"
$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster
$ kubectl delete ns demo
```

Expand Down
16 changes: 9 additions & 7 deletions docs/guides/milvus/autoscaler/compute/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ This guide will give an overview on how the KubeDB Autoscaler operator autoscale

## How Compute Autoscaling Works

A `MilvusAutoscaler` of type `compute` watches the resource usage of the Milvus pods (via the metrics server / VPA recommender) and, when usage drifts far enough from the requested resources, it creates a `VerticalScaling` `MilvusOpsRequest` to right-size the pods.
A `MilvusAutoscaler` of type `compute` watches the resource usage of the Milvus pods (via the metrics server and the autoscaler's built-in recommender) and, when usage drifts far enough from the requested resources, it creates a `VerticalScaling` `MilvusOpsRequest` to right-size the pods.

`spec.compute` is keyed by component:

Expand All @@ -39,11 +39,11 @@ spec:
compute:
node: # or proxy/mixcoord/datanode/querynode/streamingnode
trigger: "On"
podLifeTimeThreshold: 1m
resourceDiffPercentage: 10
podLifeTimeThreshold: 10m
resourceDiffPercentage: 20
minAllowed:
cpu: 100m
memory: 256Mi
memory: 1Gi
maxAllowed:
cpu: 1000m
memory: 2Gi
Expand All @@ -54,15 +54,17 @@ spec:
- **`minAllowed`/`maxAllowed`** — the bounds the autoscaler stays within.
- **`resourceDiffPercentage`** — how far current resources must drift from the recommendation before an ops request is created.
- **`podLifeTimeThreshold`** — minimum pod age before it is considered for scaling.

Every autoscaler-created OpsRequest restarts the pods it scales, and a freshly started Milvus briefly uses much more CPU. `podLifeTimeThreshold: 10m` and `resourceDiffPercentage: 20` keep that startup spike from triggering another re-scale right after a restart. Keep `minAllowed.memory` at `1Gi` or more: the autoscaler sets the memory limit equal to the request, and Milvus needs more than `256Mi`.
- **`controlledResources`** — which resources are managed.

The flow is:

1. A user creates a `MilvusAutoscaler` with `spec.compute`.
2. The autoscaler creates a `VerticalPodAutoscaler` (VPA) object which produces resource recommendations.
2. The autoscaler's built-in recommender produces resource recommendations and records them in the autoscaler's `status.vpas` (no `VerticalPodAutoscaler` object is created).
3. When the recommendation differs from the current resources by more than `resourceDiffPercentage`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest` (subject to `spec.opsRequestOptions`).
4. The Ops-manager operator applies the vertical scaling as usual.
4. The Ops-manager operator applies the vertical scaling as usual, restarting the affected pods.

> **Prerequisite:** a **metrics server** must be installed in the cluster for the VPA recommender to produce recommendations.
> **Prerequisite:** a **metrics server** must be installed in the cluster for the autoscaler to produce recommendations.

In the next doc, we will see a step-by-step guide on compute autoscaling of a Milvus database.
Original file line number Diff line number Diff line change
Expand Up @@ -9,58 +9,58 @@ spec:
compute:
proxy:
trigger: "On"
podLifeTimeThreshold: 1m
podLifeTimeThreshold: 10m
minAllowed:
cpu: 100m
memory: 256Mi
memory: 1Gi
maxAllowed:
cpu: 1000m
memory: 2Gi
resourceDiffPercentage: 10
resourceDiffPercentage: 20
controlledResources: ["cpu", "memory"]
mixcoord:
trigger: "On"
podLifeTimeThreshold: 1m
podLifeTimeThreshold: 10m
minAllowed:
cpu: 100m
memory: 256Mi
memory: 1Gi
maxAllowed:
cpu: 1000m
memory: 2Gi
resourceDiffPercentage: 10
resourceDiffPercentage: 20
controlledResources: ["cpu", "memory"]
datanode:
trigger: "On"
podLifeTimeThreshold: 1m
podLifeTimeThreshold: 10m
minAllowed:
cpu: 100m
memory: 256Mi
memory: 1Gi
maxAllowed:
cpu: 1000m
memory: 2Gi
resourceDiffPercentage: 10
resourceDiffPercentage: 20
controlledResources: ["cpu", "memory"]
querynode:
trigger: "On"
podLifeTimeThreshold: 1m
podLifeTimeThreshold: 10m
minAllowed:
cpu: 100m
memory: 256Mi
memory: 1Gi
maxAllowed:
cpu: 1000m
memory: 2Gi
resourceDiffPercentage: 10
resourceDiffPercentage: 20
controlledResources: ["cpu", "memory"]
streamingnode:
trigger: "On"
podLifeTimeThreshold: 1m
podLifeTimeThreshold: 10m
minAllowed:
cpu: 100m
memory: 256Mi
memory: 1Gi
maxAllowed:
cpu: 1000m
memory: 2Gi
resourceDiffPercentage: 10
resourceDiffPercentage: 20
controlledResources: ["cpu", "memory"]
opsRequestOptions:
apply: IfReady
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ spec:
compute:
node:
trigger: "On"
podLifeTimeThreshold: 1m
podLifeTimeThreshold: 10m
minAllowed:
cpu: 100m
memory: 256Mi
memory: 1Gi
maxAllowed:
cpu: 1000m
memory: 2Gi
resourceDiffPercentage: 10
resourceDiffPercentage: 20
controlledResources: ["cpu", "memory"]
opsRequestOptions:
apply: IfReady
Expand Down
Loading
Loading