diff --git a/docs/guides/milvus/README.md b/docs/guides/milvus/README.md index 1c478db84f..857f9c07f0 100644 --- a/docs/guides/milvus/README.md +++ b/docs/guides/milvus/README.md @@ -26,7 +26,7 @@ Milvus is an open-source vector database built to power embedding similarity sea | External object storage support | ✓ | | KubeDB-managed or externally managed etcd metadata storage | ✓ | | Custom Configuration | ✓ | -| Monitoring using Prometheus and Grafana | ✓ | +| Monitoring using Prometheus | ✓ | | Authentication & Authorization (TLS) | ✓ | | Externally manageable Auth Secret (Rotate Auth) | ✓ | | Persistent volume | ✓ | @@ -44,7 +44,10 @@ KubeDB supports the following Milvus Versions. - `2.6.7` - `2.6.9` - `2.6.11` +- `2.6.15` +- `2.6.15-gpu` - `3.0.0` +- `3.0.0-gpu` ## User Guide diff --git a/docs/guides/milvus/autoscaler/compute/guide.md b/docs/guides/milvus/autoscaler/compute/guide.md index 6166f9adcf..cc0197de07 100644 --- a/docs/guides/milvus/autoscaler/compute/guide.md +++ b/docs/guides/milvus/autoscaler/compute/guide.md @@ -23,7 +23,7 @@ This guide will show you how to use the `KubeDB` Autoscaler operator to autoscal - [MilvusAutoscaler](/docs/guides/milvus/concepts/milvusautoscaler.md) - [Compute Autoscaling Overview](/docs/guides/milvus/autoscaler/compute/overview.md) -- Install the **KubeDB Autoscaler** operator and a **metrics server** in your cluster — the VPA recommender needs metrics to produce recommendations. +- Install the **KubeDB Autoscaler** operator and a **metrics server** in your cluster — the autoscaler's built-in recommender needs metrics to produce recommendations. ```bash $ kubectl get deploy metrics-server -n kube-system @@ -31,13 +31,24 @@ This guide will show you how to use the `KubeDB` Autoscaler operator to autoscal metrics-server 1/1 1 1 5m ``` -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/autoscaler/compute/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/autoscaler/compute/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). ## Compute Autoscaling — Standalone Milvus -Deploy a standalone Milvus and wait until it is `Ready`. Then create a `MilvusAutoscaler` targeting the standalone `node`: +Deploy the [standalone quickstart](/docs/guides/milvus/quickstart/standalone.md) manifest and wait until it is `Ready`: + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/standalone.yaml +milvus.kubedb.com/milvus-standalone created + +$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w +``` + +Then create a `MilvusAutoscaler` targeting the standalone `node`: `compute-standalone.yaml` @@ -53,26 +64,33 @@ spec: compute: node: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] opsRequestOptions: apply: IfReady timeout: 5m ``` +Here, + +- `minAllowed.memory: 1Gi` keeps the autoscaler from shrinking Milvus below what it needs. The generated OpsRequest sets the memory limit equal to the request, so a lower floor (for example `256Mi`) leaves no headroom and Milvus is OOM-killed under real load. +- `podLifeTimeThreshold: 10m` and `resourceDiffPercentage: 20` damp the autoscaler. Every OpsRequest it creates restarts the pods, and a freshly started Milvus briefly uses much more CPU. With a short `podLifeTimeThreshold` and a small `resourceDiffPercentage`, that startup spike immediately triggers another re-scale (and another restart). Waiting 10 minutes after a pod starts and ignoring differences below 20% prevents this. + ```bash $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/autoscaler/compute/yamls/compute-standalone.yaml milvusautoscaler.autoscaling.kubedb.com/milvus-standalone-compute-autoscaler created ``` -The autoscaler creates a `VerticalPodAutoscaler` (VPA) object. Once the VPA recommender produces a recommendation that differs from the current resources by more than `resourceDiffPercentage`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest`. +The autoscaler runs its own built-in recommender (it does not create a `VerticalPodAutoscaler` object). Once the recommender produces a recommendation that differs from the current resources by more than `resourceDiffPercentage`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest`. + +> **Each autoscaler-created OpsRequest restarts the pods** of the role it scales (it uses `Restart` mode). For a standalone Milvus, that means a short outage every time the autoscaler re-scales. ```bash $ kubectl get milvusautoscaler -n demo @@ -80,33 +98,42 @@ NAME AGE milvus-standalone-compute-autoscaler 59s ``` -The autoscaler runs a VPA recommender (fed by the metrics server) and records the recommendation in its status. Once enough samples are collected, the `RecommendationProvided` condition becomes `True` and a target resource set is published: +The recommender (fed by the metrics server) records its recommendation in the autoscaler status. Once enough samples are collected, the `RecommendationProvided` condition becomes `True` and a target resource set is published. The status has this shape (values and the large `checkpoints` array elided): ```bash $ kubectl get milvusautoscaler milvus-standalone-compute-autoscaler -n demo -o jsonpath='{.status}' | jq . { + "checkpoints": [ ... ], + "conditions": [ ... ], "vpas": [ { "conditions": [ - { "type": "RecommendationProvided", "status": "True", "lastTransitionTime": "2026-06-30T18:40:17Z" } + { "type": "RecommendationProvided", "status": "True", ... } ], "recommendation": { "containerRecommendations": [ { "containerName": "milvus", - "lowerBound": { "cpu": "100m", "memory": "256Mi" }, - "target": { "cpu": "143m", "memory": "256Mi" }, - "uncappedTarget": { "cpu": "143m", "memory": "262144k" } + "lowerBound": { "cpu": ..., "memory": ... }, + "target": { "cpu": ..., "memory": ... }, + "uncappedTarget": { "cpu": ..., "memory": ... }, + ... } ] }, - "ref": { "containerName": "milvus", "vpaObjectName": "milvus-standalone" } + "vpaName": "milvus-standalone" } ] } ``` -Here the recommended `target` is `cpu: 143m` / `memory: 256Mi` (the standalone idles well below its `500m` request). Because the recommendation differs from the current request by more than `resourceDiffPercentage` (10%) and stays within `minAllowed`/`maxAllowed`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest`. This is recorded in the autoscaler status as a `CreateOpsRequest` condition: +To see just the recommended target: + +```bash +$ kubectl get milvusautoscaler milvus-standalone-compute-autoscaler -n demo -o jsonpath='{.status.vpas[0].recommendation.containerRecommendations[0].target}' +``` + +An idle standalone uses far less CPU than its `500m` request, so the CPU target is lower, while the memory target is held at `minAllowed` (`1Gi`). When the recommendation differs from the current request by more than `resourceDiffPercentage` (20%) and stays within `minAllowed`/`maxAllowed`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest`. This is recorded in the autoscaler status as a `CreateOpsRequest` condition: ```bash $ kubectl get milvusautoscaler milvus-standalone-compute-autoscaler -n demo \ @@ -122,7 +149,16 @@ The Ops-manager then applies the vertical scaling exactly as in the [vertical sc ## Compute Autoscaling — Distributed Milvus -For a distributed Milvus, the autoscaler is keyed by role (`proxy`, `mixcoord`, `datanode`, `querynode`, `streamingnode`). The sample below enables compute autoscaling for all five roles: +For a distributed Milvus, the autoscaler is keyed by role (`proxy`, `mixcoord`, `datanode`, `querynode`, `streamingnode`). The sample below enables compute autoscaling for all five roles. + +Deploy the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md) manifest and wait until it is `Ready`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/distributed.yaml +milvus.kubedb.com/milvus-cluster created + +$ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w +``` `compute-distributed.yaml` @@ -138,14 +174,14 @@ spec: compute: proxy: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] mixcoord: { ... } datanode: { ... } @@ -158,13 +194,28 @@ spec: (Each role block is identical to the `proxy` block above; see the full file in the `yamls` folder.) -The behavior is identical to standalone, except a VPA object and resource recommendation are produced **per role**. When any role's recommendation differs from its current resources by more than `resourceDiffPercentage`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest` scoped to that role (see the [vertical scaling guide](/docs/guides/milvus/scaling/vertical-scaling/guide.md) for what that ops request looks like). +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/autoscaler/compute/yamls/compute-distributed.yaml +milvusautoscaler.autoscaling.kubedb.com/milvus-compute-autoscaler created +``` + +The behavior is identical to standalone, except a resource recommendation is produced **per role** (one `status.vpas[]` entry per role, for example `vpaName: milvus-cluster-proxy`). When any role's recommendation differs from its current resources by more than `resourceDiffPercentage`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest` scoped to that role (see the [vertical scaling guide](/docs/guides/milvus/scaling/vertical-scaling/guide.md) for what that ops request looks like). The OpsRequests run one at a time, and each restarts the pods of its role. ## Cleaning up +The OpsRequests created by a `MilvusAutoscaler` are owned by it, so deleting the autoscaler also deletes them, **including one that is still running**. An OpsRequest deleted mid-flight can leave the database paused or with pods that don't match its spec. Before deleting the autoscaler, wait until none of its OpsRequests is `Progressing`: + +```bash +$ kubectl get milvusopsrequest -n demo +``` + +Then clean up: + ```bash $ kubectl delete milvusautoscaler -n demo --all -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/autoscaler/compute/overview.md b/docs/guides/milvus/autoscaler/compute/overview.md index d4d6843527..6166e03a61 100644 --- a/docs/guides/milvus/autoscaler/compute/overview.md +++ b/docs/guides/milvus/autoscaler/compute/overview.md @@ -25,7 +25,7 @@ This guide will give an overview on how the KubeDB Autoscaler operator autoscale ## How Compute Autoscaling Works -A `MilvusAutoscaler` of type `compute` watches the resource usage of the Milvus pods (via the metrics server / VPA recommender) and, when usage drifts far enough from the requested resources, it creates a `VerticalScaling` `MilvusOpsRequest` to right-size the pods. +A `MilvusAutoscaler` of type `compute` watches the resource usage of the Milvus pods (via the metrics server and the autoscaler's built-in recommender) and, when usage drifts far enough from the requested resources, it creates a `VerticalScaling` `MilvusOpsRequest` to right-size the pods. `spec.compute` is keyed by component: @@ -39,11 +39,11 @@ spec: compute: node: # or proxy/mixcoord/datanode/querynode/streamingnode trigger: "On" - podLifeTimeThreshold: 1m - resourceDiffPercentage: 10 + podLifeTimeThreshold: 10m + resourceDiffPercentage: 20 minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi @@ -54,15 +54,17 @@ spec: - **`minAllowed`/`maxAllowed`** — the bounds the autoscaler stays within. - **`resourceDiffPercentage`** — how far current resources must drift from the recommendation before an ops request is created. - **`podLifeTimeThreshold`** — minimum pod age before it is considered for scaling. + +Every autoscaler-created OpsRequest restarts the pods it scales, and a freshly started Milvus briefly uses much more CPU. `podLifeTimeThreshold: 10m` and `resourceDiffPercentage: 20` keep that startup spike from triggering another re-scale right after a restart. Keep `minAllowed.memory` at `1Gi` or more: the autoscaler sets the memory limit equal to the request, and Milvus needs more than `256Mi`. - **`controlledResources`** — which resources are managed. The flow is: 1. A user creates a `MilvusAutoscaler` with `spec.compute`. -2. The autoscaler creates a `VerticalPodAutoscaler` (VPA) object which produces resource recommendations. +2. The autoscaler's built-in recommender produces resource recommendations and records them in the autoscaler's `status.vpas` (no `VerticalPodAutoscaler` object is created). 3. When the recommendation differs from the current resources by more than `resourceDiffPercentage`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest` (subject to `spec.opsRequestOptions`). -4. The Ops-manager operator applies the vertical scaling as usual. +4. The Ops-manager operator applies the vertical scaling as usual, restarting the affected pods. -> **Prerequisite:** a **metrics server** must be installed in the cluster for the VPA recommender to produce recommendations. +> **Prerequisite:** a **metrics server** must be installed in the cluster for the autoscaler to produce recommendations. In the next doc, we will see a step-by-step guide on compute autoscaling of a Milvus database. diff --git a/docs/guides/milvus/autoscaler/compute/yamls/compute-distributed.yaml b/docs/guides/milvus/autoscaler/compute/yamls/compute-distributed.yaml index b1bcc39573..fffb8050ca 100644 --- a/docs/guides/milvus/autoscaler/compute/yamls/compute-distributed.yaml +++ b/docs/guides/milvus/autoscaler/compute/yamls/compute-distributed.yaml @@ -9,58 +9,58 @@ spec: compute: proxy: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] mixcoord: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] datanode: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] querynode: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] streamingnode: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] opsRequestOptions: apply: IfReady diff --git a/docs/guides/milvus/autoscaler/compute/yamls/compute-standalone.yaml b/docs/guides/milvus/autoscaler/compute/yamls/compute-standalone.yaml index 92a874076f..4deaa4620e 100644 --- a/docs/guides/milvus/autoscaler/compute/yamls/compute-standalone.yaml +++ b/docs/guides/milvus/autoscaler/compute/yamls/compute-standalone.yaml @@ -9,14 +9,14 @@ spec: compute: node: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] opsRequestOptions: apply: IfReady diff --git a/docs/guides/milvus/autoscaler/storage/guide.md b/docs/guides/milvus/autoscaler/storage/guide.md index 312da1f253..64e1c36e10 100644 --- a/docs/guides/milvus/autoscaler/storage/guide.md +++ b/docs/guides/milvus/autoscaler/storage/guide.md @@ -31,17 +31,42 @@ This guide will show you how to use the `KubeDB` Autoscaler operator to autoscal --set kubedb-autoscaler.storage-metrics-server.enabled=true ``` - It provides the **custom metrics API** (`custom.metrics.k8s.io`) backed by the KubeDB storage-metrics apiserver. The storage autoscaler reads PVC usage from this API. + It provides the **custom metrics API** (`custom.metrics.k8s.io`) backed by the KubeDB storage-metrics apiserver. The storage autoscaler reads PVC usage from this API. It is disabled by default. To enable it on an existing installation: + + ```bash + $ helm upgrade kubedb oci://ghcr.io/appscode-charts/kubedb \ + --namespace kubedb --reuse-values \ + --set kubedb-autoscaler.storage-metrics-server.enabled=true + ``` + + Check that the API is served. The autoscaler queries `custom.metrics.k8s.io/v1beta2`; a `NotFound` error here means the storage metrics server is not running: + + ```bash + $ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta2 + ``` + + Without this API the storage autoscaler never triggers. The autoscaler's status and events stay empty; only the KubeDB autoscaler operator's log shows `custom metrics API returned 404 Not Found`. - The PVC's `StorageClass` must support volume expansion (`allowVolumeExpansion: true`) — e.g. `longhorn-custom`. -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/autoscaler/storage/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/autoscaler/storage/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). ## Storage Autoscaling — Standalone Milvus -Deploy a standalone Milvus on an expansion-capable `StorageClass` and create a `MilvusAutoscaler` targeting the standalone `node`: +Deploy a standalone Milvus on an expansion-capable `StorageClass`. The [volume expansion guide](/docs/guides/milvus/volume-expansion/guide.md)'s `milvus-standalone.yaml` is the standalone quickstart manifest with `storageClassName: longhorn-custom` and a `1Gi` volume: + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/volume-expansion/yamls/milvus-standalone.yaml +milvus.kubedb.com/milvus-standalone created + +$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w +``` + +Then create a `MilvusAutoscaler` targeting the standalone `node`: `storage-standalone.yaml` @@ -76,9 +101,14 @@ When the volume usage crosses `usageThreshold` (30%), the autoscaler creates a ` ```bash $ kubectl get milvusautoscaler -n demo -NAME AGE -milvus-standalone-compute-autoscaler 94s -milvus-storage-autoscaler 93s +NAME AGE +milvus-storage-autoscaler 93s +``` + +You can query the same metric the autoscaler uses for the PVC: + +```bash +$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta2/namespaces/demo/persistentvolumeclaims/data-milvus-standalone-0/volume_used_percentage ``` The storage autoscaler watches the `streamingnode`/`node` PVC usage through the `custom.metrics.k8s.io` API backed by the KubeDB storage-metrics apiserver. When usage crosses `usageThreshold`, it creates a `VolumeExpansion` `MilvusOpsRequest`. In this walkthrough the volume stayed well below the threshold (a freshly-created, near-empty `1Gi` volume), so no expansion was triggered: @@ -89,11 +119,27 @@ NAME SIZE data-milvus-standalone-0 1Gi ``` -> To see the expansion fire, write enough data to push PVC usage past `usageThreshold` (30% here). When it does, the autoscaler creates a `VolumeExpansion` `MilvusOpsRequest` (with `expansionMode` as configured), which the Ops-manager applies — see the [volume expansion guide](/docs/guides/milvus/volume-expansion/guide.md) for the resulting flow and output. +> To see the expansion fire, write enough data to push PVC usage past `usageThreshold` (30% here). For a quick test, write a 400MiB file into the data volume (about 39% of `1Gi`), and remove it afterwards: +> +> ```bash +> $ kubectl exec -n demo milvus-standalone-0 -c milvus -- fallocate -l 400M /var/lib/milvus/fill.bin +> ``` +> When it does, the autoscaler creates a `VolumeExpansion` `MilvusOpsRequest` (with `expansionMode` as configured), which the Ops-manager applies — see the [volume expansion guide](/docs/guides/milvus/volume-expansion/guide.md) for the resulting flow and output. ## Storage Autoscaling — Distributed Milvus -For a distributed Milvus, storage autoscaling targets **`streamingnode`** — the only distributed role with a persistent volume: +For a distributed Milvus, storage autoscaling targets **`streamingnode`** — the only distributed role with a persistent volume. + +Deploy a distributed Milvus on `longhorn-custom`, for example with the volume expansion guide's `milvus-distributed.yaml` (two `streamingnode` replicas with `1Gi` each). The distributed autoscaler reuses the name `milvus-storage-autoscaler`, so delete the standalone one first: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/volume-expansion/yamls/milvus-distributed.yaml +milvus.kubedb.com/milvus-cluster created + +$ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w + +$ kubectl delete milvusautoscaler -n demo milvus-storage-autoscaler +``` `storage-distributed.yaml` @@ -119,13 +165,21 @@ spec: timeout: 10m ``` +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/autoscaler/storage/yamls/storage-distributed.yaml +milvusautoscaler.autoscaling.kubedb.com/milvus-storage-autoscaler created +``` + Because only `streamingnode` carries a persistent volume among the distributed roles, storage autoscaling targets `streamingnode` exclusively. When the `streamingnode` PVC usage crosses `usageThreshold` (34% here), the autoscaler creates a `VolumeExpansion` `MilvusOpsRequest` against `streamingnode` (`expansionMode: Online`), which the Ops-manager applies as shown in the [volume expansion guide](/docs/guides/milvus/volume-expansion/guide.md). ## Cleaning up ```bash +$ kubectl get milvusopsrequest -n demo # wait until none is Progressing $ kubectl delete milvusautoscaler -n demo --all -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/autoscaler/storage/yamls/storage-distributed.yaml b/docs/guides/milvus/autoscaler/storage/yamls/storage-distributed.yaml index 9d1dfc2075..306c2b5ec2 100644 --- a/docs/guides/milvus/autoscaler/storage/yamls/storage-distributed.yaml +++ b/docs/guides/milvus/autoscaler/storage/yamls/storage-distributed.yaml @@ -11,7 +11,6 @@ spec: trigger: "On" usageThreshold: 34 expansionMode: "Online" - scalingRules: - appliesUpto: "100Ti" threshold: "50%" diff --git a/docs/guides/milvus/concepts/milvus.md b/docs/guides/milvus/concepts/milvus.md index f521f2d247..17ddd70172 100644 --- a/docs/guides/milvus/concepts/milvus.md +++ b/docs/guides/milvus/concepts/milvus.md @@ -79,7 +79,7 @@ spec: version: "2.6.11" objectStorage: configSecret: - name: milvus-storage-config + name: milvus-cluster-storage-config topology: mode: Distributed distributed: @@ -131,12 +131,36 @@ spec: name: milvus-storage-config ``` +The secret's keys (`address`, `port`, `accessKeyID`, `secretAccessKey`, `bucketName`, `rootPath`, ...) are copied into the `minio` section of the rendered `milvus.yaml`. Every Milvus instance needs its own `rootPath`; see [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#3-create-the-storage-config-secrets). + ### spec.metaStorage Milvus uses etcd as its metadata store. -- If you omit `spec.metaStorage`, KubeDB provisions and manages an internal etcd cluster for you. -- If you set `spec.metaStorage`, you can point Milvus at an externally managed etcd deployment. +- If you omit `spec.metaStorage`, or set it with `externallyManaged: false`, KubeDB provisions and manages an internal etcd cluster for you as a KubeDB `Etcd` object named `-etcd`. This needs the `Etcd` feature gate in the KubeDB install. +- If you set `spec.metaStorage.externallyManaged: true`, you can point Milvus at an externally managed etcd deployment. + +`spec.metaStorage` has the following fields: + +- `externallyManaged`: `true` to use an etcd that you manage. Only `endpoints`, `authSecret`, `clientTLSSecret` and `rootPath` are used then. +- `endpoints`: client endpoints of the external etcd (for example `http://etcd-0.example.svc:2379`). Required when `externallyManaged` is `true`. +- `size`: number of members of the internal etcd. +- `storageType` (`Durable` or `Ephemeral`) and `storage`: the PVC settings of the internal etcd. +- `tls`: cert-manager `issuerRef` and `certificates` for the internal etcd's client, server and peer traffic. +- `authSecret`: the etcd root credential. For the internal etcd, KubeDB generates `-etcd-auth` when it is omitted. For an external etcd, it names an existing basic-auth secret (`username`, `password`); omit it if the external etcd has authentication disabled. +- `clientTLSSecret`: for an external etcd only, a secret with the client certificate (`tls.crt`, `tls.key`) and the CA that signed the etcd server certificate (`ca.crt`). +- `rootPath`: the etcd key prefix Milvus stores its metadata under (`etcd.rootPath` in `milvus.yaml`). Defaults to `-etcd`. Changing it on a running Milvus points it at a different, likely empty, metadata tree. + +```yaml +spec: + metaStorage: + externallyManaged: true + endpoints: + - http://etcd-0.example.svc.cluster.local:2379 + authSecret: + kind: Secret + name: my-etcd-auth +``` The internal-etcd path is what the current quickstart guides use. @@ -199,6 +223,43 @@ The guides use cert-manager-backed TLS via `spec.tls.issuerRef`. Once TLS is ena See [Configure TLS for Milvus](/docs/guides/milvus/tls/guide.md) for the end-to-end flow. +### spec.configuration + +`spec.configuration` provides custom Milvus configuration. KubeDB merges it into the `milvus.yaml` it renders into the configuration secret (`-`) that every Milvus pod mounts. + +- `secretName`: name of a secret you create in the same namespace. Its `milvus.yaml` key is merged over KubeDB's default configuration. +- `inline`: a map whose `milvus.yaml` key holds configuration that is merged last, so it overrides both the defaults and the values from `secretName`. + +```yaml +spec: + configuration: + secretName: mv-configuration + inline: + milvus.yaml: | + log: + level: info +``` + +To change the configuration of a running database, use a [Reconfigure](/docs/guides/milvus/reconfigure/guide.md) `MilvusOpsRequest`, which also restarts the pods so the new values take effect. + +### spec.serviceTemplates + +`spec.serviceTemplates` customizes the services KubeDB creates. Each entry has an `alias` and the `metadata` / `spec` to apply to that service: + +- `primary`: the client service named `` (gRPC `19530`, metrics `9091`, REST `8080`). +- `stats`: the `-stats` metrics service created when `spec.monitor` is set. + +```yaml +spec: + serviceTemplates: + - alias: primary + metadata: + annotations: + example.com/owner: ml-team + spec: + type: NodePort +``` + ### spec.monitor `spec.monitor` configures metrics exposure for monitoring. The current Milvus guides use Prometheus Operator integration: @@ -245,6 +306,7 @@ Once a `Milvus` object becomes `Ready`, KubeDB creates and manages: - the client and metrics services, - persistent volume claims for standalone Milvus or distributed `streamingnode`, - the authentication secret, +- the internal etcd (a KubeDB `Etcd` named `-etcd`, with PVCs `data--etcd-` and auth secret `-etcd-auth`) unless `spec.metaStorage.externallyManaged` is `true`, - the rendered configuration secret, - TLS certificate secrets when TLS is enabled, - an `AppBinding` that describes how to connect to the database. diff --git a/docs/guides/milvus/concepts/milvusautoscaler.md b/docs/guides/milvus/concepts/milvusautoscaler.md index 5ebc0f54b8..c628feec0f 100644 --- a/docs/guides/milvus/concepts/milvusautoscaler.md +++ b/docs/guides/milvus/concepts/milvusautoscaler.md @@ -39,14 +39,14 @@ spec: compute: node: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] storage: node: @@ -75,58 +75,58 @@ spec: compute: proxy: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] mixcoord: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] datanode: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] querynode: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] streamingnode: trigger: "On" - podLifeTimeThreshold: 1m + podLifeTimeThreshold: 10m minAllowed: cpu: 100m - memory: 256Mi + memory: 1Gi maxAllowed: cpu: 1000m memory: 2Gi - resourceDiffPercentage: 10 + resourceDiffPercentage: 20 controlledResources: ["cpu", "memory"] storage: streamingnode: @@ -171,7 +171,7 @@ Each compute block supports: - `resourceDiffPercentage` - minimum percentage difference between current resources and the recommendation before a scaling operation is created. - `controlledResources` - which resources may be autoscaled, typically `cpu` and `memory`. -For compute autoscaling, the autoscaler creates a `VerticalPodAutoscaler` recommendation first and then generates a `VerticalScaling` [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) when the recommendation passes the configured thresholds. +For compute autoscaling, the autoscaler's built-in recommender produces a recommendation first (recorded in `status.vpas`; no `VerticalPodAutoscaler` object is created) and then generates a `VerticalScaling` [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) when the recommendation passes the configured thresholds. See [Compute Autoscaling Overview](/docs/guides/milvus/autoscaler/compute/overview.md). @@ -193,7 +193,7 @@ Each storage block supports: - `expansionMode` - `Online` or `Offline`. - `scalingRules` - how much storage should be added when usage crosses the threshold. -For storage autoscaling, the autoscaler watches PVC usage from Prometheus and creates a `VolumeExpansion` [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) when the configured threshold is crossed. +For storage autoscaling, the autoscaler reads PVC usage from the `custom.metrics.k8s.io` API served by KubeDB's storage metrics server and creates a `VolumeExpansion` [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) when the configured threshold is crossed. See [Storage Autoscaling Overview](/docs/guides/milvus/autoscaler/storage/overview.md). @@ -220,21 +220,21 @@ spec: ### Compute autoscaling 1. A user creates a `MilvusAutoscaler` with `spec.compute`. -2. The autoscaler creates one VPA recommendation source per targeted component. +2. The autoscaler's built-in recommender produces one recommendation per targeted component. 3. When the recommendation differs from current resources by more than `resourceDiffPercentage`, the autoscaler creates a `VerticalScaling` `MilvusOpsRequest`. 4. The Ops-manager applies the scaling operation. ### Storage autoscaling 1. A user creates a `MilvusAutoscaler` with `spec.storage`. -2. The autoscaler watches persistent-volume usage from Prometheus. +2. The autoscaler reads persistent-volume usage from the `custom.metrics.k8s.io` API (KubeDB storage metrics server). 3. When usage exceeds `usageThreshold`, the autoscaler creates a `VolumeExpansion` `MilvusOpsRequest`. 4. The Ops-manager applies the volume expansion. ## Prerequisites -- Compute autoscaling requires a metrics server and VPA recommender. -- Storage autoscaling requires Prometheus volume metrics. +- Compute autoscaling requires a metrics server. +- Storage autoscaling requires KubeDB's storage metrics server, enabled with `--set kubedb-autoscaler.storage-metrics-server.enabled=true` in the KubeDB install. - Storage autoscaling also requires an expansion-capable `StorageClass`. ## Related Concepts diff --git a/docs/guides/milvus/concepts/milvusopsrequest.md b/docs/guides/milvus/concepts/milvusopsrequest.md index 90f9e7a23b..b3cc7034d1 100644 --- a/docs/guides/milvus/concepts/milvusopsrequest.md +++ b/docs/guides/milvus/concepts/milvusopsrequest.md @@ -264,7 +264,7 @@ resources: Used when `spec.type: HorizontalScaling`. -Horizontal scaling is distributed-only. The requested replica counts are set under `spec.horizontalScaling.topology`: +Horizontal scaling is distributed-only. The requested replica counts are set under `spec.horizontalScaling.topology`. The data node key here is `dataNode` (camelCase), unlike `spec.verticalScaling`, where it is `datanode`: ```yaml spec: @@ -272,7 +272,7 @@ spec: topology: proxy: 2 mixcoord: 2 - datanode: 2 + dataNode: 2 querynode: 2 streamingnode: 2 ``` @@ -317,7 +317,7 @@ The Milvus reconfigure flow supports: - `configSecret` - reference to a secret whose `milvus.yaml` key holds configuration. - `applyConfig` - inline configuration to merge under the `milvus.yaml` key. - `removeCustomConfig` - remove previously applied custom configuration before applying the new one. -- `restart` - whether pods should be restarted after the config change. +- `restart` - whether pods should be restarted after the config change (`auto`, the default, `true` or `false`). Milvus does not reload its configuration while running, so with `false` the new values take effect only at the next pod restart. ### spec.tls diff --git a/docs/guides/milvus/monitoring/using-prometheus-operator.md b/docs/guides/milvus/monitoring/using-prometheus-operator.md index e6e8343973..4803e9b41c 100644 --- a/docs/guides/milvus/monitoring/using-prometheus-operator.md +++ b/docs/guides/milvus/monitoring/using-prometheus-operator.md @@ -24,7 +24,7 @@ section_menu_id: guides - [Milvus](/docs/guides/milvus/concepts/milvus.md) - [Monitoring Overview](/docs/guides/milvus/monitoring/overview.md) -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/monitoring/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/monitoring/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). @@ -47,7 +47,16 @@ spec: - `serviceMonitor.labels` are applied to the generated `ServiceMonitor` so the Prometheus Operator picks it up (`release: prometheus` must match your Prometheus `serviceMonitorSelector`). - `serviceMonitor.interval` is the scrape interval. -Deploy the database and wait until it is `Ready`. +The `monitoring/yamls/standalone.yaml` and `monitoring/yamls/distributed.yaml` files are the quickstart manifests with this block added. Deploy the standalone database and wait until it is `Ready`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/monitoring/yamls/standalone.yaml +milvus.kubedb.com/milvus-standalone created + +$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w +``` + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. ## Stats Service @@ -118,11 +127,30 @@ Key points: Once the Prometheus Operator reconciles this `ServiceMonitor`, Milvus metrics begin appearing in Prometheus. +## Verify in Prometheus + +Port-forward the Prometheus service created by your Prometheus Operator installation (with kube-prometheus-stack release `prometheus` in namespace `monitoring`, it is `prometheus-kube-prometheus-prometheus`; adjust the name and namespace to your installation): + +```bash +$ kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 +``` + +Then open `http://localhost:9090/targets` and check that the `serviceMonitor/demo/milvus-standalone-stats/0` target is `UP`, or query the `up` metric for the namespace: + +```bash +$ curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=up{namespace="demo"}' | jq '.data.result[] | {job: .metric.job, pod: .metric.pod, value: .value[1]}' +``` + +Each scraped Milvus pod reports `"value": "1"`. Milvus metrics are prefixed with `milvus_`, for example `milvus_num_node`. + ## Distributed Milvus Monitoring works identically for a distributed Milvus. A single stats service and `ServiceMonitor` named `milvus-cluster-stats` are created, and metrics are scraped from the distributed components (each role's pods expose port `9091`). ```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/monitoring/yamls/distributed.yaml +milvus.kubedb.com/milvus-cluster created + $ kubectl get svc -n demo -l app.kubernetes.io/instance=milvus-cluster NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE milvus-cluster ClusterIP 10.43.221.1 19530/TCP,9091/TCP,8080/TCP 3m @@ -140,22 +168,32 @@ spec: interval: 10s path: /metrics port: metrics + relabelings: + - action: replace + sourceLabels: + - __meta_kubernetes_endpoint_address_target_name + targetLabel: pod scheme: http namespaceSelector: matchNames: - demo selector: matchLabels: + app.kubernetes.io/component: database app.kubernetes.io/instance: milvus-cluster app.kubernetes.io/managed-by: kubedb.com app.kubernetes.io/name: milvuses.kubedb.com kubedb.com/role: stats ``` +In Prometheus, the `milvus-cluster-stats` job has one target per role pod (mixcoord, datanode, querynode, streamingnode and proxy), each with `up` = `1`. + ## Cleaning up ```bash -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/monitoring/yamls/distributed.yaml b/docs/guides/milvus/monitoring/yamls/distributed.yaml index 4f39844611..8b2170f91c 100644 --- a/docs/guides/milvus/monitoring/yamls/distributed.yaml +++ b/docs/guides/milvus/monitoring/yamls/distributed.yaml @@ -7,7 +7,7 @@ spec: version: "2.6.11" objectStorage: configSecret: - name: "milvus-storage-config" + name: "milvus-cluster-storage-config" topology: mode: Distributed distributed: @@ -27,12 +27,3 @@ spec: labels: release: prometheus interval: 10s - tls: - issuerRef: - name: milvus-issuer - kind: Issuer - apiGroup: "cert-manager.io" - external: - mode: mTLS - internal: - mode: TLS \ No newline at end of file diff --git a/docs/guides/milvus/monitoring/yamls/standalone.yaml b/docs/guides/milvus/monitoring/yamls/standalone.yaml index a5e2e7b98f..5f4d88ff9e 100644 --- a/docs/guides/milvus/monitoring/yamls/standalone.yaml +++ b/docs/guides/milvus/monitoring/yamls/standalone.yaml @@ -25,12 +25,3 @@ spec: labels: release: prometheus interval: 10s - tls: - issuerRef: - name: milvus-issuer - kind: Issuer - apiGroup: "cert-manager.io" - external: - mode: mTLS - internal: - mode: TLS \ No newline at end of file diff --git a/docs/guides/milvus/quickstart/distributed.md b/docs/guides/milvus/quickstart/distributed.md index b0ac652cdf..35a8448055 100644 --- a/docs/guides/milvus/quickstart/distributed.md +++ b/docs/guides/milvus/quickstart/distributed.md @@ -19,8 +19,8 @@ This tutorial shows how to use KubeDB to provision a **distributed** [Milvus](ht ## Before You Begin - You need a Kubernetes cluster and `kubectl` configured to talk to it. -- Install KubeDB with `--set global.featureGates.Milvus=true`. -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). That guide installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Install KubeDB with `--set global.featureGates.Milvus=true --set global.featureGates.Etcd=true` (the internal metadata store is a KubeDB `Etcd`). +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). That guide installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). - This quickstart intentionally uses the smallest working distributed manifest. It does **not** require Prometheus Operator or cert-manager. > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/quickstart/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). @@ -55,7 +55,7 @@ spec: version: "2.6.11" objectStorage: configSecret: - name: "milvus-storage-config" + name: "milvus-cluster-storage-config" topology: mode: Distributed distributed: @@ -70,7 +70,7 @@ spec: storage: 1Gi ``` -This manifest also omits `spec.metaStorage`, so KubeDB creates and manages the etcd metadata cluster through the installed etcd operator. +This manifest also omits `spec.metaStorage`, so KubeDB creates and manages the etcd metadata cluster as a KubeDB `Etcd` object named `milvus-cluster-etcd`. It uses the `milvus-cluster-storage-config` secret, so its object-storage `rootPath` is separate from the standalone quickstart's (see [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#3-create-the-storage-config-secrets)). ```bash $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/distributed.yaml @@ -79,7 +79,7 @@ milvus.kubedb.com/milvus-cluster created ## Wait for the Cluster to be Ready -Distributed Milvus takes longer than standalone because multiple components have to settle: +KubeDB first brings up the internal etcd (`milvus-cluster-etcd`, three pods), then the Milvus roles. Expect a few minutes, longer on the first run while images are pulled: ```bash $ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w @@ -146,12 +146,13 @@ data-milvus-cluster-streamingnode-0 Bound pvc-... 1Gi RWO ### Auth Secret and AppBinding -As with standalone, KubeDB auto-generates the auth secret (`milvus-cluster-auth`, user `root`), the rendered configuration secret, and an `AppBinding`. Because this quickstart does not enable TLS, the AppBinding scheme is `http`: +As with standalone, KubeDB auto-generates the auth secret (`milvus-cluster-auth`, user `root`), the rendered configuration secret, the internal etcd's auth secret (`milvus-cluster-etcd-auth`), and an `AppBinding`. Because this quickstart does not enable TLS, the AppBinding scheme is `http`: ```bash $ kubectl get secret -n demo | grep milvus-cluster -milvus-cluster-auth kubernetes.io/basic-auth 2 3m -milvus-cluster-d7497a Opaque 2 3m +milvus-cluster-auth kubernetes.io/basic-auth 2 3m +milvus-cluster-d7497a Opaque 2 3m +milvus-cluster-etcd-auth kubernetes.io/basic-auth 2 3m $ kubectl get appbinding milvus-cluster -n demo -o jsonpath='{.spec.clientConfig.service}' {"name":"milvus-cluster","path":"/","port":19530,"scheme":"http"} @@ -208,6 +209,14 @@ $ curl -s -X POST "http://localhost:8080/v2/vectordb/entities/insert" \ {"id": 2, "vector": [0.5, 0.4, 0.3, 0.2]} ] }' | jq . +{ + "code": 0, + "cost": 0, + "data": { + "insertCount": 2, + "insertIds": [1, 2] + } +} $ curl -s -X POST "http://localhost:8080/v2/vectordb/entities/search" \ -H "Authorization: Bearer root:${PASSWORD}" \ @@ -220,23 +229,42 @@ $ curl -s -X POST "http://localhost:8080/v2/vectordb/entities/search" \ "metricType": "L2", "params": {"nprobe": 10} }' | jq . +{ + "code": 0, + "cost": 0, + "data": [ + { + "distance": 0, + "id": 2 + }, + { + "distance": 0.64, + "id": 1 + } + ], + "topks": [2] +} ``` ## Use an External etcd Cluster Instead The default quickstart omits `spec.metaStorage`, so KubeDB manages etcd for you. If you want Milvus to use an external etcd cluster instead: -1. Install the etcd operator. -2. Manage your etcd cluster yourself. -3. Add `spec.metaStorage.externallyManaged: true` and the external endpoints to your `Milvus` manifest as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#optional-path-use-external-etcd). +1. Manage your etcd cluster yourself. +2. Add `spec.metaStorage.externallyManaged: true` and the external endpoints to your `Milvus` manifest as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#optional-path-use-external-etcd). ## Cleaning up ```bash $ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" $ kubectl delete milvus.kubedb.com milvus-cluster -n demo +$ kubectl delete secret -n demo milvus-cluster-etcd-auth ``` +The internal etcd's auth secret is not removed by `WipeOut`, so the last command deletes it. + +KubeDB does not delete the database's data in object storage (`mlv-release/milvus-cluster/` in this guide). If you will create a new Milvus on the same `rootPath` later, for example by following another guide, delete that prefix first, as shown in [Delete a Milvus Instance's Object-Storage Data](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. + If you want to remove the dependencies too, either delete the whole `demo` namespace or follow the cleanup steps in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#cleanup). ## Next Steps diff --git a/docs/guides/milvus/quickstart/prerequisites.md b/docs/guides/milvus/quickstart/prerequisites.md index b758585e25..4c381413c9 100644 --- a/docs/guides/milvus/quickstart/prerequisites.md +++ b/docs/guides/milvus/quickstart/prerequisites.md @@ -16,20 +16,21 @@ section_menu_id: guides Milvus will not start from a bare `Milvus` manifest alone. Every Milvus deployment in KubeDB needs: -- Object storage, exposed through a secret named `milvus-storage-config`. +- Object storage, exposed through a secret (one per Milvus instance; `milvus-storage-config` and `milvus-cluster-storage-config` in these guides). - etcd for metadata. -This guide sets up both dependencies in the `demo` namespace and clarifies when you need only the **etcd operator** and when you instead want to point Milvus at an **external etcd cluster**. +This guide sets up object storage in the `demo` namespace and explains the two etcd options: an internal etcd that KubeDB creates for you (the default), or an **external etcd cluster** you manage yourself. ## Before You Begin - You need a Kubernetes cluster and `kubectl` configured to talk to it. -- Install KubeDB with the Milvus feature gate enabled: +- Install KubeDB with the Milvus and Etcd feature gates enabled. The internal metadata store that KubeDB creates for each Milvus is a KubeDB `Etcd` object, so the Etcd feature gate is required too: ```bash helm install kubedb oci://ghcr.io/appscode-charts/kubedb \ --namespace kubedb --create-namespace \ - --set global.featureGates.Milvus=true + --set global.featureGates.Milvus=true \ + --set global.featureGates.Etcd=true ``` ## Create the Demo Namespace @@ -55,9 +56,13 @@ $ helm upgrade --install --namespace "minio-operator" --create-namespace "minio- ### 2. Deploy a MinIO Tenant +The tenant chart's default image (`quay.io/minio/minio`) can no longer be pulled anonymously (the pull fails with `401 UNAUTHORIZED` and the pod stays in `ImagePullBackOff`), so the command below overrides it with a pullable MinIO image: + ```bash $ helm upgrade --install --namespace "demo" milvus-minio minio/tenant \ --set tenant.name=milvus-minio \ + --set tenant.image.repository=docker.io/bitnamilegacy/minio \ + --set tenant.image.tag=2025.7.23-debian-12-r3 \ --set tenant.pools[0].servers=1 \ --set tenant.pools[0].volumesPerServer=1 \ --set tenant.pools[0].size=1Gi \ @@ -66,6 +71,8 @@ $ helm upgrade --install --namespace "demo" milvus-minio minio/tenant \ --set tenant.pools[0].name="default" ``` +> The MinIO Operator allows only one Tenant per namespace. If the namespace already has a Tenant, reuse it as described in [If You Already Have S3 or MinIO](#if-you-already-have-s3-or-minio). Also note that `tenant.name` does not rename the chart's config secret (`myminio-env-configuration`), so a second tenant release in the same namespace collides with the first one. + Wait for the Tenant to become initialized: ```bash @@ -74,9 +81,15 @@ NAME STATE HEALTH AGE milvus-minio Initialized green 1m ``` -### 3. Create the Storage Config Secret +### 3. Create the Storage Config Secrets + +Create a secret with your MinIO connection details. The keys must use the milvus.yaml config key names directly (KubeDB copies them into the `minio` section of `milvus.yaml`). + +**Use a distinct `rootPath` for every Milvus instance.** `rootPath` is the prefix under which a Milvus stores all of its object-storage data (segments, logs and its write-ahead log). Two Milvus instances on the same bucket and `rootPath` read and write the same objects: one instance's garbage collection can delete the other's files, and their write-ahead-log lock objects collide. A secret holds one `rootPath`, so create one secret per Milvus instance. + +Deleting a Milvus (with any `deletionPolicy`) does **not** delete its objects under `//`. Reusing that `rootPath` for a **new** Milvus is not supported: the new instance gets a new metadata etcd, collides with the old write-ahead-log lock objects under `/wp/`, and can hang in `Provisioning` with `writer lock session has expired` in its logs. Before you create a new database on a `rootPath` that was used before, either change `rootPath` in its secret or [delete the old prefix](#delete-a-milvus-instances-object-storage-data) first. -Create a secret with your MinIO connection details. The keys must use the milvus.yaml config key names directly: +The guides deploy a standalone Milvus named `milvus-standalone`, which uses `milvus-storage-config`, and a distributed Milvus named `milvus-cluster`, which uses `milvus-cluster-storage-config`. Both can run at the same time: ```yaml apiVersion: v1 @@ -90,28 +103,44 @@ stringData: accessKeyID: "minio" secretAccessKey: "minio123" bucketName: "mlv-release" - rootPath: "files" + rootPath: "milvus-standalone" + port: "9000" +--- +apiVersion: v1 +kind: Secret +metadata: + name: milvus-cluster-storage-config + namespace: demo +type: Opaque +stringData: + address: "milvus-minio-hl.demo.svc.cluster.local" + accessKeyID: "minio" + secretAccessKey: "minio123" + bucketName: "mlv-release" + rootPath: "milvus-cluster" port: "9000" ``` -Apply the secret: +Apply the secrets: ```bash $ kubectl apply -f milvus-storage-config.yaml secret/milvus-storage-config created +secret/milvus-cluster-storage-config created ``` Verify: ```bash -$ kubectl get secret milvus-storage-config -n demo -NAME TYPE DATA AGE -milvus-storage-config Opaque 6 1m +$ kubectl get secret -n demo milvus-storage-config milvus-cluster-storage-config +NAME TYPE DATA AGE +milvus-storage-config Opaque 6 1m +milvus-cluster-storage-config Opaque 6 1m ``` ### If You Already Have S3 or MinIO -You do not need to use the MinIO Operator. You can instead create only the secret and point Milvus at your existing object store: +You do not need to use the MinIO Operator. You can instead create only the secrets and point Milvus at your existing object store (again, one secret with its own `rootPath` per Milvus instance): ```yaml apiVersion: v1 @@ -131,35 +160,17 @@ stringData: If you use a different secret name, update `spec.objectStorage.configSecret.name` in the `Milvus` manifest accordingly. -## Install the etcd Operator +## etcd for Metadata Milvus always uses etcd as its metadata store. In KubeDB, there are two supported patterns: -1. **KubeDB-managed etcd**: omit `spec.metaStorage`. KubeDB creates the internal etcd cluster for you. +1. **KubeDB-managed etcd**: omit `spec.metaStorage`. KubeDB creates an internal etcd for you as a KubeDB `Etcd` object. This needs the `Etcd` feature gate (`--set global.featureGates.Etcd=true`) in the KubeDB install; no separate etcd operator is needed. 2. **Externally managed etcd**: set `spec.metaStorage.externallyManaged: true` and provide endpoints yourself. -In both cases, the **etcd operator must already be installed** in the cluster. - -The simplest installation path is the upstream install manifest: +Verify that the KubeDB `Etcd` CRD is registered: ```bash -$ kubectl apply -f https://raw.githubusercontent.com/etcd-io/etcd-operator/refs/heads/main/dist/install-v0.1.0.yaml -namespace/etcd-operator-system created -customresourcedefinition.apiextensions.k8s.io/etcdclusters.operator.etcd.io created -deployment.apps/etcd-operator-controller-manager created -... -``` - -Verify it: - -```bash -$ kubectl get deployment -n etcd-operator-system -NAME READY UP-TO-DATE AVAILABLE AGE -etcd-operator-controller-manager 1/1 1 1 1m - -$ kubectl get crd etcdclusters.operator.etcd.io -NAME CREATED AT -etcdclusters.operator.etcd.io 2026-07-08T... +$ kubectl get crd etcds.kubedb.com ``` ## Default Path: KubeDB-Managed etcd @@ -168,11 +179,11 @@ For the [standalone](/docs/guides/milvus/quickstart/standalone.md) and [distribu If you **omit** `spec.metaStorage` from the `Milvus` manifest: -- KubeDB creates an internal etcd cluster +- KubeDB creates an internal `Etcd` object named `-etcd` (3 members) and an auth secret `-etcd-auth` for it - KubeDB wires Milvus to that internal etcd automatically -- You do **not** need to apply any external etcd YAML yourself +- You do **not** need to apply any etcd YAML yourself -So for the default quickstarts, **having the etcd operator running is enough**. +So for the default quickstarts, **the `Etcd` feature gate in the KubeDB install is enough**. ## Optional Path: Use External etcd @@ -191,7 +202,7 @@ Requirements for external etcd: - The endpoints must be reachable from the Milvus pods. - The etcd cluster must already be healthy before you create the `Milvus` object. -- The etcd operator is still required in the cluster for the default KubeDB-managed path, but this external-endpoint configuration does not require any sample external etcd YAML from these docs. +- If the external etcd requires authentication or TLS, reference its credentials with `spec.metaStorage.authSecret` and its client certificate with `spec.metaStorage.clientTLSSecret` (see the [Milvus concepts](/docs/guides/milvus/concepts/milvus.md#specmetastorage)). Only choose this path if you intentionally want Milvus to use an external etcd cluster. The default quickstarts do not require any external etcd manifest. @@ -213,34 +224,48 @@ namespace "demo" deleted If you want to keep the namespace and clean up dependencies separately: -1. Delete your external etcd resources using whatever workflow manages them: +1. Delete your Milvus objects first (see the cleanup section of each guide). With `deletionPolicy: Delete` or `WipeOut`, KubeDB also deletes the internal etcd and its PVCs. - ```bash - # Example only: - # kubectl delete - ``` - -2. Delete leftover etcd PVCs if your etcd management workflow leaves them behind: +2. Delete what the internal etcd leaves behind. The `-etcd-auth` secret is kept even after a `WipeOut` delete. If any etcd PVCs remain (for example, after `deletionPolicy: Halt`), delete them too. Their names follow the pattern `data--etcd-`: ```bash - # Example only: - # kubectl delete pvc -n demo + $ kubectl delete pvc -n demo data-milvus-standalone-etcd-0 data-milvus-standalone-etcd-1 data-milvus-standalone-etcd-2 + $ kubectl delete secret -n demo milvus-standalone-etcd-auth ``` - For KubeDB-managed etcd, the PVC names follow the pattern `etcd-data--etcd-`. + If you use an external etcd, clean it up with whatever workflow manages it. -3. Delete the MinIO Tenant: +3. Delete each Milvus instance's object-storage data, as shown in [Delete a Milvus Instance's Object-Storage Data](#delete-a-milvus-instances-object-storage-data). + +4. Delete the MinIO Tenant: ```bash $ helm uninstall milvus-minio -n demo ``` -4. Delete the storage config secret: +5. Delete the storage config secrets: ```bash - $ kubectl delete secret -n demo milvus-storage-config + $ kubectl delete secret -n demo milvus-storage-config milvus-cluster-storage-config ``` +## Delete a Milvus Instance's Object-Storage Data + +KubeDB does not delete a Milvus instance's objects from object storage, even with `deletionPolicy: WipeOut`. Once the `Milvus` object is gone, remove its `rootPath` prefix with the MinIO client (`mc`) before reusing that `rootPath` for a new Milvus. For the tenant created above, port-forward the MinIO service in a separate terminal: + +```bash +$ kubectl port-forward svc/milvus-minio-hl -n demo 9000:9000 +``` + +Then register an alias and delete the prefix of the deleted database, `milvus-standalone` in this example: + +```bash +$ mc alias set milvus-minio http://localhost:9000 minio minio123 +$ mc rm -r --force milvus-minio/mlv-release/milvus-standalone +``` + +For your own object store, use its endpoint, credentials, `bucketName` and `rootPath` instead (`mc rm -r --force //`). + ## Next Steps - [Deploy standalone Milvus](/docs/guides/milvus/quickstart/standalone.md) diff --git a/docs/guides/milvus/quickstart/standalone.md b/docs/guides/milvus/quickstart/standalone.md index 9a5236fead..7441bd1b79 100644 --- a/docs/guides/milvus/quickstart/standalone.md +++ b/docs/guides/milvus/quickstart/standalone.md @@ -19,8 +19,8 @@ This tutorial shows how to use KubeDB to provision a **standalone** [Milvus](htt ## Before You Begin - You need a Kubernetes cluster and `kubectl` configured to talk to it. -- Install KubeDB with `--set global.featureGates.Milvus=true`. -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). That guide installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Install KubeDB with `--set global.featureGates.Milvus=true --set global.featureGates.Etcd=true` (the internal metadata store is a KubeDB `Etcd`). +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). That guide installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). - This quickstart intentionally uses the smallest working manifest. It does **not** require Prometheus Operator or cert-manager. > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/quickstart/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). @@ -31,10 +31,14 @@ When you install the KubeDB operator, it registers a CRD named `MilvusVersion`. ```bash $ kubectl get milvusversions -NAME VERSION DB_IMAGE DEPRECATED AGE -2.6.11 2.6.11 ghcr.io/appscode-images/milvus:2.6.11 11h -2.6.7 2.6.7 ghcr.io/appscode-images/milvus:2.6.7 11h -2.6.9 2.6.9 ghcr.io/appscode-images/milvus:2.6.9 11h +NAME VERSION DB_IMAGE DEPRECATED AGE +2.6.11 2.6.11 ghcr.io/appscode-images/milvus:2.6.11 21h +2.6.15 2.6.15 ghcr.io/appscode-images/milvus:2.6.15 21h +2.6.15-gpu 2.6.15 ghcr.io/appscode-images/milvus:2.6.15-gpu 21h +2.6.7 2.6.7 ghcr.io/appscode-images/milvus:2.6.7 21h +2.6.9 2.6.9 ghcr.io/appscode-images/milvus:2.6.9 21h +3.0.0 3.0.0 ghcr.io/appscode-images/milvus:3.0.0 21h +3.0.0-gpu 3.0.0 ghcr.io/appscode-images/milvus:3.0.0-gpu 21h ``` ## Create a Standalone Milvus @@ -71,7 +75,8 @@ Here, - `spec.version` is the name of a `MilvusVersion` object. - `spec.topology.mode: Standalone` deploys Milvus as a single all-in-one workload. - `spec.objectStorage.configSecret` points to the required MinIO/object-storage secret. -- `spec.metaStorage` is omitted, so KubeDB creates and manages an internal etcd cluster through the installed etcd operator. +- `spec.metaStorage` is omitted, so KubeDB creates and manages an internal etcd cluster as a KubeDB `Etcd` object named `milvus-standalone-etcd`. +- `spec.objectStorage.configSecret` uses `milvus-storage-config`, whose `rootPath` must not be shared with any other Milvus instance (see [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#3-create-the-storage-config-secrets)). - `spec.storageType: Durable` tells KubeDB to provision persistent storage for the standalone workload. Create the database: @@ -91,10 +96,10 @@ Watch the `Milvus` object until its `STATUS` becomes `Ready`: $ kubectl get milvuses.kubedb.com -n demo -w NAME VERSION STATUS AGE milvus-standalone 2.6.11 Provisioning 24s -milvus-standalone 2.6.11 Ready 39s +milvus-standalone 2.6.11 Ready 4m ``` -Standalone Milvus typically becomes ready within a few minutes. +KubeDB first brings up the internal etcd (`milvus-standalone-etcd`, three pods), then starts Milvus. Expect the database to take a few minutes to become `Ready`, longer on the first run while images are pulled. ## Verify the Created Resources @@ -136,7 +141,7 @@ NAME STATUS VOLUME C data-milvus-standalone-0 Bound pvc-a6333ee2-f0ab-4ec2-8437-599d270b9ed0 1Gi RWO local-path 90s ``` -The internal etcd metadata store provisions its own PVCs (`etcd-data--etcd-*`), and MinIO has separate PVCs as well. +The internal etcd metadata store provisions its own PVCs (`data-milvus-standalone-etcd-0`, `-1`, `-2`), which do not carry the `app.kubernetes.io/instance=milvus-standalone` label, and MinIO has separate PVCs as well. ### Auth Secret @@ -144,8 +149,9 @@ Milvus authentication is enabled by default. Because `spec.authSecret` was not p ```bash $ kubectl get secret -n demo | grep milvus-standalone -milvus-standalone-42559a Opaque 2 92s -milvus-standalone-auth kubernetes.io/basic-auth 2 92s +milvus-standalone-42559a Opaque 2 92s +milvus-standalone-auth kubernetes.io/basic-auth 2 92s +milvus-standalone-etcd-auth kubernetes.io/basic-auth 2 92s $ kubectl get secret milvus-standalone-auth -n demo -o jsonpath='{.data.username}' | base64 -d root @@ -154,7 +160,7 @@ $ kubectl get secret milvus-standalone-auth -n demo -o jsonpath='{.data.password ``` -The other secret (`milvus-standalone-42559a`) is the rendered configuration secret holding `milvus.yaml` and `glog.conf`. +The `milvus-standalone-42559a` secret is the rendered configuration secret holding `milvus.yaml` and `glog.conf`; its suffix is random. `milvus-standalone-etcd-auth` holds the credentials of the internal etcd. It is kept when the database is deleted, see [Cleaning up](#cleaning-up). ### AppBinding @@ -291,17 +297,21 @@ $ curl -s -X POST "http://localhost:8080/v2/vectordb/entities/search" \ The default quickstart omits `spec.metaStorage`, so KubeDB manages etcd for you. If you want Milvus to use an external etcd cluster instead: -1. Install the etcd operator. -2. Manage your etcd cluster yourself. -3. Add `spec.metaStorage.externallyManaged: true` and the external endpoints to your `Milvus` manifest as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#optional-path-use-external-etcd). +1. Manage your etcd cluster yourself. +2. Add `spec.metaStorage.externallyManaged: true` and the external endpoints to your `Milvus` manifest as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#optional-path-use-external-etcd). ## Cleaning up ```bash $ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" $ kubectl delete milvus.kubedb.com milvus-standalone -n demo +$ kubectl delete secret -n demo milvus-standalone-etcd-auth ``` +The internal etcd's auth secret is not removed by `WipeOut`, so the last command deletes it. + +KubeDB does not delete the database's data in object storage (`mlv-release/milvus-standalone/` in this guide). If you will create a new Milvus on the same `rootPath` later, for example by following another guide, delete that prefix first, as shown in [Delete a Milvus Instance's Object-Storage Data](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. + If you want to remove the dependencies too, either delete the whole `demo` namespace or follow the cleanup steps in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#cleanup). ## Next Steps diff --git a/docs/guides/milvus/quickstart/yamls/distributed.yaml b/docs/guides/milvus/quickstart/yamls/distributed.yaml index 6c8223df8a..ffb51ecc78 100644 --- a/docs/guides/milvus/quickstart/yamls/distributed.yaml +++ b/docs/guides/milvus/quickstart/yamls/distributed.yaml @@ -7,7 +7,7 @@ spec: version: "2.6.11" objectStorage: configSecret: - name: "milvus-storage-config" + name: "milvus-cluster-storage-config" topology: mode: Distributed distributed: diff --git a/docs/guides/milvus/recommendation/guide.md b/docs/guides/milvus/recommendation/guide.md index 8ad6b2a956..dccde67bce 100644 --- a/docs/guides/milvus/recommendation/guide.md +++ b/docs/guides/milvus/recommendation/guide.md @@ -34,12 +34,28 @@ The KubeDB Recommendation Engine watches your databases and proactively generate recommendations.supervisor.appscode.com 2026-06-30T05:18:01Z ``` -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). + +- Install [cert-manager](https://cert-manager.io/docs/installation/). The sample manifests set `spec.tls` so that a `ReconfigureTLS` recommendation can be generated, and they reference an `Issuer` named `milvus-issuer` (created below). > Milvus has no `autoOps` field, so there is no `autoOps.disabled` toggle — recommendations are always generated for the conditions above when their triggers are met. > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/recommendation/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/recommendation/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). +## Create the Issuer + +The sample manifests request their certificates from the `milvus-issuer` `Issuer`. Without it, the Milvus pod stays in `ContainerCreating` because its certificate secret is never created. Create a CA secret and the issuer, as in the [TLS guide](/docs/guides/milvus/tls/guide.md#create-a-cert-manager-issuer): + +```bash +$ openssl genrsa -out ca.key 2048 +$ openssl req -x509 -new -nodes -key ca.key -subj "/CN=milvus-ca/O=kubedb" -days 3650 -out ca.crt +$ kubectl create secret tls milvus-ca --cert=ca.crt --key=ca.key -n demo +secret/milvus-ca created + +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/tls/yamls/issuer.yaml +issuer.cert-manager.io/milvus-issuer created +``` + ## Deploy Milvus with Recommendation-Triggering Settings The sample manifests deliberately use version **`2.6.9`** (so an `UpdateVersion` recommendation can point to a newer catalog version such as `2.6.11`), a short `authSecret.rotateAfter` of `15m` (so a `RotateAuth` recommendation is generated), and short-lived certificates (so a `ReconfigureTLS` recommendation is generated as expiry approaches). @@ -83,7 +99,12 @@ spec: duration: 61m ``` -The distributed sample (`distributed.yaml`) is equivalent, targeting `milvus-cluster` with `streamingnode` storage. +The distributed sample (`distributed.yaml`) is equivalent, targeting `milvus-cluster` with `streamingnode` storage. Deploy it the same way if you want recommendations for a distributed database: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/recommendation/yamls/distributed.yaml +milvus.kubedb.com/milvus-cluster created +``` - `spec.authSecret.rotateAfter: 15m` — the credential is considered due for rotation 15 minutes after `activeFrom`. - `spec.tls.certificates[].duration: 61m` — short certificate lifetimes so renewal/rotation recommendations surface quickly. @@ -94,7 +115,13 @@ $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" > milvus.kubedb.com/milvus-standalone created ``` -Wait for the database to become `Ready`. +Wait for the database to become `Ready`: + +```bash +$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w +``` + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. ## Observe the Generated Recommendations @@ -169,7 +196,11 @@ $ kubectl get recommendation -n demo -o jsonpath='{.spec.operation}' | ku ```bash $ kubectl delete recommendation -n demo --all -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster +$ kubectl delete issuer -n demo milvus-issuer +$ kubectl delete secret -n demo milvus-ca $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/recommendation/yamls/distributed.yaml b/docs/guides/milvus/recommendation/yamls/distributed.yaml index 9e92864016..d7afcb883b 100644 --- a/docs/guides/milvus/recommendation/yamls/distributed.yaml +++ b/docs/guides/milvus/recommendation/yamls/distributed.yaml @@ -7,7 +7,7 @@ spec: version: "2.6.9" objectStorage: configSecret: - name: "milvus-storage-config" + name: "milvus-cluster-storage-config" topology: mode: Distributed distributed: diff --git a/docs/guides/milvus/reconfigure-tls/guide.md b/docs/guides/milvus/reconfigure-tls/guide.md index 280ea78fb5..2f177e04ca 100644 --- a/docs/guides/milvus/reconfigure-tls/guide.md +++ b/docs/guides/milvus/reconfigure-tls/guide.md @@ -25,7 +25,7 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to add, ro - Install [cert-manager](https://cert-manager.io/docs/installation/) in your cluster — KubeDB uses it to manage Milvus certificates. -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/reconfigure-tls/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/reconfigure-tls/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). @@ -62,6 +62,26 @@ issuer.cert-manager.io/milvus-issuer created The four TLS operations below are demonstrated on the standalone database `milvus-standalone`. (For the distributed database, point `spec.databaseRef.name` at `milvus-cluster` — see the [distributed section](#reconfigure-tls--distributed-milvus).) +### Deploy a Milvus Without TLS + +Start from a database without TLS. The [standalone quickstart](/docs/guides/milvus/quickstart/standalone.md) manifest has no `spec.tls`, so deploy it and wait until it is `Ready`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/standalone.yaml +milvus.kubedb.com/milvus-standalone created + +$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w +``` + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. + +Without TLS, the AppBinding scheme is `http`: + +```bash +$ kubectl get appbinding milvus-standalone -n demo -o jsonpath='{.spec.clientConfig.service.scheme}' +http +``` + ### 1. Add TLS Add TLS to a database that does not currently have it: @@ -157,9 +177,10 @@ NAME TYPE STATUS AGE mvops-rotate ReconfigureTLS Successful 52s ``` -The server certificate serial number changes, confirming the certificate was re-issued: +The server certificate serial number changes, confirming the certificate was re-issued. Run this before and after applying the OpsRequest: ```bash +$ kubectl get secret milvus-standalone-server-cert -n demo -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -serial # before rotation serial=7E91C774E29D1C7F9EF578F956480A55C09DAC27 # after rotation @@ -168,7 +189,34 @@ serial=391AEF6EC65BFF56119784DFB629C080C4FB5FB3 ### 3. Change the Issuer -Point the database at a different issuer (here, `mv-new-issuer`, backed by a different CA). Create the new issuer first (same procedure as above, with its own CA secret): +Point the database at a different issuer (here, `mv-new-issuer`, backed by a different CA). Create the new CA, its secret `mv-new-ca`, and the issuer first: + +```bash +$ openssl genrsa -out mvnew-ca.key 2048 +$ openssl req -x509 -new -nodes -key mvnew-ca.key -subj "/CN=mvnew-ca/O=kubedb" -days 3650 -out mvnew-ca.crt +$ kubectl create secret tls mv-new-ca --cert=mvnew-ca.crt --key=mvnew-ca.key -n demo +secret/mv-new-ca created +``` + +`new-issuer.yaml` + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: mv-new-issuer + namespace: demo +spec: + ca: + secretName: mv-new-ca +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/reconfigure-tls/yamls/new-issuer.yaml +issuer.cert-manager.io/mv-new-issuer created +``` + +Then change the issuer: `reconfigureTls-add-new-issuer-standalone.yaml` @@ -204,10 +252,13 @@ The database's issuer reference is updated and the new certificate chains to the $ kubectl get milvuses.kubedb.com milvus-standalone -n demo -o jsonpath='{.spec.tls.issuerRef}' {"apiGroup":"cert-manager.io","kind":"Issuer","name":"mv-new-issuer"} -# certificate issuer before: issuer=CN=milvus-ca, O=kubedb -# certificate issuer after: issuer=CN=mvnew-ca, O=kubedb +$ kubectl get secret milvus-standalone-server-cert -n demo -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -issuer +# before: issuer=CN=milvus-ca, O=kubedb +# after: issuer=CN=mvnew-ca, O=kubedb ``` +Clients must use the new CA (`ca.crt` in the `milvus-standalone-server-cert` secret) and the new client certificate from `milvus-standalone-client-cert`; certificates issued by the old CA are no longer accepted. + ### 4. Remove TLS Remove TLS from the database: @@ -252,9 +303,24 @@ $ kubectl get secret -n demo | grep -E 'milvus-standalone-(server|client)-cert' ## Reconfigure TLS — Distributed Milvus -The same four operations apply to a distributed Milvus; only `spec.databaseRef.name` differs (`milvus-cluster`). The operator issues/rotates the `server` and `client` certificates and propagates them to every distributed role. +The same four operations apply to a distributed Milvus; only `spec.databaseRef.name` differs (`milvus-cluster`). The operator issues/rotates the `server` and `client` certificates and propagates them to every distributed role, restarting every role's pods (`mixcoord`, `datanode`, `querynode`, `streamingnode`, `proxy`, and any extra replicas) one at a time — so each flow takes proportionally longer. + +Deploy the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md) manifest, which has no `spec.tls`, and wait until it is `Ready`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/distributed.yaml +milvus.kubedb.com/milvus-cluster created + +$ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w +``` + +The distributed OpsRequests use the same names as the standalone ones, so delete the standalone OpsRequests first: + +```bash +$ kubectl delete milvusopsrequest -n demo mvops-add-tls mvops-rotate mv-change-issuer mvops-remove +``` -For example, adding TLS to the distributed database: +**1. Add TLS** (distributed): `reconfigureTls-add-distributed.yaml` @@ -279,27 +345,12 @@ spec: mode: TLS ``` -The `reconfigureTls-rotate-distributed.yaml`, `reconfigureTls-add-new-issuer-distributed.yaml` and `reconfigureTls-remove-distributed.yaml` files mirror the standalone ones, targeting `milvus-cluster`. - -On the distributed database the operator drives each flow exactly as for standalone, restarting every role's pods (`mixcoord`, `datanode`, `querynode`, `streamingnode`, `proxy`, and any extra replicas) one at a time — so each flow takes proportionally longer. - -**Remove TLS** (distributed): +The `server`/`client` certificate secrets are created and mounted into every role's pods: ```bash -$ kubectl get milvusopsrequest mvops-remove -n demo -NAME TYPE STATUS AGE -mvops-remove ReconfigureTLS Successful 11m - -# after removal: certificate secrets are deleted and the AppBinding scheme reverts to http -$ kubectl get appbinding milvus-cluster -n demo -o jsonpath='{.spec.clientConfig.service.scheme}' -http -$ kubectl get secret -n demo | grep -E 'milvus-cluster-(server|client)-cert' -# (no cert secrets) -``` - -**Add TLS** (distributed) — the `server`/`client` certificate secrets are recreated and mounted into every role's pods: +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/reconfigure-tls/yamls/reconfigureTls-add-distributed.yaml +milvusopsrequest.ops.kubedb.com/mvops-add-tls created -```bash $ kubectl get milvusopsrequest mvops-add-tls -n demo NAME TYPE STATUS AGE mvops-add-tls ReconfigureTLS Successful 9m @@ -312,7 +363,34 @@ $ kubectl get appbinding milvus-cluster -n demo -o jsonpath='{.spec.clientConfig https ``` -**Rotate certificates** and **Change issuer** behave identically to the standalone flows shown above — applying `reconfigureTls-rotate-distributed.yaml` re-issues the `server`/`client` certificates (the serial numbers change), and `reconfigureTls-add-new-issuer-distributed.yaml` repoints `spec.tls.issuerRef` to `mv-new-issuer` so new certificates chain to the new CA. +**2. Rotate certificates** and **3. Change issuer** behave identically to the standalone flows shown above. `reconfigureTls-rotate-distributed.yaml` re-issues the `server`/`client` certificates (the serial numbers change), and `reconfigureTls-add-new-issuer-distributed.yaml` repoints `spec.tls.issuerRef` to `mv-new-issuer` so new certificates chain to the new CA: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/reconfigure-tls/yamls/reconfigureTls-rotate-distributed.yaml +milvusopsrequest.ops.kubedb.com/mvops-rotate created + +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/reconfigure-tls/yamls/reconfigureTls-add-new-issuer-distributed.yaml +milvusopsrequest.ops.kubedb.com/mv-change-issuer created +``` + +Apply each one only after the previous OpsRequest is `Successful`. + +**4. Remove TLS** (distributed): + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/reconfigure-tls/yamls/reconfigureTls-remove-distributed.yaml +milvusopsrequest.ops.kubedb.com/mvops-remove created + +$ kubectl get milvusopsrequest mvops-remove -n demo +NAME TYPE STATUS AGE +mvops-remove ReconfigureTLS Successful 11m + +# after removal: certificate secrets are deleted and the AppBinding scheme reverts to http +$ kubectl get appbinding milvus-cluster -n demo -o jsonpath='{.spec.clientConfig.service.scheme}' +http +$ kubectl get secret -n demo | grep -E 'milvus-cluster-(server|client)-cert' +# (no cert secrets) +``` > On a single-node test cluster, each distributed `ReconfigureTLS` flow can take several minutes because every role (plus any scaled-out replicas) is restarted sequentially. @@ -320,7 +398,11 @@ https ```bash $ kubectl delete milvusopsrequest -n demo mvops-add-tls mvops-rotate mv-change-issuer mvops-remove -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster +$ kubectl delete issuer -n demo milvus-issuer mv-new-issuer +$ kubectl delete secret -n demo milvus-ca mv-new-ca $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/reconfigure-tls/yamls/new-issuer.yaml b/docs/guides/milvus/reconfigure-tls/yamls/new-issuer.yaml new file mode 100644 index 0000000000..b7c1546e13 --- /dev/null +++ b/docs/guides/milvus/reconfigure-tls/yamls/new-issuer.yaml @@ -0,0 +1,8 @@ +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: mv-new-issuer + namespace: demo +spec: + ca: + secretName: mv-new-ca diff --git a/docs/guides/milvus/reconfigure/guide.md b/docs/guides/milvus/reconfigure/guide.md index 56131109a4..2c796625cc 100644 --- a/docs/guides/milvus/reconfigure/guide.md +++ b/docs/guides/milvus/reconfigure/guide.md @@ -23,7 +23,7 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to reconfi - [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) - [Reconfigure Overview](/docs/guides/milvus/reconfigure/overview.md) -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). - To keep things isolated, this tutorial uses a separate namespace called `demo`: @@ -40,9 +40,14 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to reconfi ### Deploy Milvus -Deploy a standalone Milvus and wait for it to become `Ready` (see the [standalone quickstart](/docs/guides/milvus/quickstart/standalone.md)): +Deploy the [standalone quickstart](/docs/guides/milvus/quickstart/standalone.md) manifest and wait for it to become `Ready`: + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. ```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/standalone.yaml +milvus.kubedb.com/milvus-standalone created + $ kubectl get milvuses.kubedb.com -n demo milvus-standalone NAME VERSION STATUS AGE milvus-standalone 2.6.11 Ready 2m @@ -95,7 +100,6 @@ spec: maxAge: 30 queryNode: gracefulTime: 500 - restart: "false" timeout: 5m apply: IfReady ``` @@ -104,9 +108,9 @@ Here, - `spec.databaseRef.name` is the Milvus we are reconfiguring. - `spec.configuration.configSecret` references a secret whose `milvus.yaml` holds configuration. -- `spec.configuration.applyConfig` merges an inline `milvus.yaml` on top — here the final, effective values for `log` and `queryNode`. -- `spec.configuration.removeCustomConfig: true` discards any previously applied custom configuration first. -- `spec.configuration.restart: "false"` requests the configuration be applied without forcing a restart. +- `spec.configuration.applyConfig` merges an inline `milvus.yaml` on top. Keys set in both places take the `applyConfig` value, so `log.level`, `log.file.maxAge` and `queryNode.gracefulTime` end up as `info`, `30` and `500`. Keys set only in the config secret keep the secret's value, so `dataNode.segment.maxSize` becomes `400`. +- `spec.configuration.removeCustomConfig: true` discards any previously applied custom configuration (config secret reference and inline configuration) first. The `configSecret` and `applyConfig` of this same request are still applied. +- `spec.configuration.restart` is omitted, so it defaults to `auto`: because the configuration changes, the pods are restarted. Milvus does not reload these settings while running, so a restart is what makes them take effect. With `restart: "false"`, the new `milvus.yaml` is written to the pods, but the running Milvus keeps its old values until the pods restart. ```bash $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/reconfigure/yamls/reconfigure-standalone.yaml @@ -136,6 +140,10 @@ Status: Message: successfully reconciled the milvus with new configuration Reason: UpdatePetSets Type: UpdatePetSets + ... + Message: Successfully Restarted Pods after reconfiguration + Reason: RestartPods + Type: RestartPods Message: Successfully completed reconfigure milvus Reason: Successful Type: Successful @@ -143,6 +151,7 @@ Status: Events: Normal Starting Pausing Milvus databse: demo/milvus-standalone Normal UpdatePetSets successfully reconciled the milvus with new configuration + ... Normal Starting Resuming Milvus database: demo/milvus-standalone Normal Successful Successfully resumed Milvus database: demo/milvus-standalone for MilvusOpsRequest: reconfigure-1 ``` @@ -152,7 +161,8 @@ Events: The applied values are rendered into the configuration secret's `milvus.yaml`: ```bash -$ CFG=$(kubectl get secret -n demo -o name | grep -oE 'milvus-standalone-[a-f0-9]{6}' | head -1) +$ DB_UID=$(kubectl get milvuses.kubedb.com milvus-standalone -n demo -o jsonpath='{.metadata.uid}') +$ CFG=milvus-standalone-${DB_UID: -6} $ kubectl get secret $CFG -n demo -o jsonpath='{.data.milvus\.yaml}' | base64 -d | grep -A3 -E '^log:|^queryNode:' log: file: @@ -166,11 +176,51 @@ queryNode: port: 19536 ``` -The `log.level` is now `info`, `log.file.maxAge` is `30`, and `queryNode.gracefulTime` is `500` — exactly the values supplied through `applyConfig`. +The `log.level` is now `info`, `log.file.maxAge` is `30`, and `queryNode.gracefulTime` is `500` — exactly the values supplied through `applyConfig`. `dataNode.segment.maxSize: 400` from the config secret is rendered too. + +The rendered configuration secret is named `-`, as computed above. The database records the custom configuration in `spec.configuration`: `secretName` is `mv-configuration`, and `inline` holds the `applyConfig` values under the `milvus.yaml` key: + +```bash +$ kubectl get milvuses.kubedb.com milvus-standalone -n demo -o jsonpath='{.spec.configuration}' +``` + +### Verify the Running Milvus + +The rendered secret only shows what Milvus *will* load. Check that the pod was restarted (its `AGE` is reset) and that the running Milvus reads the new file: + +```bash +$ kubectl get pods -n demo milvus-standalone-0 + +$ kubectl exec -n demo milvus-standalone-0 -c milvus -- grep -A3 -E '^log:|^queryNode:' /milvus/configs/milvus.yaml +``` + +You can also ask Milvus for its effective configuration through the management endpoint on port `9091`. Port-forward it in a separate terminal (`kubectl port-forward svc/milvus-standalone -n demo 9091:9091`), then run: + +```bash +$ PASSWORD=$(kubectl get secret milvus-standalone-auth -n demo -o jsonpath='{.data.password}' | base64 -d) +$ curl -s -H "Authorization: Bearer root:${PASSWORD}" http://localhost:9091/api/v1/_cluster/configs | jq . +``` + +The response lists each effective setting with lowercase keys. After the restart, `log.file.maxage` is `30`, `querynode.gracefultime` is `500` and `datanode.segment.maxsize` is `400`. ## Reconfigure Distributed Milvus -For a distributed Milvus, the flow is identical; only `spec.databaseRef.name` points at the distributed database (`milvus-cluster`). The same `milvus.yaml` configuration is rendered into the configuration secret and propagated to every distributed role (`mixcoord`, `datanode`, `querynode`, `streamingnode`, `proxy`). +For a distributed Milvus, the flow is identical; only `spec.databaseRef.name` points at the distributed database (`milvus-cluster`). The same `milvus.yaml` configuration is rendered into the configuration secret and propagated to every distributed role (`mixcoord`, `datanode`, `querynode`, `streamingnode`, `proxy`), whose pods are then restarted one at a time. + +Deploy the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md) manifest and wait for it to become `Ready`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/distributed.yaml +milvus.kubedb.com/milvus-cluster created + +$ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w +``` + +The distributed sample reuses the OpsRequest name `reconfigure-1`, so delete the standalone OpsRequest first. The `mv-configuration` secret has the same content in both samples; if it still exists, `kubectl apply` reports it as `unchanged`. + +```bash +$ kubectl delete milvusopsrequest -n demo reconfigure-1 +``` `reconfigure-distributed.yaml` @@ -215,7 +265,6 @@ spec: maxAge: 30 queryNode: gracefulTime: 500 - restart: "false" timeout: 5m apply: IfReady ``` @@ -233,7 +282,8 @@ reconfigure-1 Reconfigure Successful 21s The applied configuration is rendered into the cluster's configuration secret and propagated to all roles: ```bash -$ CFG=$(kubectl get secret -n demo -o name | grep -oE 'milvus-cluster-[a-f0-9]{6}' | head -1) +$ DB_UID=$(kubectl get milvuses.kubedb.com milvus-cluster -n demo -o jsonpath='{.metadata.uid}') +$ CFG=milvus-cluster-${DB_UID: -6} $ kubectl get secret $CFG -n demo -o jsonpath='{.data.milvus\.yaml}' | base64 -d | grep -A2 -E '^log:|^queryNode:|level:' log: file: @@ -247,14 +297,16 @@ queryNode: port: 21123 ``` -As with standalone, `log.level` is now `info`, `log.file.maxAge` is `30`, and `queryNode.gracefulTime` is `500`. +As with standalone, `log.level` is now `info`, `log.file.maxAge` is `30`, and `queryNode.gracefulTime` is `500`. To check the running Milvus, use the same management-endpoint request through `svc/milvus-cluster` (backed by the proxy), or read `/milvus/configs/milvus.yaml` in any role pod, for example `milvus-cluster-querynode-0`. ## Cleaning up ```bash $ kubectl delete milvusopsrequest -n demo reconfigure-1 +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster $ kubectl delete secret -n demo mv-configuration -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/reconfigure/overview.md b/docs/guides/milvus/reconfigure/overview.md index e6aadf0a67..a8fb551994 100644 --- a/docs/guides/milvus/reconfigure/overview.md +++ b/docs/guides/milvus/reconfigure/overview.md @@ -28,17 +28,19 @@ Milvus is configured through a single configuration file that **must** be named A `MilvusOpsRequest` of type `Reconfigure` lets you change that custom configuration on a running database. The `spec.configuration` block supports: -- **`configSecret`** — a reference to a `Secret` whose `milvus.yaml` key holds the new configuration. -- **`applyConfig`** — an inline map; the value under the `milvus.yaml` key is merged into the existing configuration. This is convenient for small, incremental changes. -- **`removeCustomConfig`** — when `true`, the previously applied custom configuration is discarded before the new configuration is applied. -- **`restart`** — `auto` (default), `true`, or `false`. Controls whether the pods are restarted after the configuration change. +- **`configSecret`** — a reference to a `Secret` whose `milvus.yaml` key holds the new configuration. It becomes the database's `spec.configuration.secretName`. +- **`applyConfig`** — an inline map; the value under the `milvus.yaml` key is merged into the existing inline configuration (`spec.configuration.inline`). This is convenient for small, incremental changes. +- **`removeCustomConfig`** — when `true`, the previously applied custom configuration (both the config secret reference and the inline configuration) is discarded first. A `configSecret` and `applyConfig` in the same request are still applied on top of the defaults. +- **`restart`** — `auto` (default), `true`, or `false`. Controls whether the pods are restarted after the configuration change. Milvus does not reload these settings while running, so a new configuration takes effect only when the pods restart. With `auto`, the pods are restarted whenever the request changes the configuration. With `false`, the new `milvus.yaml` is written to the pods, but the running Milvus keeps the old values until its pods are restarted for some other reason. + +When the configuration is rendered, KubeDB's defaults come first, the config secret's `milvus.yaml` is merged over them, and the inline configuration is merged last. So a key set in both the config secret and `applyConfig` takes the `applyConfig` value, and keys set only in the config secret keep the secret's value. The flow is: 1. A user creates a `MilvusOpsRequest` of type `Reconfigure`. 2. The operator validates the request and pauses the `Milvus` database. 3. The new configuration is merged and written to the rendered configuration secret (`-`), under the `milvus.yaml` key. -4. The PetSets are reconciled and (depending on `restart`) the pods are recreated to load the new configuration. +4. The PetSets are reconciled and, unless `restart` is `false`, the pods are recreated one at a time to load the new configuration. 5. The operator resumes the database and marks the `MilvusOpsRequest` as `Successful`. > **Important:** the configuration key is always `milvus.yaml` — use that exact key in your config secret and in `applyConfig`. diff --git a/docs/guides/milvus/reconfigure/yamls/reconfigure-distributed.yaml b/docs/guides/milvus/reconfigure/yamls/reconfigure-distributed.yaml index 7d925293fe..262c0e1074 100644 --- a/docs/guides/milvus/reconfigure/yamls/reconfigure-distributed.yaml +++ b/docs/guides/milvus/reconfigure/yamls/reconfigure-distributed.yaml @@ -13,7 +13,6 @@ stringData: maxAge: 20 queryNode: gracefulTime: 10 - dataNode: segment: maxSize: 400 @@ -39,6 +38,5 @@ spec: maxAge: 30 queryNode: gracefulTime: 500 - restart: "false" timeout: 5m apply: IfReady \ No newline at end of file diff --git a/docs/guides/milvus/reconfigure/yamls/reconfigure-standalone.yaml b/docs/guides/milvus/reconfigure/yamls/reconfigure-standalone.yaml index 711a2f33b2..8f7306eb51 100644 --- a/docs/guides/milvus/reconfigure/yamls/reconfigure-standalone.yaml +++ b/docs/guides/milvus/reconfigure/yamls/reconfigure-standalone.yaml @@ -13,7 +13,6 @@ stringData: maxAge: 20 queryNode: gracefulTime: 10 - dataNode: segment: maxSize: 400 @@ -39,6 +38,5 @@ spec: maxAge: 30 queryNode: gracefulTime: 500 - restart: "false" timeout: 5m apply: IfReady \ No newline at end of file diff --git a/docs/guides/milvus/restart/guide.md b/docs/guides/milvus/restart/guide.md index 86c21bbdd6..7f95f486aa 100644 --- a/docs/guides/milvus/restart/guide.md +++ b/docs/guides/milvus/restart/guide.md @@ -22,13 +22,22 @@ KubeDB supports restarting a Milvus database via a `MilvusOpsRequest`. Restartin - [Milvus](/docs/guides/milvus/concepts/milvus.md) - [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/restart/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/restart/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). ## Restart Standalone Milvus -Deploy a standalone Milvus and wait until it is `Ready` (see the [quickstart](/docs/guides/milvus/quickstart/standalone.md)). +Deploy the [standalone quickstart](/docs/guides/milvus/quickstart/standalone.md) manifest and wait until it is `Ready`: + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/standalone.yaml +milvus.kubedb.com/milvus-standalone created + +$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w +``` ### Apply the Restart OpsRequest @@ -108,6 +117,17 @@ milvus-standalone-0 1/1 Running 0 14s For a distributed Milvus, the same `Restart` OpsRequest is used, pointing at the distributed database (`milvus-cluster`). The operator restarts the pods of every distributed role (`mixcoord`, `datanode`, `querynode`, `streamingnode`, `proxy`) one workload at a time. +Deploy the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md) manifest and wait until it is `Ready`. The distributed OpsRequest reuses the name `restart`, so delete the standalone one first: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/distributed.yaml +milvus.kubedb.com/milvus-cluster created + +$ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w + +$ kubectl delete milvusopsrequest -n demo restart +``` + `restart-distributed.yaml` ```yaml @@ -169,7 +189,9 @@ milvus-cluster-streamingnode-0 1/1 Running 0 25s ```bash $ kubectl delete milvusopsrequest -n demo restart -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/rotate-auth/guide.md b/docs/guides/milvus/rotate-auth/guide.md index 238ff20577..06dedf4226 100644 --- a/docs/guides/milvus/rotate-auth/guide.md +++ b/docs/guides/milvus/rotate-auth/guide.md @@ -23,10 +23,23 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to rotate - [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) - [Rotate Authentication Overview](/docs/guides/milvus/rotate-auth/overview.md) -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/rotate-auth/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/rotate-auth/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). +## Deploy Milvus + +This guide rotates the credentials of the databases from the quickstarts. Deploy the [standalone quickstart](/docs/guides/milvus/quickstart/standalone.md) manifest, which does not set `spec.authSecret`, and wait until it is `Ready`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/standalone.yaml +milvus.kubedb.com/milvus-standalone created + +$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w +``` + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. + ## Auth Secret Basics Milvus authentication is enabled by default. When `spec.authSecret` is omitted, KubeDB creates a `kubernetes.io/basic-auth` secret named `-auth` with a `root` user and a random password: @@ -121,15 +134,49 @@ The database now references the new secret: ```bash $ kubectl get milvuses.kubedb.com milvus-standalone -n demo -o jsonpath='{.spec.authSecret.name}' milvus-new-auth1 +``` + +The new activation time is recorded in `spec.authSecret.activeFrom` and in the `kubedb.com/auth-active-from` annotation of the **new** secret. (The old `milvus-standalone-auth` secret keeps its original annotation, from when the database was created.) -$ kubectl get secret milvus-standalone-auth -n demo -o jsonpath='{.metadata.annotations.kubedb\.com/auth-active-from}' +```bash +$ kubectl get milvuses.kubedb.com milvus-standalone -n demo -o jsonpath='{.spec.authSecret.activeFrom}' +2026-06-30T17:19:33Z + +$ kubectl get secret milvus-new-auth1 -n demo -o jsonpath='{.metadata.annotations.kubedb\.com/auth-active-from}' 2026-06-30T17:19:33Z ``` +Finally, check the credentials against Milvus itself. Port-forward the REST port in a separate terminal (`kubectl port-forward svc/milvus-standalone -n demo 8080:8080`), then try the old password, which is still stored in `milvus-standalone-auth`: + +```bash +$ OLD_PASSWORD=$(kubectl get secret milvus-standalone-auth -n demo -o jsonpath='{.data.password}' | base64 -d) +$ curl -s -X POST "http://localhost:8080/v2/vectordb/collections/list" \ + -H "Authorization: Bearer root:${OLD_PASSWORD}" \ + -H "Content-Type: application/json" -d '{}' +{"code":1800,"message":"user hasn't authenticated"} +``` + +The same request with `Bearer root:NewPassword1` succeeds (`"code":0`) and lists the collections. + ## Rotate Distributed Milvus For a distributed Milvus, point `spec.databaseRef.name` at the distributed database (`milvus-cluster`). The credential is updated dynamically inside every distributed role. +Deploy the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md) manifest and wait until it is `Ready`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/distributed.yaml +milvus.kubedb.com/milvus-cluster created + +$ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w +``` + +The distributed sample reuses the OpsRequest name of the standalone one, so delete the standalone OpsRequest first. If the `milvus-new-auth1` secret from the standalone section still exists, `kubectl apply` reports it as `unchanged` instead of `created`. + +```bash +$ kubectl delete milvusopsrequest -n demo milvus-rotate-auth-user-secret +``` + `rotate-auth-distributed.yaml` ```yaml @@ -204,8 +251,10 @@ See the [Recommendation Engine guide](/docs/guides/milvus/recommendation/guide.m ```bash $ kubectl delete milvusopsrequest -n demo milvus-rotate-auth-user-secret +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster $ kubectl delete secret -n demo milvus-new-auth1 -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/scaling/horizontal-scaling/guide.md b/docs/guides/milvus/scaling/horizontal-scaling/guide.md index c36c40151e..514165ea7b 100644 --- a/docs/guides/milvus/scaling/horizontal-scaling/guide.md +++ b/docs/guides/milvus/scaling/horizontal-scaling/guide.md @@ -25,15 +25,22 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to horizon - [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) - [Horizontal Scaling Overview](/docs/guides/milvus/scaling/horizontal-scaling/overview.md) -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/scaling/horizontal-scaling/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/scaling/horizontal-scaling/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). ## Deploy a Distributed Milvus -Deploy the distributed database (`milvus-cluster`) and wait until it is `Ready` (see the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md)). By default each role runs a single replica. +Deploy the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md) manifest (`milvus-cluster`) and wait until it is `Ready`. By default each role runs a single replica. + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. ```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/distributed.yaml +milvus.kubedb.com/milvus-cluster created + +$ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w + $ kubectl get petset milvus-cluster-proxy milvus-cluster-streamingnode -n demo -o custom-columns=NAME:.metadata.name,REPLICAS:.spec.replicas NAME REPLICAS milvus-cluster-proxy 1 @@ -42,7 +49,7 @@ milvus-cluster-streamingnode 1 ## Apply the HorizontalScaling OpsRequest -The sample changes only `proxy` and `streamingnode`. (The provided sample requests `1` for each; because the default is already `1`, this walkthrough requests `2` to demonstrate a scale-up.) +The sample scales `proxy` and `streamingnode` from `1` to `2` replicas. `horizontal-scaling-distributed.yaml` @@ -64,8 +71,10 @@ spec: Here, `spec.horizontalScaling.topology` carries the desired replica count per role. The API also accepts `mixcoord`, `querynode` and `dataNode`; this sample only scales `proxy` and `streamingnode`, but the other roles are scaled the same way. +> The data node key is `dataNode` (camelCase) in `spec.horizontalScaling.topology`, unlike every other role key and unlike `spec.verticalScaling`, where it is `datanode`. `datanode` is rejected here as an unknown field. + ```bash -$ kubectl apply -f horizontal-scaling-distributed.yaml +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/scaling/horizontal-scaling/yamls/horizontal-scaling-distributed.yaml milvusopsrequest.ops.kubedb.com/milvus-hscale-up created ``` @@ -111,12 +120,40 @@ milvus-cluster-streamingnode-0 1/1 Running 0 119s milvus-cluster-streamingnode-1 1/1 Running 0 18s ``` -> Scaling **down** works the same way — set lower replica counts in `spec.horizontalScaling.topology`. +### Scale Down + +Scaling **down** works the same way — set lower replica counts in `spec.horizontalScaling.topology`. `horizontal-scaling-down-distributed.yaml` scales both roles back to `1`: + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: MilvusOpsRequest +metadata: + name: milvus-hscale-down + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: milvus-cluster + horizontalScaling: + topology: + proxy: 1 + streamingnode: 1 +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/scaling/horizontal-scaling/yamls/horizontal-scaling-down-distributed.yaml +milvusopsrequest.ops.kubedb.com/milvus-hscale-down created + +$ kubectl get milvusopsrequest milvus-hscale-down -n demo +``` + +When a `streamingnode` is removed, its PVC is deleted too. Right after a scale-down reports `Successful`, the `Milvus` object can show `Critical` for about a minute while Milvus rebalances, then returns to `Ready`. ## Cleaning up ```bash -$ kubectl delete milvusopsrequest -n demo milvus-hscale-up +$ kubectl delete milvusopsrequest -n demo milvus-hscale-up milvus-hscale-down +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" $ kubectl delete milvus.kubedb.com -n demo milvus-cluster $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/scaling/horizontal-scaling/overview.md b/docs/guides/milvus/scaling/horizontal-scaling/overview.md index cde679c802..061afd3515 100644 --- a/docs/guides/milvus/scaling/horizontal-scaling/overview.md +++ b/docs/guides/milvus/scaling/horizontal-scaling/overview.md @@ -35,9 +35,9 @@ spec: type: HorizontalScaling horizontalScaling: topology: - proxy: 1 - streamingnode: 1 - # mixcoord / querynode / datanode are also supported by the API + proxy: 2 + streamingnode: 2 + # mixcoord / querynode / dataNode are also supported by the API ``` The flow is: @@ -48,6 +48,6 @@ The flow is: 4. The operator waits for the affected roles to become healthy. 5. The operator resumes the database and marks the `MilvusOpsRequest` as `Successful`. -The `spec.horizontalScaling.topology` API accepts `proxy`, `mixcoord`, `querynode`, `streamingnode` and `dataNode`. The sample used in the guide only scales `proxy` and `streamingnode`; the other roles are scaled the same way. +The `spec.horizontalScaling.topology` API accepts `proxy`, `mixcoord`, `querynode`, `streamingnode` and `dataNode`. Note the camelCase `dataNode`: vertical scaling uses `datanode` for the same role. The sample used in the guide only scales `proxy` and `streamingnode`; the other roles are scaled the same way. In the next doc, we will see a step-by-step guide on horizontally scaling a distributed Milvus database. diff --git a/docs/guides/milvus/scaling/horizontal-scaling/yamls/horizontal-scaling-distributed.yaml b/docs/guides/milvus/scaling/horizontal-scaling/yamls/horizontal-scaling-distributed.yaml index 43aaa91903..684e772ca9 100644 --- a/docs/guides/milvus/scaling/horizontal-scaling/yamls/horizontal-scaling-distributed.yaml +++ b/docs/guides/milvus/scaling/horizontal-scaling/yamls/horizontal-scaling-distributed.yaml @@ -9,5 +9,5 @@ spec: name: milvus-cluster horizontalScaling: topology: - proxy: 1 - streamingnode: 1 \ No newline at end of file + proxy: 2 + streamingnode: 2 \ No newline at end of file diff --git a/docs/guides/milvus/scaling/horizontal-scaling/yamls/horizontal-scaling-down-distributed.yaml b/docs/guides/milvus/scaling/horizontal-scaling/yamls/horizontal-scaling-down-distributed.yaml new file mode 100644 index 0000000000..bb1969409a --- /dev/null +++ b/docs/guides/milvus/scaling/horizontal-scaling/yamls/horizontal-scaling-down-distributed.yaml @@ -0,0 +1,13 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: MilvusOpsRequest +metadata: + name: milvus-hscale-down + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: milvus-cluster + horizontalScaling: + topology: + proxy: 1 + streamingnode: 1 diff --git a/docs/guides/milvus/scaling/vertical-scaling/guide.md b/docs/guides/milvus/scaling/vertical-scaling/guide.md index 61d26420d0..e872a46ed6 100644 --- a/docs/guides/milvus/scaling/vertical-scaling/guide.md +++ b/docs/guides/milvus/scaling/vertical-scaling/guide.md @@ -23,15 +23,22 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to update - [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) - [Vertical Scaling Overview](/docs/guides/milvus/scaling/vertical-scaling/overview.md) -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/scaling/vertical-scaling/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/scaling/vertical-scaling/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). ## Vertical Scaling Standalone Milvus -Deploy a standalone Milvus and wait until it is `Ready`. By default the standalone workload requests `cpu: 500m` / `memory: 1Gi`: +Deploy the [standalone quickstart](/docs/guides/milvus/quickstart/standalone.md) manifest and wait until it is `Ready`. By default the standalone workload requests `cpu: 500m` / `memory: 1Gi`: + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. ```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/standalone.yaml +milvus.kubedb.com/milvus-standalone created + +$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w + $ kubectl get petset milvus-standalone -n demo -o jsonpath='{.spec.template.spec.containers[0].resources}' {"limits":{"memory":"1Gi"},"requests":{"cpu":"500m","memory":"1Gi"}} ``` @@ -121,6 +128,12 @@ To resize the Pods **without a restart**, set `spec.verticalScaling.mode` to `In `MilvusOpsRequest`. The operator resizes the running containers via the Kubernetes `pods/resize` subresource and only restarts a Pod if its Node cannot accommodate the new resources. +This sample continues from the previous step and raises the standalone workload from `1` CPU / `2Gi` to `2` CPU / `3Gi`. Before applying it, record the Pod's UID, restart count and current resources: + +```bash +$ kubectl get pod milvus-standalone-0 -n demo -o jsonpath='{.metadata.uid} {.status.containerStatuses[0].restartCount} {.status.containerStatuses[0].resources}{"\n"}' +``` + `vertical-scaling-standalone-inplace.yaml` ```yaml @@ -138,11 +151,11 @@ spec: node: resources: requests: - memory: "2Gi" - cpu: "1" + memory: "3Gi" + cpu: "2" limits: - memory: "2Gi" - cpu: "1" + memory: "3Gi" + cpu: "2" timeout: 5m apply: IfReady ``` @@ -152,11 +165,26 @@ $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" > milvusopsrequest.ops.kubedb.com/vertical-scaling-standalone-inplace created ``` -Apply it the same way as above; the resources update in place with no Pod restart. +Once the OpsRequest is `Successful`, run the same command again. The Pod UID and restart count are unchanged, which shows the Pod was neither recreated nor restarted, while its resources now report `2` CPU / `3Gi`: + +```bash +$ kubectl get milvusopsrequest vertical-scaling-standalone-inplace -n demo + +$ kubectl get pod milvus-standalone-0 -n demo -o jsonpath='{.metadata.uid} {.status.containerStatuses[0].restartCount} {.status.containerStatuses[0].resources}{"\n"}' +``` ## Vertical Scaling Distributed Milvus -For a distributed Milvus, set the resources per role under `spec.verticalScaling`. You can scale several roles in a single OpsRequest. The sample below scales `mixcoord` and `proxy`: +For a distributed Milvus, set the resources per role under `spec.verticalScaling`. You can scale several roles in a single OpsRequest. The sample below scales `mixcoord` and `proxy`. + +Deploy the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md) manifest and wait until it is `Ready`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/distributed.yaml +milvus.kubedb.com/milvus-cluster created + +$ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w +``` `vertical-scaling-distributed.yaml` @@ -191,7 +219,7 @@ spec: apply: IfReady ``` -The same approach applies to `datanode`, `querynode` and `streamingnode`. +The same approach applies to `datanode`, `querynode` and `streamingnode`. (In `spec.verticalScaling` the key is `datanode`; horizontal scaling uses `dataNode` for the same role.) ```bash $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/scaling/vertical-scaling/yamls/vertical-scaling-distributed.yaml @@ -215,8 +243,10 @@ $ kubectl get petset milvus-cluster-proxy -n demo -o jsonpath='{.spec.template.s ## Cleaning up ```bash -$ kubectl delete milvusopsrequest -n demo vertical-scaling-standalone -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone +$ kubectl delete milvusopsrequest -n demo vertical-scaling-standalone vertical-scaling-standalone-inplace vertical-scaling +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/scaling/vertical-scaling/yamls/vertical-scaling-standalone-inplace.yaml b/docs/guides/milvus/scaling/vertical-scaling/yamls/vertical-scaling-standalone-inplace.yaml index 8b295bca7f..bebaa6af5e 100644 --- a/docs/guides/milvus/scaling/vertical-scaling/yamls/vertical-scaling-standalone-inplace.yaml +++ b/docs/guides/milvus/scaling/vertical-scaling/yamls/vertical-scaling-standalone-inplace.yaml @@ -12,10 +12,10 @@ spec: node: resources: requests: - memory: "2Gi" - cpu: "1" + memory: "3Gi" + cpu: "2" limits: - memory: "2Gi" - cpu: "1" + memory: "3Gi" + cpu: "2" timeout: 5m apply: IfReady diff --git a/docs/guides/milvus/storage-migration/guide.md b/docs/guides/milvus/storage-migration/guide.md index 0c4154179e..8738c84e24 100644 --- a/docs/guides/milvus/storage-migration/guide.md +++ b/docs/guides/milvus/storage-migration/guide.md @@ -22,7 +22,7 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to migrate - [Milvus](/docs/guides/milvus/concepts/milvus.md) - [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). - You need **at least two** `StorageClass`es — the current one and the target one. This guide migrates from `local-path` to `longhorn-custom`: @@ -37,9 +37,16 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to migrate ## Storage Migration — Standalone Milvus -Deploy a standalone Milvus on `local-path` and wait until it is `Ready`: +Deploy the [standalone quickstart](/docs/guides/milvus/quickstart/standalone.md) manifest, which uses `local-path`, and wait until it is `Ready`: + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. ```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/quickstart/yamls/standalone.yaml +milvus.kubedb.com/milvus-standalone created + +$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w + $ kubectl get pvc -n demo -l app.kubernetes.io/instance=milvus-standalone NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-milvus-standalone-0 Bound pvc-... 1Gi RWO local-path 14m @@ -99,7 +106,7 @@ Status: Phase: Successful ``` -During migration, the operator runs a migrator job to copy the data, recreates the PVC on the new `StorageClass`, and recreates the pod. +During migration, the operator runs a migrator job to copy the data, recreates the PVC on the new `StorageClass`, and recreates the pod. A standalone Milvus has a single pod, so the database is unavailable while it is migrated (about a minute and a half in our test). ### Verify the Migration @@ -116,7 +123,22 @@ longhorn-custom ## Storage Migration — Distributed Milvus -For a distributed Milvus, migration targets the workloads that carry persistent storage — i.e. `streamingnode`. Point `spec.databaseRef.name` at `milvus-cluster`: +For a distributed Milvus, migration targets the workloads that carry persistent storage — i.e. `streamingnode`. Point `spec.databaseRef.name` at `milvus-cluster`. + +This example uses two `streamingnode` replicas on `local-path`. `storage-migration/yamls/milvus-distributed.yaml` is the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md) manifest with `streamingnode.replicas: 2`. Deploy it and wait until it is `Ready`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/storage-migration/yamls/milvus-distributed.yaml +milvus.kubedb.com/milvus-cluster created + +$ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w +``` + +The distributed OpsRequest reuses the name of the standalone one, so delete the standalone OpsRequest first: + +```bash +$ kubectl delete milvusopsrequest -n demo storage-migration +``` `storage-migration-distributed.yaml` @@ -145,6 +167,13 @@ data-milvus-cluster-streamingnode-0 1Gi local-path data-milvus-cluster-streamingnode-1 1Gi local-path ``` +Apply the OpsRequest. The `streamingnode` replicas are migrated one at a time, so one of them keeps running throughout; the other roles are not restarted: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/storage-migration/yamls/storage-migration-distributed.yaml +milvusopsrequest.ops.kubedb.com/storage-migration created +``` + After the migration completes: ```bash @@ -161,13 +190,15 @@ $ kubectl get milvuses.kubedb.com milvus-cluster -n demo -o jsonpath='{.spec.top longhorn-custom ``` -(This example was run with `streamingnode` scaled to two replicas; both PVCs are migrated.) +Both `streamingnode` PVCs are migrated. ## Cleaning up ```bash $ kubectl delete milvusopsrequest -n demo storage-migration -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/storage-migration/yamls/milvus-distributed.yaml b/docs/guides/milvus/storage-migration/yamls/milvus-distributed.yaml new file mode 100644 index 0000000000..688aac7186 --- /dev/null +++ b/docs/guides/milvus/storage-migration/yamls/milvus-distributed.yaml @@ -0,0 +1,23 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus-cluster + namespace: demo +spec: + version: "2.6.11" + objectStorage: + configSecret: + name: "milvus-cluster-storage-config" + topology: + mode: Distributed + distributed: + streamingnode: + replicas: 2 + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + storageClassName: local-path + resources: + requests: + storage: 1Gi diff --git a/docs/guides/milvus/tls/guide.md b/docs/guides/milvus/tls/guide.md index 9a108807da..6c35708150 100644 --- a/docs/guides/milvus/tls/guide.md +++ b/docs/guides/milvus/tls/guide.md @@ -24,7 +24,7 @@ This guide will show you how to deploy a Milvus database with TLS/SSL enabled fr - [Milvus](/docs/guides/milvus/concepts/milvus.md) - [TLS Overview](/docs/guides/milvus/tls/overview.md) -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/tls/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/tls/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). @@ -61,7 +61,32 @@ NAME READY AGE milvus-issuer True 5s ``` -> A `ClusterIssuer` works the same way; a sample `cluster-issuer.yaml` (backed by secret `milvus-cluster-ca`) is included in the `yamls` folder. With a `ClusterIssuer`, set `spec.tls.issuerRef.kind: ClusterIssuer`. +### Use a ClusterIssuer Instead + +A `ClusterIssuer` works the same way, with one difference: cert-manager reads a `ClusterIssuer`'s CA secret from its cluster resource namespace, which is `cert-manager` by default, not from the database namespace. If the secret is in `demo`, the `ClusterIssuer` never becomes Ready (`Error getting keypair for CA issuer: secrets "milvus-cluster-ca" not found`). Create the CA secret in `cert-manager`, then apply the sample `cluster-issuer.yaml`: + +```bash +$ kubectl create secret tls milvus-cluster-ca --cert=ca.crt --key=ca.key -n cert-manager +secret/milvus-cluster-ca created + +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/tls/yamls/cluster-issuer.yaml +clusterissuer.cert-manager.io/milvus-cluster-issuer created + +$ kubectl get clusterissuer milvus-cluster-issuer +``` + +Then reference it from the `Milvus` manifest with `kind: ClusterIssuer`: + +```yaml +spec: + tls: + issuerRef: + name: milvus-cluster-issuer + kind: ClusterIssuer + apiGroup: "cert-manager.io" +``` + +The rest of this guide uses the namespaced `milvus-issuer`. ## Deploy a TLS-Secured Standalone Milvus @@ -107,7 +132,13 @@ $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" > milvus.kubedb.com/milvus-standalone created ``` -Wait until it is `Ready`. +Wait until it is `Ready`: + +```bash +$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w +``` + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. ## Verify TLS @@ -136,10 +167,12 @@ server.pem ### Rendered Configuration -The rendered `milvus.yaml` wires the certificates into Milvus: +The rendered `milvus.yaml` wires the certificates into Milvus. It is stored in a secret named `-`, for example `milvus-standalone-f93293`: ```bash -$ kubectl get secret -n demo -o jsonpath='{.data.milvus\.yaml}' | base64 -d | grep -A4 internaltls +$ DB_UID=$(kubectl get milvuses.kubedb.com milvus-standalone -n demo -o jsonpath='{.metadata.uid}') +$ CFG=milvus-standalone-${DB_UID: -6} +$ kubectl get secret $CFG -n demo -o jsonpath='{.data.milvus\.yaml}' | base64 -d | grep -A4 internaltls internaltls: caPemPath: /milvus/tls/ca.pem serverKeyPath: /milvus/tls/server.key @@ -147,6 +180,8 @@ internaltls: sni: milvus-standalone ``` +> The rendered `milvus.yaml` also contains the root password in plain text (`common.security.defaultRootPassword`), so treat this secret as sensitive. + ### AppBinding Scheme Because TLS is enabled, the AppBinding connection scheme is `https`: @@ -250,7 +285,7 @@ spec: version: "2.6.11" objectStorage: configSecret: - name: "milvus-storage-config" + name: "milvus-cluster-storage-config" topology: mode: Distributed distributed: @@ -274,6 +309,11 @@ spec: mode: TLS ``` +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/tls/yamls/distributed.yaml +milvus.kubedb.com/milvus-cluster created +``` + After it becomes `Ready`, the certificate secrets exist, the certificates are mounted into every role's pods, and the AppBinding scheme is `https`: ```bash @@ -310,12 +350,18 @@ You should see a successful TLS handshake. ## Cleaning up ```bash -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster $ kubectl delete issuer -n demo milvus-issuer $ kubectl delete secret -n demo milvus-ca +$ kubectl delete clusterissuer milvus-cluster-issuer +$ kubectl delete secret -n cert-manager milvus-cluster-ca $ kubectl delete ns demo ``` +Skip the two `ClusterIssuer` commands if you did not create it. + ## Next Steps - Add, rotate or remove TLS on a running database with [Reconfigure TLS](/docs/guides/milvus/reconfigure-tls/guide.md). diff --git a/docs/guides/milvus/tls/yamls/distributed.yaml b/docs/guides/milvus/tls/yamls/distributed.yaml index 4f39844611..2c1c52cd69 100644 --- a/docs/guides/milvus/tls/yamls/distributed.yaml +++ b/docs/guides/milvus/tls/yamls/distributed.yaml @@ -7,7 +7,7 @@ spec: version: "2.6.11" objectStorage: configSecret: - name: "milvus-storage-config" + name: "milvus-cluster-storage-config" topology: mode: Distributed distributed: @@ -20,13 +20,6 @@ spec: resources: requests: storage: 1Gi - monitor: - agent: prometheus.io/operator - prometheus: - serviceMonitor: - labels: - release: prometheus - interval: 10s tls: issuerRef: name: milvus-issuer diff --git a/docs/guides/milvus/tls/yamls/standalone.yaml b/docs/guides/milvus/tls/yamls/standalone.yaml index a5e2e7b98f..13c3d94666 100644 --- a/docs/guides/milvus/tls/yamls/standalone.yaml +++ b/docs/guides/milvus/tls/yamls/standalone.yaml @@ -18,13 +18,6 @@ spec: resources: requests: storage: 1Gi - monitor: - agent: prometheus.io/operator - prometheus: - serviceMonitor: - labels: - release: prometheus - interval: 10s tls: issuerRef: name: milvus-issuer diff --git a/docs/guides/milvus/update-version/guide.md b/docs/guides/milvus/update-version/guide.md index c4dd5e8847..03ecae13aa 100644 --- a/docs/guides/milvus/update-version/guide.md +++ b/docs/guides/milvus/update-version/guide.md @@ -23,7 +23,7 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to update - [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) - [Update Version Overview](/docs/guides/milvus/update-version/overview.md) -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). > Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/update-version/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/update-version/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). @@ -31,17 +31,26 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to update ```bash $ kubectl get milvusversions -NAME VERSION DB_IMAGE DEPRECATED AGE -2.6.11 2.6.11 ghcr.io/appscode-images/milvus:2.6.11 11h -2.6.7 2.6.7 ghcr.io/appscode-images/milvus:2.6.7 11h -2.6.9 2.6.9 ghcr.io/appscode-images/milvus:2.6.9 11h +NAME VERSION DB_IMAGE DEPRECATED AGE +2.6.11 2.6.11 ghcr.io/appscode-images/milvus:2.6.11 21h +2.6.15 2.6.15 ghcr.io/appscode-images/milvus:2.6.15 21h +2.6.15-gpu 2.6.15 ghcr.io/appscode-images/milvus:2.6.15-gpu 21h +2.6.7 2.6.7 ghcr.io/appscode-images/milvus:2.6.7 21h +2.6.9 2.6.9 ghcr.io/appscode-images/milvus:2.6.9 21h +3.0.0 3.0.0 ghcr.io/appscode-images/milvus:3.0.0 21h +3.0.0-gpu 3.0.0 ghcr.io/appscode-images/milvus:3.0.0-gpu 21h ``` ## Update Version of Standalone Milvus -Deploy a standalone Milvus at version `2.6.9` and wait until it is `Ready`: +Deploy a standalone Milvus at version `2.6.9` and wait until it is `Ready`. `update-version/yamls/standalone.yaml` is the [standalone quickstart](/docs/guides/milvus/quickstart/standalone.md) manifest with `spec.version: "2.6.9"` (the quickstart itself uses `2.6.11`, which would make the update below a no-op): + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. ```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/update-version/yamls/standalone.yaml +milvus.kubedb.com/milvus-standalone created + $ kubectl get milvuses.kubedb.com milvus-standalone -n demo NAME VERSION STATUS AGE milvus-standalone 2.6.9 Ready 46s @@ -112,10 +121,31 @@ NAME VERSION STATUS AGE milvus-standalone 2.6.11 Ready 2m3s ``` +The pod now runs the `2.6.11` image: + +```bash +$ kubectl get pods -n demo -l app.kubernetes.io/instance=milvus-standalone -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image +``` + ## Update Version of Distributed Milvus For a distributed Milvus, point `spec.databaseRef.name` at the distributed database (`milvus-cluster`). The operator updates the image of every distributed role and restarts each one. +Deploy a distributed Milvus at version `2.6.9` (the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md) manifest with `spec.version: "2.6.9"`) and wait until it is `Ready`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/update-version/yamls/distributed.yaml +milvus.kubedb.com/milvus-cluster created + +$ kubectl get milvuses.kubedb.com milvus-cluster -n demo -w +``` + +The distributed sample reuses the OpsRequest name of the standalone one, so delete the standalone OpsRequest first: + +```bash +$ kubectl delete milvusopsrequest -n demo milvus-update-version +``` + `update-version-distributed.yaml` ```yaml @@ -143,13 +173,19 @@ milvusopsrequest.ops.kubedb.com/milvus-update-version created The distributed flow is mechanically identical to the standalone flow shown above: the operator validates the target `MilvusVersion`, pauses the database, updates the container image of **each** distributed role (`mixcoord`, `datanode`, `querynode`, `streamingnode`, `proxy`), restarts them one workload at a time, and resumes the database. Because `apply: IfReady` is set, the ops request runs only once the database is `Ready`, after which it reports `Successful`. -> **Test-cluster note:** the live distributed run for this guide used catalog version `2.6.9`, and on the single-node test cluster the `2.6.9` distributed instance's read-write health check (test collection creation) did not stabilize, so the database never reached `Ready` and the `IfReady` ops request could not proceed there. The same `UpdateVersion` operation completes successfully on the standalone topology (shown above) and on a healthy distributed cluster. +Afterwards, every role pod runs the `2.6.11` image: + +```bash +$ kubectl get pods -n demo -l app.kubernetes.io/instance=milvus-cluster -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image +``` ## Cleaning up ```bash $ kubectl delete milvusopsrequest -n demo milvus-update-version -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/update-version/yamls/distributed.yaml b/docs/guides/milvus/update-version/yamls/distributed.yaml new file mode 100644 index 0000000000..4dfa4c48f5 --- /dev/null +++ b/docs/guides/milvus/update-version/yamls/distributed.yaml @@ -0,0 +1,22 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus-cluster + namespace: demo +spec: + version: "2.6.9" + objectStorage: + configSecret: + name: "milvus-cluster-storage-config" + topology: + mode: Distributed + distributed: + streamingnode: + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + storageClassName: local-path + resources: + requests: + storage: 1Gi diff --git a/docs/guides/milvus/update-version/yamls/standalone.yaml b/docs/guides/milvus/update-version/yamls/standalone.yaml new file mode 100644 index 0000000000..2ee1937e19 --- /dev/null +++ b/docs/guides/milvus/update-version/yamls/standalone.yaml @@ -0,0 +1,20 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus-standalone + namespace: demo +spec: + version: "2.6.9" + topology: + mode: Standalone + objectStorage: + configSecret: + name: "milvus-storage-config" + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + storageClassName: local-path + resources: + requests: + storage: 1Gi diff --git a/docs/guides/milvus/volume-expansion/guide.md b/docs/guides/milvus/volume-expansion/guide.md index 9046eaf53a..1831b93232 100644 --- a/docs/guides/milvus/volume-expansion/guide.md +++ b/docs/guides/milvus/volume-expansion/guide.md @@ -23,7 +23,7 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to expand - [MilvusOpsRequest](/docs/guides/milvus/concepts/milvusopsrequest.md) - [Volume Expansion Overview](/docs/guides/milvus/volume-expansion/overview.md) -- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO, creates the `milvus-storage-config` secret, and installs the etcd operator required by Milvus. +- Complete the dependency setup from [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md). It installs MinIO and creates the object-storage secrets (`milvus-storage-config` for `milvus-standalone`, `milvus-cluster-storage-config` for `milvus-cluster`). - The PVC's `StorageClass` **must** support volume expansion (`allowVolumeExpansion: true`). The base examples use `local-path`, which does **not** support expansion, so this guide uses `longhorn-custom`: @@ -36,9 +36,16 @@ This guide will show you how to use the `KubeDB` Ops-manager operator to expand ## Volume Expansion — Standalone Milvus -Deploy a standalone Milvus on an expansion-capable `StorageClass` (here `longhorn-custom`) with a `1Gi` volume: +Deploy a standalone Milvus on an expansion-capable `StorageClass` (here `longhorn-custom`) with a `1Gi` volume. `volume-expansion/yamls/milvus-standalone.yaml` is the [standalone quickstart](/docs/guides/milvus/quickstart/standalone.md) manifest with `storageClassName: longhorn-custom`: + +> If you deleted a Milvus of the same name in an earlier guide, delete its object-storage prefix first, as shown in [Prepare Dependencies](/docs/guides/milvus/quickstart/prerequisites.md#delete-a-milvus-instances-object-storage-data). Otherwise the new database can hang in `Provisioning`. ```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/volume-expansion/yamls/milvus-standalone.yaml +milvus.kubedb.com/milvus-standalone created + +$ kubectl get milvuses.kubedb.com -n demo milvus-standalone -w + $ kubectl get pvc -n demo -l app.kubernetes.io/instance=milvus-standalone -o custom-columns=NAME:.metadata.name,SIZE:.status.capacity.storage,SC:.spec.storageClassName NAME SIZE SC data-milvus-standalone-0 1Gi longhorn-custom @@ -84,7 +91,7 @@ data-milvus-standalone-0 4Gi longhorn-custom ### Online Volume Expansion -Online expansion grows the volume without taking the pod down. The example file requests `node: 4Gi`; since the volume is already `4Gi` after the offline step above, this walkthrough requests `6Gi` to demonstrate a second (online) expansion: +Online expansion grows the volume without taking the pod down. The volume is already `4Gi` after the offline step above, so this request grows it to `6Gi`. (A request for the current size is accepted and reported `Successful` without changing anything.) `volume-expansion-online-standalone.yaml` @@ -104,7 +111,7 @@ spec: ``` ```bash -$ kubectl apply -f volume-expansion-online-standalone.yaml +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/volume-expansion/yamls/volume-expansion-online-standalone.yaml milvusopsrequest.ops.kubedb.com/volume-expansion-online created $ kubectl get milvusopsrequest volume-expansion-online -n demo @@ -125,7 +132,22 @@ $ kubectl get milvuses.kubedb.com milvus-standalone -n demo -o jsonpath='{.spec. ## Volume Expansion — Distributed Milvus -For a distributed Milvus, volume expansion targets `streamingnode` — the only distributed role that carries a persistent volume. Point `spec.databaseRef.name` at `milvus-cluster` and use the `streamingnode` key: +For a distributed Milvus, volume expansion targets `streamingnode` — the only distributed role that carries a persistent volume. Point `spec.databaseRef.name` at `milvus-cluster` and use the `streamingnode` key. + +Deploy a distributed Milvus with two `streamingnode` replicas on `longhorn-custom`, each with a `1Gi` volume. `volume-expansion/yamls/milvus-distributed.yaml` is the [distributed quickstart](/docs/guides/milvus/quickstart/distributed.md) manifest with `storageClassName: longhorn-custom` and `streamingnode.replicas: 2`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/volume-expansion/yamls/milvus-distributed.yaml +milvus.kubedb.com/milvus-cluster created + +$ kubectl get milvuses.kubedb.com -n demo milvus-cluster -w +``` + +The distributed OpsRequests reuse the names of the standalone ones, so delete the standalone OpsRequests first: + +```bash +$ kubectl delete milvusopsrequest -n demo volume-expansion-offline volume-expansion-online +``` `volume-expansion-online-distributed.yaml` @@ -146,15 +168,21 @@ spec: (An `Offline` variant — `volume-expansion-offline-distributed.yaml` — uses `streamingnode: 3Gi` with `mode: Offline`.) -The operator expands the PVC of every `streamingnode` replica. Starting from `1Gi` on `longhorn-custom`, an **offline** expansion to `3Gi` followed by an **online** expansion to `4Gi`: +The operator expands the PVC of every `streamingnode` replica, one replica at a time. Starting from `1Gi` on `longhorn-custom`, an **offline** expansion to `3Gi` followed by an **online** expansion to `4Gi`. Apply the online request only after the offline one is `Successful`: ```bash # offline: streamingnode 1Gi -> 3Gi +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/volume-expansion/yamls/volume-expansion-offline-distributed.yaml +milvusopsrequest.ops.kubedb.com/volume-expansion-offline created + $ kubectl get milvusopsrequest volume-expansion-offline -n demo NAME TYPE STATUS AGE volume-expansion-offline VolumeExpansion Successful ... # online: streamingnode 3Gi -> 4Gi +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/volume-expansion/yamls/volume-expansion-online-distributed.yaml +milvusopsrequest.ops.kubedb.com/volume-expansion-online created + $ kubectl get pvc -n demo -l app.kubernetes.io/instance=milvus-cluster -o custom-columns=NAME:.metadata.name,SIZE:.status.capacity.storage NAME SIZE data-milvus-cluster-streamingnode-0 4Gi @@ -169,7 +197,9 @@ Both `streamingnode` replicas are expanded. The stateless roles (`mixcoord`, `da ```bash $ kubectl delete milvusopsrequest -n demo volume-expansion-offline volume-expansion-online -$ kubectl delete milvus.kubedb.com -n demo milvus-standalone +$ kubectl patch -n demo milvus.kubedb.com milvus-standalone -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl patch -n demo milvus.kubedb.com milvus-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +$ kubectl delete milvus.kubedb.com -n demo milvus-standalone milvus-cluster $ kubectl delete ns demo ``` diff --git a/docs/guides/milvus/volume-expansion/yamls/milvus-distributed.yaml b/docs/guides/milvus/volume-expansion/yamls/milvus-distributed.yaml new file mode 100644 index 0000000000..530ee6306c --- /dev/null +++ b/docs/guides/milvus/volume-expansion/yamls/milvus-distributed.yaml @@ -0,0 +1,23 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus-cluster + namespace: demo +spec: + version: "2.6.11" + objectStorage: + configSecret: + name: "milvus-cluster-storage-config" + topology: + mode: Distributed + distributed: + streamingnode: + replicas: 2 + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + storageClassName: longhorn-custom + resources: + requests: + storage: 1Gi diff --git a/docs/guides/milvus/volume-expansion/yamls/milvus-standalone.yaml b/docs/guides/milvus/volume-expansion/yamls/milvus-standalone.yaml new file mode 100644 index 0000000000..fbe0f1bf98 --- /dev/null +++ b/docs/guides/milvus/volume-expansion/yamls/milvus-standalone.yaml @@ -0,0 +1,20 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus-standalone + namespace: demo +spec: + version: "2.6.11" + topology: + mode: Standalone + objectStorage: + configSecret: + name: "milvus-storage-config" + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + storageClassName: longhorn-custom + resources: + requests: + storage: 1Gi diff --git a/docs/guides/milvus/volume-expansion/yamls/volume-expansion-online-standalone.yaml b/docs/guides/milvus/volume-expansion/yamls/volume-expansion-online-standalone.yaml index 9eb2a85eda..4ef324d48f 100644 --- a/docs/guides/milvus/volume-expansion/yamls/volume-expansion-online-standalone.yaml +++ b/docs/guides/milvus/volume-expansion/yamls/volume-expansion-online-standalone.yaml @@ -8,5 +8,5 @@ spec: databaseRef: name: milvus-standalone volumeExpansion: - node: 4Gi + node: 6Gi mode: Online \ No newline at end of file