diff --git a/docs/guides/milvus/adopt/_index.md b/docs/guides/milvus/adopt/_index.md new file mode 100644 index 0000000000..ca386ceb5a --- /dev/null +++ b/docs/guides/milvus/adopt/_index.md @@ -0,0 +1,10 @@ +--- +title: Adopt an Existing Cluster +menu: + docs_{{ .version }}: + identifier: milvus-adopt + name: Adopt an Existing Cluster + parent: milvus-guides + weight: 95 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/milvus/adopt/guide.md b/docs/guides/milvus/adopt/guide.md new file mode 100644 index 0000000000..46fb4686ea --- /dev/null +++ b/docs/guides/milvus/adopt/guide.md @@ -0,0 +1,361 @@ +--- +title: Adopt an Existing Milvus Cluster +menu: + docs_{{ .version }}: + identifier: milvus-adopt-guide + name: Guide + parent: milvus-adopt + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Adopt an Existing Milvus Cluster + +This guide moves a Distributed Milvus that already runs in Kubernetes (for example from a Helm chart) under KubeDB management **in place**. The metadata stays in the existing etcd, the segments, indexes and write-ahead log stay in the existing bucket, and the credentials stay the same. Only the workloads change hands: the chart's Deployments and StatefulSets are replaced by KubeDB PetSets that run the same Milvus version against the same data. + +Adoption is a short, ordered handover. `kubectl dba adopt milvus` runs it for you: it reads the running cluster, checks it, creates the Milvus object and moves the roles one at a time, and it can give the cluster back. The rest of this guide is the contract that command follows, and the way to do the same by hand. + +## Before You Begin + +- You should be familiar with the [Milvus](/docs/guides/milvus/concepts/milvus.md) CRD and a [Distributed](/docs/guides/milvus/quickstart/distributed.md) deployment. +- Rehearse the whole procedure on a copy of the setup first, with test data. Measure how long QueryNodes take to load your collections; that is most of the handover time. +- Take a backup you have restored at least once (see [Backup and Restore](/docs/guides/milvus/backup/logical/guide.md)). + +> Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/adopt/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/adopt/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## What must already be true + +| Requirement | Why | +|---|---| +| Milvus 2.6 with a matching KubeDB `MilvusVersion` (use the `-gpu` entry for a GPU build) | Adoption keeps the version; it is not an upgrade | +| Write-ahead log is Woodpecker on object storage (`mq.type: woodpecker`, `woodpecker.storage.type: minio`) | KubeDB always runs Woodpecker on object storage and rejects any other WAL. Changing the WAL on a live cluster is a migration, not an adoption | +| An etcd used only by this Milvus | KubeDB connects to it as an externally managed meta store and never changes it | + +## Adopt with kubectl dba + +The [`kubectl dba`](/docs/setup/README.md) plugin adopts a cluster in three commands. `RELEASE` names the running cluster: its workloads are those with `app.kubernetes.io/instance=RELEASE`, and the `component` label names each workload's role, as the upstream Milvus Helm chart sets them. For other charts, select the workloads with `--selector` and `--role-label`, or map each one with `--workload role[/group]=Kind/name`. + +### 1. Check: `plan` + +```bash +$ kubectl dba adopt milvus plan my-release -n demo +Pre-flight checks for my-release in namespace demo: + [OK ] proxy Deployment my-release-milvus-proxy, 3 replicas + [OK ] mixcoord Deployment my-release-milvus-mixcoord, 2 replicas + ... + [OK ] config read from ConfigMap my-release-milvus key default.yaml as default.yaml; ConfigMap my-release-milvus key user.yaml as user.yaml + [OK ] client Service my-release-milvus (10.43.248.56) is taken over: same name, IP and ports + [OK ] write-ahead log: woodpecker + [OK ] etcd http://my-release-etcd-0...:2379,..., root path "by-dev" (kept) + [OK ] object storage minio.storage.svc, bucket milvus-data, root path "prod" (kept) + [OK ] image milvusdb/milvus:v2.6.15 -> MilvusVersion 2.6.15 + [OK ] root credentials from common.security.defaultRootPassword log in to my-release-milvus-proxy-... + [OK ] Service my-release-milvus-mixcoord is not headless; run deletes it (KubeDB recreates it headless; Milvus does not use it) +``` + +`plan` changes nothing. It finds the workloads, the effective `milvus.yaml` (the chart's ConfigMap files, merged in the order Milvus reads them), the image and the Service clients use, and checks every row of [the contract](#the-contract). It logs in to a running proxy with the root credentials, because Milvus applies a root password only on its first start: a `defaultRootPassword` that was changed since is rejected here, and you give the current one with `--auth-secret` (an existing basic-auth Secret) or `--root-password-file`. An `ERROR` blocks the adoption; a `WARN` (an init container, environment variables, affinity or secondary networks that are not carried over) needs a look before you continue. With `--output-dir` it writes the objects it would create, so you can review or edit them and `kubectl apply` them yourself; `run` then uses what exists. + +### 2. Adopt: `run` + +```bash +$ kubectl dba adopt milvus run my-release -n demo --yes +``` + +`run` repeats the checks, then: + +1. Creates the Secrets (`-auth`, `-object-storage`, `-adopt-config` with the carried `milvus.yaml`), a ConfigMap `-adopt` that records the adoption, and the Milvus with every role at 0 replicas. It deletes a chart Service that sits on a name KubeDB needs headless, and records it. +2. Waits for the operator to render the config and create the PetSets, and compares the rendered config with the running one on every setting that decides where data is read from: object storage, etcd root path, write-ahead log, security and ports. On any difference it stops before a pod moves. +3. Hands the roles over one at a time, waiting for each pod to be Ready, and records each role in the ConfigMap. If the command stops, run it again and it continues where it was. + +Two methods: + +| `--method` | How | Client impact | Capacity | +|---|---|---|---| +| `overlap` (default) | Proxy, MixCoord, DataNode, StreamingNode, QueryNode. Start KubeDB's pods of a role, wait until they are Ready, then stop the chart's. QueryNodes move one at a time: one starts, serves for `--querynode-delay` (default 1m; raise it for large collections), then one old QueryNode stops | No role stops serving. On a test cluster (single-replica collections) clients saw one 3 s insert timeout while the StreamingNodes moved, and, in some runs, a 30 to 60 s stall of searches and inserts at the first or last QueryNode swap, while chart and KubeDB QueryNodes ran side by side. No acknowledged write was lost | One extra set of pods of a role at a time (one extra QueryNode) | +| `stop-start` | MixCoord, DataNode, QueryNode, StreamingNode, Proxy. Stop the chart's pods of a role, then start KubeDB's | Each role is briefly unavailable (searches while QueryNodes reload, writes while StreamingNodes restart) | None | + +`kubectl dba adopt milvus status my-release -n demo` shows where each role runs. + +After the adoption, leave the Helm release alone: `helm upgrade` would start the chart's workloads again next to KubeDB's, and `helm uninstall` would delete the client Service KubeDB now uses. To retire the chart, see [Keep the client endpoint](#keep-the-client-endpoint). + +### 3. Give it back, if needed: `rollback` + +```bash +$ kubectl dba adopt milvus rollback my-release -n demo --yes --cleanup +``` + +`rollback` deletes the Milvus with `kubedb.com/milvus-orphan-on-delete`, so its pods keep serving, then moves each role back to the chart's workloads, starting the chart's pods before it stops KubeDB's. Once the chart's proxies are Ready, the client Service gets back its original selector, labels and ports. `--cleanup` then deletes the PetSets (at 0 replicas), the Services and the rendered config KubeDB left, and recreates the chart Services the adoption removed. PVCs and the credential Secrets are kept. It works on a partly adopted cluster too, and for a Milvus adopted by hand (pass `--name`). Expect a short pause in writes while the StreamingNodes move back: the chart's StreamingNodes take write-ahead log channels as soon as they start, before their readiness probe passes (about 30 s on the test cluster). + +### Options in a file + +Instead of repeating flags on every command, put the options of an adoption in one file and pass it with `-f`. `plan --output-dir DIR` writes one, `DIR/adopt.yaml`, with everything it found; review it, adjust it, and use it for the rest: + +```bash +$ kubectl dba adopt milvus plan my-release -n demo --output-dir ./adopt +$ kubectl dba adopt milvus run -f ./adopt/adopt.yaml --yes +$ kubectl dba adopt milvus status -f ./adopt/adopt.yaml +$ kubectl dba adopt milvus rollback -f ./adopt/adopt.yaml --yes +``` + +A complete file, for a cluster whose chart does not set the upstream labels and runs QueryNodes on two GPU pools: + +```yaml +apiVersion: cli.kubedb.com/v1alpha1 +kind: MilvusAdoption +release: milvus-prod # replaces the RELEASE argument +namespace: milvus # used unless -n is given +name: "" # default: the name of the client Service +version: "" # default: the MilvusVersion matching the running image +source: + workloads: # or selector / roleLabel for labelled workloads + - {role: proxy, kind: Deployment, name: milvus-proxy} + - {role: mixcoord, kind: Deployment, name: milvus-mixcoord} + - {role: datanode, kind: Deployment, name: milvus-datanode} + - {role: querynode, group: l40s, kind: Deployment, name: milvus-querynode-l40s} + - {role: querynode, group: h200, kind: Deployment, name: milvus-querynode-h200} + - {role: streamingnode, kind: Deployment, name: milvus-streamingnode} + configFile: "" # the effective milvus.yaml, only if it is not mounted +auth: + authSecret: milvus-root # basic-auth Secret with the current root password + rootPasswordFile: "" # or a file with the password + etcdAuthSecret: "" # only if the etcd uses authentication + skipCredentialCheck: false +tls: # see "Keep the TLS of the cluster" + caSecret: "" # Secret with the CA and its private key; KubeDB takes over TLS + issuer: "" # or a cert-manager issuer on that CA + clientSecret: "" # client certificate for KubeDB, for mTLS with neither of the above +restrictedSCC: false # true on OpenShift +handover: + method: overlap # or stop-start + querynodeDelay: 10m # about the time one QueryNode takes to load its data + settle: 20s + timeout: 30m + readyTimeout: 30m + skipConfigCheck: false +rollback: + cleanup: true +``` + +A flag given on the command line wins over the same option in the file. An unknown key is an error, so a misspelt option is never ignored. `--yes` is never read from the file: a file alone does not start a handover. + +### What you must give + +| Option (flag / file) | Needed | +|---|---| +| `RELEASE` / `release` | Always | +| `-n` / `namespace` | Unless the kubeconfig context already points at the namespace | +| `--yes` | For `run` and `rollback` to act; without it they only show what they would do | +| `--auth-secret` or `--root-password-file` / `auth` | When the root password is not the chart's `defaultRootPassword`; `plan` tells you | +| `--etcd-auth-secret` / `auth.etcdAuthSecret` | When the etcd uses authentication | +| `--tls-ca-secret`, `--tls-issuer` or `--tls-client-secret` / `tls` | For KubeDB to take over TLS, or when client mTLS is on; see [Keep the TLS of the cluster](#keep-the-tls-of-the-cluster) | +| `--workload` / `source.workloads` (or `--selector`, `--role-label`) | When the workloads do not carry `app.kubernetes.io/instance=RELEASE` and a `component` label | +| `--config-file` / `source.configFile` | When the Milvus config is not mounted from a ConfigMap or Secret | + +Everything else has a default. The version is not needed: it is read from the running image (`milvusdb/milvus:v2.6.15-gpu` selects MilvusVersion `2.6.15-gpu`); give `--version` only for an image pinned by digest or with a custom tag. Neither is the Helm chart version: the command reads the workloads, ConfigMap and Service the chart created, not Helm. + +`kubectl dba` is a kubectl plugin that runs on your machine, with your kubeconfig; installing KubeDB in the cluster does not install it. See [Setup](/docs/setup/README.md). + +## Keep the TLS of the cluster + +Clients keep working through an adoption: they trust the same CA and connect to the same name. When KubeDB can issue from that CA, it takes over TLS completely, including renewal and `ReconfigureTLS`. `plan` reads the running config and picks one of three cases: + +| The cluster has | What the adoption does | +|---|---| +| A cert-manager issuer on the CA clients trust | Sets `spec.tls.issuerRef` to it. KubeDB issues and renews the server and client certificates. `plan` finds the issuer by itself when the running certificate came from cert-manager; otherwise give `--tls-issuer` | +| The CA Secret, with its private key, and no issuer | Give `--tls-ca-secret`. `run` creates the Issuer `-ca` from that Secret, then as above. `rollback --cleanup` removes the Issuer and leaves the Secret | +| Neither | Uses the running certificate as it is, through `spec.tls.existingSecrets.server`. If its Secret does not have the keys `tls.crt`, `tls.key` and `ca.crt`, it is copied to `-tls-server`, which `rollback --cleanup` removes. Renewing it stays with you, and a `ReconfigureTLS` rotation is refused | + +TLS that is off on a connection stays off: switching it on during a handover would split the old pods from the new ones. Do that afterwards with a `ReconfigureTLS` request. An etcd that uses TLS keeps its client certificate, through `spec.metaStorage.clientTLSSecret`, because KubeDB does not manage that etcd. + +```bash +# Create the Issuer from the CA Secret +$ kubectl dba adopt milvus plan my-release -n demo --tls-ca-secret milvus-ca --output-dir ./adopt +# Or use an issuer that exists +$ kubectl dba adopt milvus plan my-release -n demo --tls-issuer ClusterIssuer/corporate-ca --output-dir ./adopt +``` + +KubeDB's certificates cover the client Service name. Any other name in the running certificate, such as an ingress host, is carried over to `spec.tls.certificates`, and `plan` lists them. + +The new certificate must chain to the CA clients already trust, so this is checked twice: + +- `plan` stops if a CA issuer or `--tls-ca-secret` is not the CA that signed the running certificate. It cannot read the CA of a `ClusterIssuer` or of an issuer that is not a CA issuer, so it warns. +- `run` waits for the certificates and checks them **before any pod moves**. It stops if they do not verify against the running CA, or if the issuer does not fill `ca.crt`, which KubeDB needs to reach Milvus. An ACME issuer does not, so use the third case there. + +When KubeDB does not issue the certificates, `plan` also stops if: + +- the running certificate does not cover the client Service name in all three forms (``, `..svc` and `..svc.cluster.local`), which KubeDB connects to, +- client and internal TLS use different certificates, because KubeDB mounts one for both, +- client mTLS is on and no client certificate is given with `--tls-client-secret`. + +In every case `plan` stops if a certificate and key do not form a pair, a certificate does not verify against its CA, or a certificate file does not come from a Secret or ConfigMap, for example a `hostPath`. It warns when a certificate expires within 30 days. + +Three things to know: + +- A client that pins the server certificate itself, rather than trusting the CA, breaks when KubeDB issues a new one. Use the third case for it. +- Milvus refuses to serve the RESTful API on the gRPC port when TLS is on, and the proxy exits with `tls mode must be 0`. A cluster with TLS therefore already sets `proxy.http.port`, and the adoption keeps it. +- With client mTLS, `plan` has no client certificate to log in with, so it cannot check the root credentials and warns instead. Make sure of them yourself. + +Without `kubectl dba`, set the same fields on the Milvus object: + +```yaml +spec: + tls: + issuerRef: # or existingSecrets.server.name, with the keys tls.crt, tls.key, ca.crt + apiGroup: cert-manager.io + kind: Issuer + name: milvus-ca + external: + mode: TLS # the running common.security.tlsMode: 1 is TLS, 2 is mTLS + internal: + mode: TLS # only if common.security.internaltlsEnabled is true +``` + +Remove `tls`, `internaltls`, `common.security.tlsMode` and `common.security.internaltlsEnabled` from the carried `milvus.yaml`: it wins over what the operator renders, so the old paths would point at files the new pods do not have. + +## The contract + +Each row is a way an adoption can go wrong silently. Gather the current values from the running cluster first: its effective `milvus.yaml`, its Services, PVCs and pod specs. + +| Field | Set it to | What happens otherwise | +|---|---|---| +| `spec.metaStorage.externallyManaged` | `true` | KubeDB creates a new, empty etcd | +| `spec.metaStorage.endpoints` | The current etcd client endpoints. One Service address is enough | The operator cannot reach the metadata | +| `spec.metaStorage.rootPath` | The current `etcd.rootPath` (the Milvus default is `by-dev`) | KubeDB defaults it to `-etcd`. Milvus starts on an empty metadata tree and looks healthy but empty | +| `spec.metaStorage.authSecret`, `clientTLSSecret` | Only if the etcd uses auth or TLS | Connection fails | +| `spec.objectStorage.configSecret` | A Secret holding every non-default key of the current `minio:` section: `address`, `port`, `useSSL`, `bucketName`, `rootPath`, `useIAM`, `cloudProvider`, `region` and the credentials | KubeDB defaults `rootPath` to `files`, `useSSL` and `useIAM` to `false`. Segments are read from the wrong place | +| `spec.configuration.secretName` | A Secret whose `milvus.yaml` key holds the current effective config, without the `etcd` section (the Milvus object sets the etcd endpoints and root path). Keep the `woodpecker` section as it is, with `storage.type: minio` | KubeDB's own defaults apply, for example `quotaAndLimits.enabled: true`. KubeDB's WAL defaults replace the whole `woodpecker` section, so dropping it loses settings such as `woodpecker.storage.rootPath` and can move where the write-ahead log is read from | +| `proxy.http.port` in the carried `milvus.yaml` | `""` if clients use the RESTful API on the gRPC port (the Milvus default) | KubeDB defaults the REST API to port 8080. A running cluster serves it on 19530, so REST clients would break | +| `spec.authSecret` | A `kubernetes.io/basic-auth` Secret with the **current** root user and password | Milvus applies a root password only on first start. A new password does not reach the running cluster, and the health checker cannot log in | +| `spec.disableSecurity` | `true` only if the cluster runs without authentication | Clients that connect without credentials break | +| `metadata.name` | The name of the Service clients use today, if possible | The proxy Service is named after the Milvus. If a Service of that name exists, KubeDB takes it over and points it at the new proxies, so clients keep their address | +| `spec.tls` | Only if the cluster uses client or internal TLS; see [Keep the TLS of the cluster](#keep-the-tls-of-the-cluster) | KubeDB pods serve plain text and clients that use TLS fail | +| `spec.healthChecker.disableWriteCheck` | `true` | The health checker creates a `health_check_test` collection and writes to it every period | +| `spec.autoOps.disabled` | `true`, if every change must go through your own change control | The ops manager can propose version updates and credential rotations for the database | +| `spec.deletionPolicy` | `DoNotTerminate` | Deleting the object would delete the workloads | +| `kubedb.com/milvus-orphan-on-delete` annotation | `"true"` | Without it there is no clean way to hand the cluster back. See [Hand the cluster back](#hand-the-cluster-back) | + +Also check: + +- The StreamingNode PVC. KubeDB names it `data--streamingnode-`. With Woodpecker on object storage it holds no data that must survive, but confirm that before the handover. If it does, pre-create PVCs with those names from the existing volumes. +- The pod user. KubeDB images run as UID 1000 (`spec.restrictedSCC: true` lets OpenShift assign it). Volumes written as root by the old pods must be writable by the new user. +- No `MilvusArchiver` selects the namespace. The archiver needs a KubeDB-managed etcd and is rejected for an externally managed one. +- GPU and SR-IOV settings, if used: see [GPU and SR-IOV](/docs/guides/milvus/gpu-sriov/guide.md). Remove any advertise-IP init container of your own; KubeDB adds its own. + +## Keep the client endpoint + +Name the Milvus after the Service your clients use. KubeDB then takes that Service over instead of creating a new one: + +- The Service keeps its name and cluster IP, so DNS and IP-based clients are unaffected. +- Its selector moves to the KubeDB proxies only once at least one of them is Ready. Until then it keeps serving from the old proxies, so new connections are never left without endpoints. It gets no owner reference, so it is not deleted with the Milvus. +- KubeDB sets three ports: `proxy` (19530), `metrics` (9091) and `http` (8080). A port that shares a name, or a number and protocol, with one of these is replaced. Every other port is kept as it is. +- Labels, annotations and the Service type come from KubeDB and `spec.serviceTemplates`. + +If clients use another port number, or the Service is a `LoadBalancer` or `NodePort`, describe it in the `primary` service template. Ports in the template are matched by name: + +```yaml +spec: + serviceTemplates: + - alias: primary + metadata: + annotations: + # copy the annotations the Service carries today, e.g. for the load balancer + spec: + type: LoadBalancer + ports: + - name: proxy + port: 19531 # the port clients use today + nodePort: 31953 # only for NodePort/LoadBalancer, to keep the node port +``` + +Inside the pods the proxy always listens on 19530; the template only changes the Service side. So do not carry a different `proxy.port` in the `milvus.yaml` you give KubeDB. + +When you later retire the chart, do not run `helm uninstall`: Helm deletes every object in its release, including the Service KubeDB now uses. Delete the chart's other objects one by one, then remove the release record with `kubectl delete secret -l owner=helm,name=`. + +## Before you create the Milvus + +- Check the write-ahead log the cluster really uses: `mq.type` (or `messageQueue`) in its effective `milvus.yaml` must be `woodpecker`, and the StreamingNode logs must show `walName=woodpecker`. Some charts default cluster mode to Pulsar even when `woodpecker.storage` settings are present; the upstream Milvus Helm chart needs `woodpecker.enabled: true`. +- KubeDB creates headless Services named `-mixcoord`, `-datanode`, `-querynode` and `-streamingnode`. A Service of one of these names left by the chart that is **not** headless cannot be converted (a Service's cluster IP is immutable), and KubeDB reports it. Milvus components reach each other through their etcd sessions, not through these Services, so delete such a Service before you create the Milvus. +- If you rehearse with a second, temporary Milvus object, give it its own copy of the auth Secret. KubeDB records on an auth Secret which Milvus uses it, and a second Milvus is then refused that Secret. + +## Prepare the Secrets + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/adopt/yamls/object-storage-secret.yaml +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/adopt/yamls/auth-secret.yaml +$ kubectl create secret generic -n demo milvus-current-config --from-file=milvus.yaml=./current-milvus.yaml +``` + +## Hand over the workloads by hand + +This is what `kubectl dba adopt milvus run` does; follow it only if you cannot use the command. + +The pods of one role, old and new, can run side by side for the length of a handover: every Milvus 2.6 component registers its own session in etcd, MixCoord runs active-standby, and the other roles share work across their pods. So a role can move either by starting the KubeDB pods before stopping the old ones (overlap, no client impact, needs room for the extra pods), or by stopping the old ones first (stop-start, a short gap per role). + +### Staged, one role at a time + +1. Create the Milvus with every role at `replicas: 0`. KubeDB validates the metadata store, renders the config and creates empty PetSets and the Services. Nothing runs yet: + + ```bash + $ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/adopt/yamls/milvus-adopt.yaml + ``` + +2. Compare the rendered config with the running one before any pod moves: + + ```bash + # the rendered config Secret is named - + $ MV_UID=$(kubectl get milvus -n demo milvus -o jsonpath='{.metadata.uid}') + $ kubectl get secret -n demo milvus-${MV_UID: -6} -o jsonpath='{.data.milvus\.yaml}' | base64 -d > rendered.yaml + $ diff <(yq -P 'sort_keys(..)' current-milvus.yaml) <(yq -P 'sort_keys(..)' rendered.yaml) + ``` + + Every difference in `etcd`, `minio`, `common.security` and the role sections must be explained before you continue. + +3. Overlap: for each role, in the order Proxy, MixCoord, DataNode, StreamingNode, QueryNode, start the KubeDB pods, wait until they are Ready, then stop the old ones. Move QueryNodes one at a time (KubeDB to 1, wait Ready and let it serve a while, old to N-1, and so on), so at most one extra QueryNode runs: + + ```bash + $ kubectl patch milvus -n demo milvus --type=merge -p '{"spec":{"topology":{"distributed":{"mixcoord":{"replicas":2}}}}}' + $ kubectl wait -n demo --for=condition=Ready pod -l app.kubernetes.io/instance=milvus,kubedb.com/role=mixcoord --timeout=15m + # then stop the old workload of the role; its PVCs are kept + $ kubectl scale -n demo deployment/ --replicas=0 + $ kubectl wait -n demo --for=delete pod -l --timeout=10m + ``` + + Stop-start: for each role, in the order MixCoord, DataNode, QueryNode, StreamingNode, Proxy, stop the old workload first, wait until its pods are gone, then raise the KubeDB role. + + Move to the next role once the new pods are Ready. While the KubeDB proxies are at 0 the Milvus reports `NotReady`; that is expected. + +### Single window + +Scale every old workload to 0, then apply the Milvus with the final replicas. This is simpler, and the whole cluster is unavailable until the QueryNodes have loaded the collections. + +## Validate + +```bash +$ kubectl get milvus -n demo milvus +$ kubectl get milvus -n demo milvus -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}' +``` + +Then, from a client, check the collection count against the number taken before the handover, run known queries with known results, and run a smoke test from each application. + +## Roll back + +The data never moved, so a rollback only changes who runs the pods. `kubectl dba adopt milvus rollback` does it; by hand: + +1. Make sure the annotation `kubedb.com/milvus-orphan-on-delete: "true"` is set, set `spec.deletionPolicy` to `Halt`, and delete the Milvus. Its pods keep serving under their PetSets, and the operator no longer moves the client Service. +2. For each role, in the order QueryNode, StreamingNode, DataNode, MixCoord, Proxy: scale the old workload back up, wait until its pods are Ready, then scale the KubeDB PetSet to 0. +3. Before the proxy PetSet goes to 0, point the client Service back at the old proxies: its original selector, and ports whose `targetPort` the old proxy pods define (KubeDB's ports target the port names `proxy`, `metrics` and `http` of its own pods). + +## Hand the cluster back + +With the annotation `kubedb.com/milvus-orphan-on-delete: "true"` set, deleting the Milvus no longer deletes the cluster. Before it lets the object go, the operator removes the Milvus owner reference from the PetSets, Services, config Secret, ServiceAccount, Role and RoleBinding (and from a KubeDB-managed meta etcd, if one is used), and keeps PVCs and Secrets as with `deletionPolicy: Halt`, whatever the policy says. The pods keep serving under the PetSet controller. + +```bash +$ kubectl patch milvus -n demo milvus --type=merge -p '{"spec":{"deletionPolicy":"Halt"}}' +$ kubectl delete milvus -n demo milvus +$ kubectl get petset,pods,svc -n demo -l app.kubernetes.io/instance=milvus +``` + +`DoNotTerminate` still blocks deletion, which is why the policy is changed first. Keep the PetSet controller installed while the pods are run by PetSets. To move the cluster back to a chart, start its workloads role by role while scaling the PetSets down, in the same order as the handover. diff --git a/docs/guides/milvus/adopt/yamls/auth-secret.yaml b/docs/guides/milvus/adopt/yamls/auth-secret.yaml new file mode 100644 index 0000000000..653d9209b8 --- /dev/null +++ b/docs/guides/milvus/adopt/yamls/auth-secret.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Secret +metadata: + name: milvus-root + namespace: demo +type: kubernetes.io/basic-auth +stringData: + # the root credentials the running cluster uses today + username: root + password: diff --git a/docs/guides/milvus/adopt/yamls/milvus-adopt.yaml b/docs/guides/milvus/adopt/yamls/milvus-adopt.yaml new file mode 100644 index 0000000000..e28a55ad39 --- /dev/null +++ b/docs/guides/milvus/adopt/yamls/milvus-adopt.yaml @@ -0,0 +1,39 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + # the Service of the same name becomes the client endpoint + name: milvus + namespace: demo + annotations: + kubedb.com/milvus-orphan-on-delete: "true" +spec: + version: "2.6.15" + deletionPolicy: DoNotTerminate + authSecret: + kind: Secret + name: milvus-root + healthChecker: + disableWriteCheck: true + metaStorage: + externallyManaged: true + endpoints: + - http://milvus-etcd.demo.svc:2379 + rootPath: by-dev + objectStorage: + configSecret: + name: milvus-object-storage + configuration: + secretName: milvus-current-config + topology: + mode: Distributed + distributed: + mixcoord: + replicas: 0 + datanode: + replicas: 0 + querynode: + replicas: 0 + streamingnode: + replicas: 0 + proxy: + replicas: 0 diff --git a/docs/guides/milvus/adopt/yamls/object-storage-secret.yaml b/docs/guides/milvus/adopt/yamls/object-storage-secret.yaml new file mode 100644 index 0000000000..c85294d145 --- /dev/null +++ b/docs/guides/milvus/adopt/yamls/object-storage-secret.yaml @@ -0,0 +1,18 @@ +apiVersion: v1 +kind: Secret +metadata: + name: milvus-object-storage + namespace: demo +type: Opaque +stringData: + # every key overrides the same key under minio: in milvus.yaml + address: s3.example.internal + port: "443" + useSSL: "true" + bucketName: milvus-data + rootPath: files + useIAM: "false" + cloudProvider: aws + region: us-east-1 + accessKeyID: + secretAccessKey: diff --git a/docs/guides/milvus/backup/_index.md b/docs/guides/milvus/backup/_index.md new file mode 100644 index 0000000000..ebb8f576d5 --- /dev/null +++ b/docs/guides/milvus/backup/_index.md @@ -0,0 +1,10 @@ +--- +title: Backup & Restore +menu: + docs_{{ .version }}: + identifier: milvus-backup + name: Backup & Restore + parent: milvus-guides + weight: 140 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/milvus/backup/logical/_index.md b/docs/guides/milvus/backup/logical/_index.md new file mode 100644 index 0000000000..9e661360c1 --- /dev/null +++ b/docs/guides/milvus/backup/logical/_index.md @@ -0,0 +1,10 @@ +--- +title: Logical Backup +menu: + docs_{{ .version }}: + identifier: milvus-backup-logical + name: Logical Backup + parent: milvus-backup + weight: 10 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/milvus/backup/logical/guide.md b/docs/guides/milvus/backup/logical/guide.md new file mode 100644 index 0000000000..d15371d867 --- /dev/null +++ b/docs/guides/milvus/backup/logical/guide.md @@ -0,0 +1,97 @@ +--- +title: Logical Backup and Restore of Milvus | KubeStash +menu: + docs_{{ .version }}: + identifier: milvus-backup-logical-guide + name: Guide + parent: milvus-backup-logical + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Logical Backup and Restore of Milvus with KubeStash + +A logical backup exports collections through Milvus itself with `milvus-backup`. It works for Standalone and Distributed Milvus, with a KubeDB-managed or an externally managed meta etcd, because it never reads etcd directly. It can back up and restore a subset of collections, and restore either alongside the existing collections or in place over them. + +## Before You Begin + +- Install [KubeStash](https://kubestash.com/docs/latest/setup/) and the KubeDB KubeStash catalog, which provides the `milvus-addon`. +- Create a `BackupStorage` and a `RetentionPolicy` in the `demo` namespace, as in the [KubeStash quickstart](https://kubestash.com/docs/latest/concepts/crds/backupstorage/). +- Have a Ready Milvus named `milvus` in `demo`. + +> Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/backup/logical/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/backup/logical/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Parameters + +Set these under `addon.tasks[].params`. All are strings. + +| Task | Parameter | Default | Meaning | +|---|---|---|---| +| `logical-backup` | `collections` | `""` | Comma-separated `.`. Empty backs up every collection | +| `logical-backup` | `backupTimeout` | `0s` | Hard limit on the backup, as a Go duration. `0s` means no limit | +| `logical-backup-restore` | `restoreSuffix` | `_restored` | Appended to every restored collection name | +| `logical-backup-restore` | `collections` | `""` | Comma-separated `.` to restore. Empty restores every collection in the snapshot | +| `logical-backup-restore` | `inPlaceRestore` | `false` | Restore over the existing collections, replacing them. Takes precedence over the default `restoreSuffix` | +| `logical-backup-restore` | `dropExistCollection` | `false` | Drop a colliding collection in the target instead of failing | +| `logical-backup-restore` | `restoreTimeout` | `0s` | Hard limit on the restore, as a Go duration | + +Users and roles are captured in every backup but are not restored. A restored user would carry the source password hash, which could lock the operator out of the target. + +## Back up + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/backup/logical/yamls/backupconfiguration.yaml +$ kubectl get backupsession -n demo -w +$ kubectl get snapshot -n demo -l kubestash.com/repo-name=milvus-logical-repo +``` + +For a large database set `backupTimeout` well above the time a backup takes, rather than leaving it unlimited, so a hung backup fails and the next schedule runs. + +## Restore + +A restore writes into the Milvus named in `spec.target`. Choose one of three forms. + +### Alongside, into another Milvus (recommended for verification) + +Restore the latest snapshot into a separate Milvus and compare it with the source before you trust the backup. Each collection is created as `_restored`: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/backup/logical/yamls/restore-suffix.yaml +``` + +### Only some collections + +Restore two collections next to the live ones in the source database: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/backup/logical/yamls/restore-collections.yaml +``` + +The granularity is the collection. A partition cannot be restored on its own. + +### In place + +`inPlaceRestore: "true"` drops each restored collection in the target and recreates it from the backup. Before dropping anything, the plugin checks that the backup holds a replacement for every collection it is about to drop. Always combine it with `collections`, so only what you mean to replace is touched: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/backup/logical/yamls/restore-in-place.yaml +``` + +Without `inPlaceRestore`, a restore always uses a suffix. Running the same restore twice fails on the second run because `_restored` already exists; nothing is dropped unless you set `dropExistCollection` or `inPlaceRestore`. + +## Check a restore + +```bash +$ kubectl get restoresession -n demo +$ kubectl describe restoresession -n demo milvus-restore-check +``` + +Then compare row counts and run a few known queries against the restored collections. + +## What a logical backup does not cover + +- **Point-in-time recovery and physical backups** come from a `MilvusArchiver`, which needs a KubeDB-managed meta etcd dedicated to the Milvus. A Milvus with `spec.metaStorage.externallyManaged: true` can use logical backups only. +- **Partition-level restore** is not available. diff --git a/docs/guides/milvus/backup/logical/yamls/backupconfiguration.yaml b/docs/guides/milvus/backup/logical/yamls/backupconfiguration.yaml new file mode 100644 index 0000000000..8de82eef99 --- /dev/null +++ b/docs/guides/milvus/backup/logical/yamls/backupconfiguration.yaml @@ -0,0 +1,37 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: BackupConfiguration +metadata: + name: milvus-logical-backup + namespace: demo +spec: + target: + apiGroup: kubedb.com + kind: Milvus + namespace: demo + name: milvus + backends: + - name: s3-backend + storageRef: + namespace: demo + name: s3-storage + retentionPolicy: + name: demo-retention + namespace: demo + sessions: + - name: daily + scheduler: + schedule: "0 1 * * *" + jobTemplate: + backoffLimit: 1 + repositories: + - name: milvus-logical-repo + backend: s3-backend + directory: /milvus/logical + addon: + name: milvus-addon + tasks: + - name: logical-backup + params: + # empty backs up every collection; otherwise comma-separated . + collections: "" + backupTimeout: "6h" diff --git a/docs/guides/milvus/backup/logical/yamls/restore-collections.yaml b/docs/guides/milvus/backup/logical/yamls/restore-collections.yaml new file mode 100644 index 0000000000..0b21165363 --- /dev/null +++ b/docs/guides/milvus/backup/logical/yamls/restore-collections.yaml @@ -0,0 +1,21 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: RestoreSession +metadata: + name: milvus-restore-two-collections + namespace: demo +spec: + target: + apiGroup: kubedb.com + kind: Milvus + namespace: demo + name: milvus + dataSource: + repository: milvus-logical-repo + snapshot: latest + addon: + name: milvus-addon + tasks: + - name: logical-backup-restore + params: + collections: "default.products,default.reviews" + restoreSuffix: "_restored" diff --git a/docs/guides/milvus/backup/logical/yamls/restore-in-place.yaml b/docs/guides/milvus/backup/logical/yamls/restore-in-place.yaml new file mode 100644 index 0000000000..cfc9c8d672 --- /dev/null +++ b/docs/guides/milvus/backup/logical/yamls/restore-in-place.yaml @@ -0,0 +1,21 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: RestoreSession +metadata: + name: milvus-restore-in-place + namespace: demo +spec: + target: + apiGroup: kubedb.com + kind: Milvus + namespace: demo + name: milvus + dataSource: + repository: milvus-logical-repo + snapshot: latest + addon: + name: milvus-addon + tasks: + - name: logical-backup-restore + params: + collections: "default.products" + inPlaceRestore: "true" diff --git a/docs/guides/milvus/backup/logical/yamls/restore-suffix.yaml b/docs/guides/milvus/backup/logical/yamls/restore-suffix.yaml new file mode 100644 index 0000000000..2d23179036 --- /dev/null +++ b/docs/guides/milvus/backup/logical/yamls/restore-suffix.yaml @@ -0,0 +1,22 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: RestoreSession +metadata: + name: milvus-restore-check + namespace: demo +spec: + target: + apiGroup: kubedb.com + kind: Milvus + namespace: restore-check + name: milvus-check + dataSource: + repository: milvus-logical-repo + namespace: demo + snapshot: latest + addon: + name: milvus-addon + tasks: + - name: logical-backup-restore + params: + restoreSuffix: "_restored" + restoreTimeout: "6h" diff --git a/docs/guides/milvus/gpu-sriov/_index.md b/docs/guides/milvus/gpu-sriov/_index.md new file mode 100644 index 0000000000..e086879994 --- /dev/null +++ b/docs/guides/milvus/gpu-sriov/_index.md @@ -0,0 +1,10 @@ +--- +title: GPU and SR-IOV +menu: + docs_{{ .version }}: + identifier: milvus-gpu-sriov + name: GPU and SR-IOV + parent: milvus-guides + weight: 90 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/milvus/gpu-sriov/guide.md b/docs/guides/milvus/gpu-sriov/guide.md new file mode 100644 index 0000000000..75f61d2d07 --- /dev/null +++ b/docs/guides/milvus/gpu-sriov/guide.md @@ -0,0 +1,160 @@ +--- +title: Run Milvus on GPUs and SR-IOV networks +menu: + docs_{{ .version }}: + identifier: milvus-gpu-sriov-guide + name: Guide + parent: milvus-gpu-sriov + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Run Milvus on GPUs and SR-IOV networks + +KubeDB can schedule Milvus roles onto GPU nodes and attach them to an SR-IOV secondary network through Multus, including RDMA and GPUDirect Storage (GDS) devices. This guide shows the fields that control both, what the operator renders from them, and how to check that a running cluster is using them. + +## Before You Begin + +- You should be familiar with the [Milvus](/docs/guides/milvus/concepts/milvus.md) CRD and a [Distributed](/docs/guides/milvus/quickstart/distributed.md) deployment. +- For GPUs: the NVIDIA device plugin (or GPU Operator) must advertise a GPU resource, by default `nvidia.com/gpu`. +- For SR-IOV: Multus, the SR-IOV device plugin and a `NetworkAttachmentDefinition` (NAD) in the namespace of the Milvus. Creating these is a cluster-admin task outside KubeDB. + +> Note: The yaml files used in this tutorial are stored in [docs/guides/milvus/gpu-sriov/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/milvus/gpu-sriov/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Use a GPU build of Milvus + +GPU requests are only accepted on a `MilvusVersion` that declares `spec.db.gpu.supported: true`. The catalog ships these as `-gpu` entries: + +```bash +$ kubectl get milvusversion -o custom-columns=NAME:.metadata.name,GPU:.spec.db.gpu.supported,IMAGE:.spec.db.image +NAME GPU IMAGE +2.6.15 ghcr.io/appscode-images/milvus:2.6.15 +2.6.15-gpu true ghcr.io/appscode-images/milvus:2.6.15-gpu +... +``` + +The webhook rejects a Milvus that requests a GPU anywhere (`spec.gpu`, a role's or group's `gpu`, or an `nvidia.com/gpu` resource written into a `podTemplate`) unless its version is a GPU build. A `MilvusOpsRequest` of type `UpdateVersion` is held to the same rule, so a GPU database cannot be moved onto a CPU-only image. + +## Request GPUs per role + +In a Distributed Milvus, each role takes a `gpu` block: + +| Field | Default | Meaning | +|---|---|---| +| `resourceName` | `nvidia.com/gpu` | Extended resource to request | +| `count` | `1` | GPUs per pod, set in both requests and limits | +| `nodeSelector` | none | Added to the pod's `nodeSelector` | +| `tolerations` | none | Appended to the pod's tolerations | + +QueryNode (search) and DataNode (index build) are the roles that use a GPU. Your own CPU and memory requests in the role's `podTemplate` are kept; only the GPU resource is added. + +### Spread one role across several GPU pools + +`gpu.nodeSelector` is a conjunction of labels, so it can only point a role at one pool. To run, for example, six QueryNodes with two on L40S nodes and four on H200 nodes, use a PetSet `PlacementPolicy` with node affinity. It assigns pods to each domain by count: + +```yaml +apiVersion: apps.k8s.appscode.com/v1 +kind: PlacementPolicy +metadata: + name: milvus-querynode-gpu-pools +spec: + affinity: + nodeAffinity: + - topologyKey: nvidia.com/gpu.product + whenUnsatisfiable: DoNotSchedule + weight: 100 + domains: + - values: ["NVIDIA-L40S"] + replicas: "2" + - values: ["NVIDIA-H200"] + replicas: "4" +``` + +Refer to it from the role's `podTemplate.spec.podPlacementPolicy` and leave `gpu.nodeSelector` unset for that role. `PlacementPolicy` is cluster-scoped. + +Alternatively, split the role into `groups`, one per pool, each with its own `gpu.nodeSelector`. Every group is a separate PetSet, but Milvus does not see them: all QueryNodes still join the same Milvus resource group. When `groups` is set, the role-level `replicas`, `podTemplate`, `network` and `gpu` are ignored. + +## Attach roles to an SR-IOV network + +Each role takes a `network.sriov` block: + +| Field | Required | Meaning | +|---|---|---| +| `attachmentRef` | yes | Name of the NAD, in the same namespace as the Milvus | +| `resourceName` | yes | SR-IOV VF resource advertised by the device plugin | +| `rdmaResourceName` | no | RDMA resource, if the VF is used for RDMA | +| `interface` | no | Interface name inside the pod, default `net1` | +| `gds.attachmentRef`, `gds.resourceName` | no | A second attachment for GPUDirect Storage, default interface `net2` | + +For each role with `network.sriov`, the operator: + +1. Adds the Multus annotation `k8s.v1.cni.cncf.io/networks` to the pods, and one unit of each named resource to the `milvus` container. +2. Adds the init container `milvus-sriov-user-config-init`. It waits up to 30 seconds for an IPv4 address on the interface and writes `MILVUS_CONF_
_IP` exports. The `milvus` container sources them before it starts, so the role registers its SR-IOV address, not the pod IP, in the meta etcd. MixCoord advertises under `rootCoord`, `queryCoord`, `dataCoord` and `indexCoord`; StreamingNode under `streamingNode` and `queryNode`. +3. Adds a readiness probe that fails when the interface loses carrier or its address, or when Milvus stops answering `/healthz` on that address. + +Set `network.sriov` on all five roles. Every role dials the others by their registered address, so a role left on the pod network may be unable to reach the rest. The webhook warns when only some roles have it. + +Do not add your own advertise-IP init container to the `podTemplate`. Init containers in the `podTemplate` are kept and would run next to the operator's. + +### Security context and RDMA + +By default the `milvus` container runs as non-root with all capabilities dropped. RDMA verbs commonly need `IPC_LOCK`. Set it in the role's `podTemplate`; the defaults only fill fields you leave empty: + +```yaml +podTemplate: + spec: + containers: + - name: milvus + securityContext: + capabilities: + add: ["IPC_LOCK"] +``` + +On OpenShift, set `spec.restrictedSCC: true` to let the SCC assign the user and group instead of the catalog's `runAsUser`. + +## Deploy + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/gpu-sriov/yamls/placement-policy.yaml +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/milvus/gpu-sriov/yamls/milvus-gpu-sriov.yaml +``` + +The example uses YAML anchors (`&sriov`, `*sriov`) to repeat the same `network` block on several roles. + +## Verify + +The Milvus reports two conditions for SR-IOV: + +| Condition | True when | +|---|---| +| `SRIOVNetworkAttached` | Multus reports an address on the configured interface for every SR-IOV pod | +| `SRIOVAddressAdvertised` | Every SR-IOV pod's session in the meta etcd carries its SR-IOV address | + +```bash +$ kubectl get milvus -n demo milvus-gpu -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}' | grep SRIOV +SRIOVNetworkAttached=True AllSRIOVNetworksAttached +SRIOVAddressAdvertised=True AllSessionsOnSRIOVNetwork +``` + +Check placement and GPU allocation: + +```bash +$ kubectl get pods -n demo -l app.kubernetes.io/instance=milvus-gpu -o wide +$ kubectl get pod -n demo milvus-gpu-querynode-0 -o jsonpath='{.spec.containers[?(@.name=="milvus")].resources}' +``` + +## Troubleshooting + +- **Pod stuck in `Init`**: the init container found no IPv4 address on the interface within 30 seconds. Check the NAD name, the VF resource and the pod's `k8s.v1.cni.cncf.io/network-status` annotation. +- **`SRIOVAddressAdvertised` is False with reason `SessionsNotOnSRIOVNetwork`**: the message names the pods whose etcd session still shows another address. Restart them with a `Restart` ops request after fixing the network. +- **Admission error about `spec.db.gpu.supported`**: the version is not a GPU build. Use the `-gpu` entry. + +## Cleaning up + +```bash +$ kubectl delete milvus -n demo milvus-gpu +$ kubectl delete placementpolicy milvus-querynode-gpu-pools +``` diff --git a/docs/guides/milvus/gpu-sriov/yamls/milvus-gpu-sriov.yaml b/docs/guides/milvus/gpu-sriov/yamls/milvus-gpu-sriov.yaml new file mode 100644 index 0000000000..dfff18e746 --- /dev/null +++ b/docs/guides/milvus/gpu-sriov/yamls/milvus-gpu-sriov.yaml @@ -0,0 +1,63 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus-gpu + namespace: demo +spec: + version: "2.6.15-gpu" + deletionPolicy: Delete + objectStorage: + configSecret: + name: milvus-storage-config + topology: + mode: Distributed + distributed: + mixcoord: + replicas: 2 + network: &sriov + sriov: + attachmentRef: sriov-net + resourceName: nvidia.com/sriov_vf + rdmaResourceName: rdma/rdma_shared_device + proxy: + replicas: 2 + network: *sriov + querynode: + replicas: 6 + podTemplate: + spec: + podPlacementPolicy: + name: milvus-querynode-gpu-pools + containers: + - name: milvus + resources: + limits: + memory: 64Gi + gpu: + count: 1 + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + network: + sriov: + attachmentRef: sriov-net + resourceName: nvidia.com/sriov_vf + rdmaResourceName: rdma/rdma_shared_device + gds: + attachmentRef: gds-net + resourceName: nvidia.com/gds_vf + datanode: + replicas: 2 + gpu: + count: 1 + nodeSelector: + nvidia.com/gpu.product: NVIDIA-L40S + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + network: *sriov + streamingnode: + replicas: 2 + network: *sriov diff --git a/docs/guides/milvus/gpu-sriov/yamls/placement-policy.yaml b/docs/guides/milvus/gpu-sriov/yamls/placement-policy.yaml new file mode 100644 index 0000000000..a7cef13fe1 --- /dev/null +++ b/docs/guides/milvus/gpu-sriov/yamls/placement-policy.yaml @@ -0,0 +1,15 @@ +apiVersion: apps.k8s.appscode.com/v1 +kind: PlacementPolicy +metadata: + name: milvus-querynode-gpu-pools +spec: + affinity: + nodeAffinity: + - topologyKey: nvidia.com/gpu.product + whenUnsatisfiable: DoNotSchedule + weight: 100 + domains: + - values: ["NVIDIA-L40S"] + replicas: "2" + - values: ["NVIDIA-H200"] + replicas: "4" diff --git a/docs/reference/cli/kubectl-dba.md b/docs/reference/cli/kubectl-dba.md index a09c8995dc..a3bd48fd0d 100644 --- a/docs/reference/cli/kubectl-dba.md +++ b/docs/reference/cli/kubectl-dba.md @@ -57,6 +57,7 @@ kubectl-dba [flags] ### SEE ALSO +* [kubectl-dba adopt](/docs/reference/cli/kubectl-dba_adopt.md) - Bring a running database under KubeDB in place, without moving its data * [kubectl-dba completion](/docs/reference/cli/kubectl-dba_completion.md) - Generate completion script * [kubectl-dba connect](/docs/reference/cli/kubectl-dba_connect.md) - Connect to a database. * [kubectl-dba data](/docs/reference/cli/kubectl-dba_data.md) - Insert, Drop or Verify data in a database diff --git a/docs/reference/cli/kubectl-dba_adopt.md b/docs/reference/cli/kubectl-dba_adopt.md new file mode 100644 index 0000000000..3b5eecb98f --- /dev/null +++ b/docs/reference/cli/kubectl-dba_adopt.md @@ -0,0 +1,60 @@ +--- +title: Kubectl-Dba Adopt +menu: + docs_{{ .version }}: + identifier: kubectl-dba-adopt + name: Kubectl-Dba Adopt + parent: reference-cli +menu_name: docs_{{ .version }} +section_menu_id: reference +--- +## kubectl-dba adopt + +Bring a running database under KubeDB in place, without moving its data + +### Synopsis + +Take over a database that already runs in Kubernetes, for example one installed by a Helm chart. The data stays in its current storage, the credentials stay the same, and clients keep their address; only the workloads change hands, one role at a time. + +``` +kubectl-dba adopt +``` + +### Options + +``` + -h, --help help for adopt +``` + +### Options inherited from parent commands + +``` + --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. + --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. + --as-uid string UID to impersonate for the operation. + --cache-dir string Default cache directory (default "/home/runner/.kube/cache") + --certificate-authority string Path to a cert file for the certificate authority + --client-certificate string Path to a client certificate file for TLS + --client-key string Path to a client key file for TLS + --cluster string The name of the kubeconfig cluster to use + --context string The name of the kubeconfig context to use + --default-seccomp-profile-type string Default seccomp profile + --disable-compression If true, opt-out of response compression for all requests to the server + --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure + --kubeconfig string Path to the kubeconfig file to use for CLI requests. + --match-server-version Require server version to match client version + -n, --namespace string If present, the namespace scope for this CLI request + --password string Password for basic authentication to the API server + --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") + -s, --server string The address and port of the Kubernetes API server + --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used + --token string Bearer token for authentication to the API server + --user string The name of the kubeconfig user to use + --username string Username for basic authentication to the API server +``` + +### SEE ALSO + +* [kubectl-dba](/docs/reference/cli/kubectl-dba.md) - kubectl plugin for KubeDB +* [kubectl-dba adopt milvus](/docs/reference/cli/kubectl-dba_adopt_milvus.md) - Adopt a running Distributed Milvus 2.6 in place + diff --git a/docs/reference/cli/kubectl-dba_adopt_milvus.md b/docs/reference/cli/kubectl-dba_adopt_milvus.md new file mode 100644 index 0000000000..f7a7bfa9b1 --- /dev/null +++ b/docs/reference/cli/kubectl-dba_adopt_milvus.md @@ -0,0 +1,70 @@ +--- +title: Kubectl-Dba Adopt Milvus +menu: + docs_{{ .version }}: + identifier: kubectl-dba-adopt-milvus + name: Kubectl-Dba Adopt Milvus + parent: reference-cli +menu_name: docs_{{ .version }} +section_menu_id: reference +--- +## kubectl-dba adopt milvus + +Adopt a running Distributed Milvus 2.6 in place + +### Synopsis + +Adopt a Distributed Milvus 2.6 that runs from a Helm chart (or any set of Deployments and StatefulSets) in place. The metadata stays in its etcd, the segments, indexes and write-ahead log stay in its bucket, the root credentials and the client Service stay the same. + + * plan: check the cluster and show what an adoption creates (read-only) + * run: create the Milvus and hand the roles over, one at a time + * status: show where each role runs + * rollback: give the cluster back to the chart's workloads + + RELEASE names the running cluster: by default its workloads are those with app.kubernetes.io/instance=RELEASE, and the "component" label names each workload's role, as in the upstream Milvus Helm chart. + +``` +kubectl-dba adopt milvus +``` + +### Options + +``` + -h, --help help for milvus +``` + +### Options inherited from parent commands + +``` + --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. + --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. + --as-uid string UID to impersonate for the operation. + --cache-dir string Default cache directory (default "/home/runner/.kube/cache") + --certificate-authority string Path to a cert file for the certificate authority + --client-certificate string Path to a client certificate file for TLS + --client-key string Path to a client key file for TLS + --cluster string The name of the kubeconfig cluster to use + --context string The name of the kubeconfig context to use + --default-seccomp-profile-type string Default seccomp profile + --disable-compression If true, opt-out of response compression for all requests to the server + --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure + --kubeconfig string Path to the kubeconfig file to use for CLI requests. + --match-server-version Require server version to match client version + -n, --namespace string If present, the namespace scope for this CLI request + --password string Password for basic authentication to the API server + --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") + -s, --server string The address and port of the Kubernetes API server + --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used + --token string Bearer token for authentication to the API server + --user string The name of the kubeconfig user to use + --username string Username for basic authentication to the API server +``` + +### SEE ALSO + +* [kubectl-dba adopt](/docs/reference/cli/kubectl-dba_adopt.md) - Bring a running database under KubeDB in place, without moving its data +* [kubectl-dba adopt milvus plan](/docs/reference/cli/kubectl-dba_adopt_milvus_plan.md) - Check a running Milvus and show what adopting it creates +* [kubectl-dba adopt milvus rollback](/docs/reference/cli/kubectl-dba_adopt_milvus_rollback.md) - Give an adopted (or partly adopted) Milvus back to the chart's workloads +* [kubectl-dba adopt milvus run](/docs/reference/cli/kubectl-dba_adopt_milvus_run.md) - Adopt a running Milvus: create the Milvus object and hand the roles over +* [kubectl-dba adopt milvus status](/docs/reference/cli/kubectl-dba_adopt_milvus_status.md) - Show where each role of an adoption runs + diff --git a/docs/reference/cli/kubectl-dba_adopt_milvus_plan.md b/docs/reference/cli/kubectl-dba_adopt_milvus_plan.md new file mode 100644 index 0000000000..ce0fffb6bf --- /dev/null +++ b/docs/reference/cli/kubectl-dba_adopt_milvus_plan.md @@ -0,0 +1,85 @@ +--- +title: Kubectl-Dba Adopt Milvus Plan +menu: + docs_{{ .version }}: + identifier: kubectl-dba-adopt-milvus-plan + name: Kubectl-Dba Adopt Milvus Plan + parent: reference-cli +menu_name: docs_{{ .version }} +section_menu_id: reference +--- +## kubectl-dba adopt milvus plan + +Check a running Milvus and show what adopting it creates + +### Synopsis + +Read-only. Finds the workloads, effective configuration, image and client Service of the running cluster, checks everything an in-place adoption depends on (Woodpecker WAL, etcd root path, object storage, a matching MilvusVersion, the current root password), and shows the Secrets, the state ConfigMap and the staged Milvus that "run" would create. + + With --output-dir the objects are written there, unredacted, together with adopt.yaml: every option of this adoption as plan found it. Review it and pass it to run, status and rollback with -f instead of repeating the flags. Edit the objects if needed and apply them; "run" then uses what exists. + +``` +kubectl-dba adopt milvus plan [RELEASE] [flags] +``` + +### Examples + +``` + kubectl dba adopt milvus plan my-release -n milvus + kubectl dba adopt milvus plan my-release -n milvus --output-dir ./adopt + kubectl dba adopt milvus plan -f ./adopt/adopt.yaml +``` + +### Options + +``` + --auth-secret string Existing basic-auth Secret with the current root user and password + --config-file string The effective milvus.yaml, when it is not mounted from a ConfigMap or Secret + --etcd-auth-secret string Basic-auth Secret for an etcd that uses authentication + -f, --filename string File with the adoption options (kind MilvusAdoption; "plan --output-dir" writes one). Flags given on the command line win + -h, --help help for plan + --name string Name of the Milvus object (default: the name of the Service clients use, which KubeDB then takes over) + --output-dir string Write the objects to this directory (Secrets unredacted, mode 0600) + --restricted-scc Set spec.restrictedSCC (OpenShift) + --role-label string Pod label that names a workload's Milvus role (default "component") + --root-password-file string File holding the current root password (default: common.security.defaultRootPassword of the running config) + --selector string Label selector of the cluster's workloads (default app.kubernetes.io/instance=RELEASE) + --skip-credential-check Do not log in to the running cluster to check the root credentials + --tls-ca-secret string Secret (tls.crt, tls.key) with the CA clients trust; an Issuer is created from it and KubeDB issues and renews the certificates + --tls-client-secret string Existing Secret (tls.crt, tls.key, ca.crt) with the client certificate KubeDB presents when client mTLS is on + --tls-issuer string cert-manager issuer on the CA clients trust, as name, Issuer/name or ClusterIssuer/name; KubeDB then issues and renews the certificates (found by itself if the running certificate came from cert-manager) + --version string MilvusVersion to use (default: the one matching the running image) + --workload stringArray Map a workload to a role, as role[/group]=Kind/name (repeat for each workload; replaces --selector) +``` + +### Options inherited from parent commands + +``` + --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. + --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. + --as-uid string UID to impersonate for the operation. + --cache-dir string Default cache directory (default "/home/runner/.kube/cache") + --certificate-authority string Path to a cert file for the certificate authority + --client-certificate string Path to a client certificate file for TLS + --client-key string Path to a client key file for TLS + --cluster string The name of the kubeconfig cluster to use + --context string The name of the kubeconfig context to use + --default-seccomp-profile-type string Default seccomp profile + --disable-compression If true, opt-out of response compression for all requests to the server + --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure + --kubeconfig string Path to the kubeconfig file to use for CLI requests. + --match-server-version Require server version to match client version + -n, --namespace string If present, the namespace scope for this CLI request + --password string Password for basic authentication to the API server + --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") + -s, --server string The address and port of the Kubernetes API server + --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used + --token string Bearer token for authentication to the API server + --user string The name of the kubeconfig user to use + --username string Username for basic authentication to the API server +``` + +### SEE ALSO + +* [kubectl-dba adopt milvus](/docs/reference/cli/kubectl-dba_adopt_milvus.md) - Adopt a running Distributed Milvus 2.6 in place + diff --git a/docs/reference/cli/kubectl-dba_adopt_milvus_rollback.md b/docs/reference/cli/kubectl-dba_adopt_milvus_rollback.md new file mode 100644 index 0000000000..fc0a26be91 --- /dev/null +++ b/docs/reference/cli/kubectl-dba_adopt_milvus_rollback.md @@ -0,0 +1,80 @@ +--- +title: Kubectl-Dba Adopt Milvus Rollback +menu: + docs_{{ .version }}: + identifier: kubectl-dba-adopt-milvus-rollback + name: Kubectl-Dba Adopt Milvus Rollback + parent: reference-cli +menu_name: docs_{{ .version }} +section_menu_id: reference +--- +## kubectl-dba adopt milvus rollback + +Give an adopted (or partly adopted) Milvus back to the chart's workloads + +### Synopsis + +Deletes the Milvus object with kubedb.com/milvus-orphan-on-delete, so its PetSets keep serving, then moves each role back to the chart's workloads, starting the chart's pods before it stops KubeDB's. Once the chart's proxies are Ready, the client Service gets back its original selector, labels and ports. The data is never touched. + + With --cleanup it then deletes the PetSets (at 0 replicas), the Services other than the client Service and the rendered config KubeDB left, and recreates the chart Services the adoption removed. PVCs are kept. + + For a Milvus adopted by hand (no state ConfigMap), give its --name: the chart's replicas are then taken from the PetSets, and the client Service gets the chart proxies' selector. + +``` +kubectl-dba adopt milvus rollback [RELEASE] [flags] +``` + +### Examples + +``` + kubectl dba adopt milvus rollback my-release -n milvus --yes --cleanup + kubectl dba adopt milvus rollback -f ./adopt/adopt.yaml --yes +``` + +### Options + +``` + --cleanup Delete what KubeDB left behind once the chart serves again (PVCs are kept) + -f, --filename string File with the adoption options (kind MilvusAdoption; "plan --output-dir" writes one). Flags given on the command line win + -h, --help help for rollback + --name string Name of the Milvus, for an adoption done by hand (no state ConfigMap) + --querynode-delay duration Overlap: time a new QueryNode serves before an old one stops (raise it for large collections) (default 1m0s) + --role-label string Pod label that names a workload's Milvus role (default "component") + --selector string Label selector of the cluster's workloads (default app.kubernetes.io/instance=RELEASE) + --settle duration Pause after each role (default 20s) + --timeout duration Longest wait for one step (pods Ready or gone) (default 15m0s) + --workload stringArray Map a workload to a role, as role[/group]=Kind/name (repeat for each workload; replaces --selector) + --yes Proceed; without it, rollback only shows what it would do +``` + +### Options inherited from parent commands + +``` + --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. + --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. + --as-uid string UID to impersonate for the operation. + --cache-dir string Default cache directory (default "/home/runner/.kube/cache") + --certificate-authority string Path to a cert file for the certificate authority + --client-certificate string Path to a client certificate file for TLS + --client-key string Path to a client key file for TLS + --cluster string The name of the kubeconfig cluster to use + --context string The name of the kubeconfig context to use + --default-seccomp-profile-type string Default seccomp profile + --disable-compression If true, opt-out of response compression for all requests to the server + --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure + --kubeconfig string Path to the kubeconfig file to use for CLI requests. + --match-server-version Require server version to match client version + -n, --namespace string If present, the namespace scope for this CLI request + --password string Password for basic authentication to the API server + --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") + -s, --server string The address and port of the Kubernetes API server + --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used + --token string Bearer token for authentication to the API server + --user string The name of the kubeconfig user to use + --username string Username for basic authentication to the API server +``` + +### SEE ALSO + +* [kubectl-dba adopt milvus](/docs/reference/cli/kubectl-dba_adopt_milvus.md) - Adopt a running Distributed Milvus 2.6 in place + diff --git a/docs/reference/cli/kubectl-dba_adopt_milvus_run.md b/docs/reference/cli/kubectl-dba_adopt_milvus_run.md new file mode 100644 index 0000000000..c8019b53e6 --- /dev/null +++ b/docs/reference/cli/kubectl-dba_adopt_milvus_run.md @@ -0,0 +1,98 @@ +--- +title: Kubectl-Dba Adopt Milvus Run +menu: + docs_{{ .version }}: + identifier: kubectl-dba-adopt-milvus-run + name: Kubectl-Dba Adopt Milvus Run + parent: reference-cli +menu_name: docs_{{ .version }} +section_menu_id: reference +--- +## kubectl-dba adopt milvus run + +Adopt a running Milvus: create the Milvus object and hand the roles over + +### Synopsis + +Runs the checks of "plan", creates the Secrets, the state ConfigMap NAME-adopt (NAME is the Milvus) and the Milvus with every role at 0 replicas, and waits for the operator. Before any pod moves, it compares the config the operator rendered with the running one on every setting that decides where the data is read from, and stops on a difference. + + Then it hands the roles over one at a time: + + * --method overlap (default): start KubeDB's pods of a role, wait until they are Ready, then stop the chart's. QueryNodes move one at a time. No role stops serving while it moves; it needs room for one extra set of pods of a role (one extra QueryNode). + * --method stop-start: stop the chart's pods of a role, then start KubeDB's. Each role is briefly unavailable; no extra capacity is needed. + + Progress is recorded in the state ConfigMap: if the command stops, run it again and it continues where it was. "rollback" gives the cluster back. + + Do not remove the chart with "helm uninstall" afterwards: it would delete the client Service that KubeDB now uses. + +``` +kubectl-dba adopt milvus run [RELEASE] [flags] +``` + +### Examples + +``` + kubectl dba adopt milvus run my-release -n milvus --yes + kubectl dba adopt milvus run my-release -n milvus --method stop-start --yes + kubectl dba adopt milvus run -f ./adopt/adopt.yaml --yes +``` + +### Options + +``` + --auth-secret string Existing basic-auth Secret with the current root user and password + --config-file string The effective milvus.yaml, when it is not mounted from a ConfigMap or Secret + --etcd-auth-secret string Basic-auth Secret for an etcd that uses authentication + -f, --filename string File with the adoption options (kind MilvusAdoption; "plan --output-dir" writes one). Flags given on the command line win + -h, --help help for run + --method string Handover method: overlap or stop-start (default "overlap") + --name string Name of the Milvus object (default: the name of the Service clients use, which KubeDB then takes over) + --querynode-delay duration Overlap: time a new QueryNode serves before an old one stops (raise it for large collections) (default 1m0s) + --ready-timeout duration Longest wait for the operator, and for the Milvus to be Ready at the end (default 15m0s) + --restricted-scc Set spec.restrictedSCC (OpenShift) + --role-label string Pod label that names a workload's Milvus role (default "component") + --root-password-file string File holding the current root password (default: common.security.defaultRootPassword of the running config) + --selector string Label selector of the cluster's workloads (default app.kubernetes.io/instance=RELEASE) + --settle duration Pause after each role (default 20s) + --skip-config-check Do not stop when the rendered config differs from the running one + --skip-credential-check Do not log in to the running cluster to check the root credentials + --timeout duration Longest wait for one step (pods Ready or gone) (default 15m0s) + --tls-ca-secret string Secret (tls.crt, tls.key) with the CA clients trust; an Issuer is created from it and KubeDB issues and renews the certificates + --tls-client-secret string Existing Secret (tls.crt, tls.key, ca.crt) with the client certificate KubeDB presents when client mTLS is on + --tls-issuer string cert-manager issuer on the CA clients trust, as name, Issuer/name or ClusterIssuer/name; KubeDB then issues and renews the certificates (found by itself if the running certificate came from cert-manager) + --version string MilvusVersion to use (default: the one matching the running image) + --workload stringArray Map a workload to a role, as role[/group]=Kind/name (repeat for each workload; replaces --selector) + --yes Proceed; without it, run only shows what it would do +``` + +### Options inherited from parent commands + +``` + --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. + --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. + --as-uid string UID to impersonate for the operation. + --cache-dir string Default cache directory (default "/home/runner/.kube/cache") + --certificate-authority string Path to a cert file for the certificate authority + --client-certificate string Path to a client certificate file for TLS + --client-key string Path to a client key file for TLS + --cluster string The name of the kubeconfig cluster to use + --context string The name of the kubeconfig context to use + --default-seccomp-profile-type string Default seccomp profile + --disable-compression If true, opt-out of response compression for all requests to the server + --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure + --kubeconfig string Path to the kubeconfig file to use for CLI requests. + --match-server-version Require server version to match client version + -n, --namespace string If present, the namespace scope for this CLI request + --password string Password for basic authentication to the API server + --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") + -s, --server string The address and port of the Kubernetes API server + --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used + --token string Bearer token for authentication to the API server + --user string The name of the kubeconfig user to use + --username string Username for basic authentication to the API server +``` + +### SEE ALSO + +* [kubectl-dba adopt milvus](/docs/reference/cli/kubectl-dba_adopt_milvus.md) - Adopt a running Distributed Milvus 2.6 in place + diff --git a/docs/reference/cli/kubectl-dba_adopt_milvus_status.md b/docs/reference/cli/kubectl-dba_adopt_milvus_status.md new file mode 100644 index 0000000000..2d6bd0cd4b --- /dev/null +++ b/docs/reference/cli/kubectl-dba_adopt_milvus_status.md @@ -0,0 +1,62 @@ +--- +title: Kubectl-Dba Adopt Milvus Status +menu: + docs_{{ .version }}: + identifier: kubectl-dba-adopt-milvus-status + name: Kubectl-Dba Adopt Milvus Status + parent: reference-cli +menu_name: docs_{{ .version }} +section_menu_id: reference +--- +## kubectl-dba adopt milvus status + +Show where each role of an adoption runs + +``` +kubectl-dba adopt milvus status [RELEASE] [flags] +``` + +### Examples + +``` + kubectl dba adopt milvus status my-release -n milvus +``` + +### Options + +``` + -f, --filename string File with the adoption options (kind MilvusAdoption; "plan --output-dir" writes one). Flags given on the command line win + -h, --help help for status +``` + +### Options inherited from parent commands + +``` + --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. + --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. + --as-uid string UID to impersonate for the operation. + --cache-dir string Default cache directory (default "/home/runner/.kube/cache") + --certificate-authority string Path to a cert file for the certificate authority + --client-certificate string Path to a client certificate file for TLS + --client-key string Path to a client key file for TLS + --cluster string The name of the kubeconfig cluster to use + --context string The name of the kubeconfig context to use + --default-seccomp-profile-type string Default seccomp profile + --disable-compression If true, opt-out of response compression for all requests to the server + --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure + --kubeconfig string Path to the kubeconfig file to use for CLI requests. + --match-server-version Require server version to match client version + -n, --namespace string If present, the namespace scope for this CLI request + --password string Password for basic authentication to the API server + --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") + -s, --server string The address and port of the Kubernetes API server + --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used + --token string Bearer token for authentication to the API server + --user string The name of the kubeconfig user to use + --username string Username for basic authentication to the API server +``` + +### SEE ALSO + +* [kubectl-dba adopt milvus](/docs/reference/cli/kubectl-dba_adopt_milvus.md) - Adopt a running Distributed Milvus 2.6 in place +