Skip to content

Add kubectl dba adopt milvus: adopt a running Milvus in place - #847

Open
Sabnaj-42 wants to merge 3 commits into
masterfrom
milvus-inplace-adoption
Open

Sabnaj-42 wants to merge 3 commits into
masterfrom
milvus-inplace-adoption

Conversation

@Sabnaj-42

@Sabnaj-42 Sabnaj-42 commented Oct 30, 2026 •

Copy link
Copy Markdown

Adds kubectl dba adopt milvus, which adopts a Distributed Milvus 2.6 that runs from a Helm chart (or any Deployments/StatefulSets) under KubeDB in place: same etcd and root path, same bucket, same root credentials, same client Service. It replaces the manual procedure of the adoption guide (gather the effective config, write three Secrets and a staged Milvus, diff the rendered config, scale each role by hand).

  • plan RELEASE (read-only): finds the workloads (app.kubernetes.io/instance=RELEASE + component label, or --selector/--role-label/--workload role[/group]=Kind/name), merges the mounted config files in Milvus' load order, finds the client Service and the MilvusVersion of the running image, checks the WAL (Woodpecker on object storage), etcd, object storage, and logs in to a running proxy with the root credentials through a port-forward (the API server's service proxy drops the Authorization header). Milvus applies a root password only on first start, so a changed defaultRootPassword is caught here. --output-dir writes the objects for review.
  • run RELEASE: creates the Secrets, a state ConfigMap <name>-adopt and the Milvus with every role at 0 (orphan-on-delete, DoNotTerminate, external etcd, autoOps off, write check off); deletes (and records) a chart Service on a name KubeDB needs headless; waits for the operator and stops if the rendered config differs from the running one on storage, etcd root path, WAL, security or ports. Then hands the roles over: --method overlap (default; KubeDB's pods first, QueryNodes one at a time) or stop-start. Each role is recorded; re-running resumes.
  • status RELEASE: where each role runs.
  • rollback RELEASE: deletes the Milvus with orphan-on-delete, moves each role back to the chart (chart pods first), restores the client Service's selector, labels and ports; --cleanup deletes what KubeDB left and recreates removed chart Services. --name handles a Milvus adopted by hand.

Options in a file (-f): every command also takes -f FILE (kind MilvusAdoption, apiVersion cli.kubedb.com/v1alpha1) with release, namespace, name, version, source (selector, roleLabel, workloads with node groups, configFile), auth, restrictedSCC, handover (method and timing) and rollback (cleanup). Flags given on the command line win; unknown keys are errors; --yes is never read from the file. plan --output-dir writes adopt.yaml with what it found, ready for run -f. Tested on the cluster: rollback, plan and run driven by the file alone (T6–T8), data PASS.

Several workloads of one role become node groups (e.g. QueryNodes on two GPU pools). Objects are handled unstructured, so no apimachinery bump is needed.

Tests: unit tests for config merge/extraction, carried config, rendered-config comparison, workload/group assignment, rendering, version selection, swap steps, Service record. End to end on a k3d cluster running the upstream Milvus chart 5.0.19 (Milvus 2.6.15; Proxy 3, MixCoord 2, DataNode 2, QueryNode 6, StreamingNode 2; 3-member etcd; Woodpecker on S3; auth on), with a client probe and a bit-exact data check after each phase:

Phase Result
rollback of a hand-adopted Milvus (--name) data PASS, 3,290 acknowledged inserts intact
plan (incl. wrong password, bad --workload) blocked / clean error as expected
run, overlap adopted in 11m17s, no manual step; data PASS
rollback with state, --cleanup 28m42s; Service and chart Service restored; data PASS
run, stop-start, killed mid-handover and re-run resumed, adopted in 2m39s; per-role outages match the manual procedure; data PASS

Known: with overlap, a 30-60 s stall of searches and inserts was seen at the first/last QueryNode swap while chart and KubeDB QueryNodes ran side by side (not seen when adding/removing a QueryNode on the chart alone); a rollback pauses writes ~30 s while StreamingNodes move back to the chart. Both are documented in the guide (kubedb/docs#1086).

Related: kubedb/milvus#85, kubedb/apimachinery#1947, kubedb/docs#1086.

Copilot AI balanced review requested due to automatic review settings October 30, 2026 14:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@gitguardian

gitguardian Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
39317783 Triggered Generic Password 07b36c5 pkg/adopt/milvus_test.go View secret
39317783 Triggered Generic Password 07b36c5 pkg/adopt/milvus_test.go View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secrets safely. Learn here the best practices.
  3. Revoke and rotate these secrets.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@Sabnaj-42
Sabnaj-42 force-pushed the milvus-inplace-adoption branch from 58e6c83 to 74f49bc Compare October 30, 2026 14:42
Adopting a Distributed Milvus 2.6 that runs from a Helm chart was a manual
procedure: gather the effective config, write three Secrets and a staged
Milvus, diff the rendered config, then scale each role by hand. The new
command group does it:

- plan RELEASE: read-only. Finds the release's workloads (by
  app.kubernetes.io/instance and the component label, or --workload), merges
  the mounted config files in Milvus' load order, finds the client Service
  and the MilvusVersion for the running image, and checks the WAL
  (Woodpecker on object storage), etcd, object storage and the root
  password, logging in to a running proxy through a port-forward (Milvus
  applies a root password only on first start). --output-dir writes the
  objects for review.
- run RELEASE: creates the Secrets, a state ConfigMap <name>-adopt and the
  Milvus with every role at 0 (orphan-on-delete, DoNotTerminate, external
  etcd with the current root path, autoOps off, write check off), removes a
  chart Service on a name KubeDB needs headless, and stops if the config the
  operator rendered differs from the running one on storage, etcd root path,
  WAL, security or ports. Then hands the roles over, overlap (default; one
  QueryNode at a time) or stop-start, recording each one; re-running resumes.
- status RELEASE: where each role runs.
- rollback RELEASE: deletes the Milvus with orphan-on-delete, moves each
  role back to the chart starting the chart's pods first, restores the
  client Service's selector, labels and ports, and with --cleanup deletes
  what KubeDB left and recreates the removed chart Services. --name covers a
  Milvus adopted by hand.

Several workloads of one role become node groups (e.g. QueryNodes on two
GPU pools). Tested end to end on a cluster running the upstream chart 5.0.19
(Milvus 2.6.15): hand-adoption rollback, plan, overlap run, state rollback,
stop-start run killed and resumed; data bit-exact after each.

Signed-off-by: sabnaj <sabnaj@appscode.com>
@Sabnaj-42
Sabnaj-42 force-pushed the milvus-inplace-adoption branch from 74f49bc to 07b36c5 Compare October 30, 2026 14:54
For a chart without the upstream labels, plan and run need about fifteen
flags (a --workload per workload, the secrets, the timing), repeated on
every command. All four commands now take -f FILE, kind MilvusAdoption
(apiVersion cli.kubedb.com/v1alpha1): release, namespace, name, version,
source (selector, roleLabel, workloads with node groups, configFile), auth,
restrictedSCC, handover (method and timing) and rollback (cleanup).

- A flag given on the command line wins; the file fills only the flags
  that were not given, through the flag set, so the parsing is shared.
- RELEASE may come from the file; an argument that differs is refused. The
  file's namespace is used unless -n is given.
- Unknown keys are errors, and --yes is never read from the file.
- plan --output-dir also writes adopt.yaml with what plan found, ready for
  run -f.

Tested on the cluster: rollback, plan and run driven by the file alone.

Signed-off-by: sabnaj <sabnaj@appscode.com>
Signed-off-by: sabnaj <sabnaj@appscode.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants