From c5d9b297d09bc4c113ee75e3c81383a7af6ad1d6 Mon Sep 17 00:00:00 2001 From: sabnaj Date: Thu, 1 Oct 2026 16:08:28 +0600 Subject: [PATCH 1/6] Add milvus full backup and archiver guide Signed-off-by: sabnaj --- docs/guides/milvus/backup/_index.md | 11 ++ docs/guides/milvus/backup/archiver.md | 240 ++++++++++++++++++++++++++ 2 files changed, 251 insertions(+) create mode 100644 docs/guides/milvus/backup/_index.md create mode 100644 docs/guides/milvus/backup/archiver.md diff --git a/docs/guides/milvus/backup/_index.md b/docs/guides/milvus/backup/_index.md new file mode 100644 index 000000000..96bd0a718 --- /dev/null +++ b/docs/guides/milvus/backup/_index.md @@ -0,0 +1,11 @@ +--- +title: Backup, Restore and Point-in-time Recovery +menu: + docs_{{ .version }}: + identifier: backup-milvus + name: Backup & Point-in-time Recovery + parent: milvus-guides + weight: 60 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- diff --git a/docs/guides/milvus/backup/archiver.md b/docs/guides/milvus/backup/archiver.md new file mode 100644 index 000000000..49e2d1579 --- /dev/null +++ b/docs/guides/milvus/backup/archiver.md @@ -0,0 +1,240 @@ +--- +title: Base Backup and Point-in-time Recovery +menu: + docs_{{ .version }}: + identifier: backup-milvus-archiver + name: Overview + parent: backup-milvus + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# KubeDB Milvus - Base Backup and Point-in-time Recovery + +KubeDB backs up a running Milvus physically: it captures the metadata etcd and the object storage bucket +under one short write fence, and keeps recording every later metadata change and new object version. A +`MilvusArchiver` ties the pieces together. A new `Milvus` can be restored from the archive to the time of any +full backup or, for a Milvus that keeps its write-ahead log on the object storage, to **any point in time** +inside the recorded window. + +## How it works + +| Step | What happens | +|------|--------------| +| Full backup | KubeStash runs the `physical-backup` task. The plugin pauses Milvus garbage collection, denies writes (`forceDeny` in the etcd dynamic config, for at most `fenceTimeout`), flushes, dumps the whole etcd keyspace at **one revision** and copies every new object version into the archive. The write fence is lifted right after the dump. | +| Continuous archiving | A `Sidekick` runs `milvus-archiver archive` next to the database. It records every etcd change and every new object version (versioned object pool, tombstones for deletions) and publishes the recoverable window on the `-incremental-snapshot` Snapshot (`status.components.wal.logStats.start/end`). | +| Restore | A new Milvus with `spec.init.archiver` is created. Before any Milvus process starts, a `RestoreSession` rebuilds the etcd image (base dump plus replay to the requested revision) and the objects as of that time. | + +The archive lives in the `BackupStorage` of the archiver (S3 compatible storage), below +`//`. The change log and the base dumps are encrypted with a key derived from the +archiver's `encryptionSecret`. + +## Point-in-time recovery needs the Woodpecker WAL + +| Milvus | WAL | Base backup | Point-in-time recovery | +|--------|-----|:-----------:|:----------------------:| +| Standalone (default) | RocksMQ on the data PVC | yes | no (only the time of a full backup) | +| Standalone with `spec.wal.type: Woodpecker` | objects in the bucket | yes | yes | +| Distributed | Woodpecker | yes | yes | + +For a RocksMQ standalone Milvus the RocksMQ directories are captured inside the write fence together with +the metadata (tar stream by the `physical-backup` task, or a CSI `VolumeSnapshot` with the `volume-snapshot` +task). The archiver records the condition `PITRUnsupported` on such a database. + +## Before you begin + +- KubeDB, KubeStash and Sidekick are installed, together with the KubeStash catalog that provides `milvus-addon`. +- The Milvus must run with `quotaAndLimits.enabled: true` (KubeDB renders this by default): the write fence relies on it. +- `kubectl create ns demo` + +## Deploy the archiver + +```yaml +apiVersion: archiver.kubedb.com/v1alpha1 +kind: MilvusArchiver +metadata: + name: milvus-archiver + namespace: demo +spec: + pause: false + databases: + namespaces: + from: Same + selector: + matchLabels: + archiver: "true" + retentionPolicy: {name: milvus-retention, namespace: demo} + encryptionSecret: {name: encrypt-secret, namespace: demo} + backupStorage: + ref: {name: s3-storage, namespace: demo} + subDir: milvus + deletionPolicy: WipeOut + fullBackup: + driver: Restic # or VolumeSnapshotter for the data PVC of a RocksMQ standalone Milvus + task: + params: + quiesce: DenyWrites # DenyWrites | Flush | None + fenceSettle: 10s + fenceTimeout: 60s + scheduler: + schedule: "0 */6 * * *" + sessionHistoryLimit: 3 + logBackup: + retentionPeriod: 7d + retentionSchedule: "0 1 * * *" + manifestBackup: + scheduler: + schedule: "0 */6 * * *" +``` + +`quiesce` selects how writes are frozen for the crash-consistent capture: + +- `DenyWrites` (default): writes and DDL are rejected for the length of the dump (about ten seconds including the settle time). +- `Flush`: no fence. The capture is only as consistent as a flush; choose it when `quotaAndLimits` cannot be enabled. +- `None`: no quiescing at all. Not recommended. + +If `quotaAndLimits.enabled` is `false` the full backup fails with an explanatory message unless +`allowQuiesceDowngrade: "true"` is set, which falls back to `Flush`. + +## Opt a Milvus in + +A database opts in by carrying the archiver's label (double opt-in). For point-in-time recovery of a +standalone Milvus set the WAL type at creation; it cannot be changed later. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus + namespace: demo + labels: + archiver: "true" +spec: + version: "2.6.11" + topology: + mode: Standalone + wal: + type: Woodpecker + objectStorage: + configSecret: + name: milvus-storage + storageType: Durable + storage: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 5Gi + deletionPolicy: WipeOut +``` + +KubeDB adds `spec.archiver.ref`, creates the `BackupConfiguration` and, after the **first successful full +backup**, the `Sidekick` and the incremental Snapshot: + +```bash +$ kubectl get backupconfiguration,sidekick,snapshot -n demo +``` + +```bash +$ kubectl get snapshot -n demo milvus-incremental-snapshot -o jsonpath='{.status.components.wal.logStats}' +{"end":"2026-09-30T16:23:04Z","lsn":"rev:552","start":"2026-09-30T16:17:37Z"} +``` + +`start` is the fence time of the oldest full backup that is still kept and `end` is the latest point that can +be restored. `lsn` is the last archived etcd revision. The Milvus conditions `LogBackupLagging`, +`LogBackupDegraded` and `LogBackupGap` mirror the state of the archiver. + +Changes to the archive policy do not interrupt the database: `spec.archiver.pause: true` on the Milvus (or +`spec.pause` on the archiver) stops the sidekick and pauses the schedules. + +## Restore + +Create a new Milvus from the archive. The object storage of the new Milvus must be **empty** and use the same +`rootPath` as the source (the statistics of text indexes embed full object keys); a different bucket is fine. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Milvus +metadata: + name: milvus-restored + namespace: demo +spec: + version: "2.6.11" + topology: + mode: Standalone + wal: + type: Woodpecker + objectStorage: + configSecret: + name: milvus-restore-storage # empty bucket, same rootPath as the source + init: + archiver: + recoveryTimestamp: "2026-09-30T16:22:29Z" + encryptionSecret: {name: encrypt-secret, namespace: demo} + fullDBRepository: {name: milvus-full, namespace: demo} + storageType: Durable + storage: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 5Gi + deletionPolicy: WipeOut +``` + +KubeDB picks the newest full backup at or before `recoveryTimestamp`, replays the change log up to that time +and starts Milvus on the result. The condition `ArchiverRecoveryPlanned` names the chosen base. Distributed +targets use `topology.mode: Distributed` with the same sizes of the source. + +What the operator refuses, with a clear message on the `SuccessfullyDataRestored` condition: + +| Situation | Behaviour | +|-----------|-----------| +| time before the oldest kept full backup | rejected, the earliest restorable time is reported | +| time after the end of the window | clamped to the end of the window (logged in the restore Job) | +| time inside a **log gap** (see below) | rejected, the nearest restorable times before and after the gap are reported | +| target bucket or etcd not empty | rejected, nothing is overwritten | +| target `rootPath`, `dmlChannelNum` or channel prefix differs from the source | rejected | +| target Milvus of another minor version, or an older patch, than the backup | rejected | + +Restored Milvus nodes register themselves again: the session keys of the source are not carried over and the +streaming node assignment of every channel is reset, so the restored Milvus never talks to the old pods. + +## Retention + +`logBackup.retentionPeriod` (`[dwmy]`) bounds the archive. KubeDB runs it as the CronJob `-log-retention` on `logBackup.retentionSchedule`. The retention job keeps the newest full backup +at or before the horizon and removes everything no restore at a later time needs: change-log segments, time +index files, superseded object versions and the older full backups with their Snapshots. A restore to a time +before the kept full backup is no longer possible afterwards. + +## Log gaps + +The change log is continuous only while every etcd revision is recorded. If the archiver is stopped for so +long that etcd compacts revisions it has not yet recorded, the archiver notices (`ErrCompacted`), records the +gap, annotates the incremental Snapshot (`archiver.kubedb.com/log-gap`), sets the `LogBackupGap` condition and +asks KubeStash for a new full backup. Restores to a time after that full backup work again; times inside the +gap cannot be restored. Increase the archiver resources or etcd's `--auto-compaction-retention` if gaps occur. + +## VolumeSnapshot backups + +Set `fullBackup.driver: VolumeSnapshotter` and `fullBackup.task.params.volumeSnapshotClassName` to snapshot +the data PVC of a RocksMQ standalone Milvus inside the write fence, instead of copying the RocksMQ directories. +Metadata and objects are archived exactly as with the Restic driver. A restore creates the data PVC from the +VolumeSnapshot. + +## Limitations + +- Only S3 compatible `BackupStorage` providers can hold the archive. +- The restored Milvus must use the same `rootPath` and an empty bucket. +- Collections, indexes and users, roles and privileges are restored as they were; nothing is selectable. +- Backing up a Milvus that uses an external etcd with TLS/authentication requires the client secrets to be + readable by the archiver service account. +- The write fence denies writes for about ten seconds per full backup. + +## Cleanup + +```bash +kubectl delete milvus -n demo milvus milvus-restored +kubectl delete milvusarchiver -n demo milvus-archiver +``` From f8ea9d5e19d99ad5d6063ff102c959060456b18a Mon Sep 17 00:00:00 2001 From: sabnaj Date: Tue, 6 Oct 2026 15:32:51 +0600 Subject: [PATCH 2/6] Milvus archiver guide: Woodpecker-only WAL, warm cache, detach and write fence safety Signed-off-by: sabnaj --- docs/guides/milvus/backup/archiver.md | 91 ++++++++++++++++----- docs/guides/milvus/quickstart/standalone.md | 4 + 2 files changed, 73 insertions(+), 22 deletions(-) diff --git a/docs/guides/milvus/backup/archiver.md b/docs/guides/milvus/backup/archiver.md index 49e2d1579..b2b5f91ae 100644 --- a/docs/guides/milvus/backup/archiver.md +++ b/docs/guides/milvus/backup/archiver.md @@ -17,8 +17,8 @@ section_menu_id: guides KubeDB backs up a running Milvus physically: it captures the metadata etcd and the object storage bucket under one short write fence, and keeps recording every later metadata change and new object version. A `MilvusArchiver` ties the pieces together. A new `Milvus` can be restored from the archive to the time of any -full backup or, for a Milvus that keeps its write-ahead log on the object storage, to **any point in time** -inside the recorded window. +full backup or to **any point in time** inside the recorded window. This works for Standalone and Distributed +Milvus alike: KubeDB always runs Milvus with the Woodpecker write-ahead log, which lives in the object storage. ## How it works @@ -32,17 +32,30 @@ The archive lives in the `BackupStorage` of the archiver (S3 compatible storage) `//`. The change log and the base dumps are encrypted with a key derived from the archiver's `encryptionSecret`. -## Point-in-time recovery needs the Woodpecker WAL +## The write-ahead log -| Milvus | WAL | Base backup | Point-in-time recovery | -|--------|-----|:-----------:|:----------------------:| -| Standalone (default) | RocksMQ on the data PVC | yes | no (only the time of a full backup) | -| Standalone with `spec.wal.type: Woodpecker` | objects in the bucket | yes | yes | -| Distributed | Woodpecker | yes | yes | +Every Milvus that KubeDB creates uses the Woodpecker write-ahead log, stored as objects in the Milvus bucket +(`mq.type: woodpecker`, `woodpecker.storage.type: minio`). There is no option to choose another one, and a +configuration that sets a different `mq.type` or `woodpecker.storage.type: local` is refused, because the +archive must see the write-ahead log in the object storage. -For a RocksMQ standalone Milvus the RocksMQ directories are captured inside the write fence together with -the metadata (tar stream by the `physical-backup` task, or a CSI `VolumeSnapshot` with the `volume-snapshot` -task). The archiver records the condition `PITRUnsupported` on such a database. +| Milvus | Full backup | Point-in-time recovery | VolumeSnapshot (warm cache) | +|--------|:-----------:|:----------------------:|:---------------------------:| +| Standalone | yes | yes | yes | +| Distributed | yes | yes | no (use the Restic driver) | +| Standalone created by an older KubeDB (RocksMQ) | no | no | no | + +### Existing Standalone Milvus (RocksMQ) + +A Standalone Milvus created by a KubeDB release before the archiver keeps its write-ahead log in RocksMQ on its +data PVC. Milvus cannot switch the write-ahead log of a running instance without losing data that is not yet +flushed, so KubeDB leaves such a database exactly as it is. The operator marks it with the annotation +`kubedb.com/milvus-wal: rocksmq` (every other Milvus gets `woodpecker`); the annotation cannot be changed. + +Such a database cannot be archived: `spec.archiver` is rejected, and if an archiver is attached anyway the Milvus +reports the condition `ArchiverUnsupported` and no backup is configured. To move it to the Woodpecker write-ahead log, take a +logical backup (KubeStash `milvus-addon`, task `logical-backup`) and restore it into a new Milvus +(task `logical-backup-restore`), then switch the clients. ## Before you begin @@ -73,7 +86,7 @@ spec: subDir: milvus deletionPolicy: WipeOut fullBackup: - driver: Restic # or VolumeSnapshotter for the data PVC of a RocksMQ standalone Milvus + driver: Restic # or VolumeSnapshotter (Standalone): also snapshots the data PVC as a warm cache task: params: quiesce: DenyWrites # DenyWrites | Flush | None @@ -101,8 +114,7 @@ If `quotaAndLimits.enabled` is `false` the full backup fails with an explanatory ## Opt a Milvus in -A database opts in by carrying the archiver's label (double opt-in). For point-in-time recovery of a -standalone Milvus set the WAL type at creation; it cannot be changed later. +A database opts in by carrying the archiver's label (double opt-in). ```yaml apiVersion: kubedb.com/v1alpha2 @@ -116,8 +128,6 @@ spec: version: "2.6.11" topology: mode: Standalone - wal: - type: Woodpecker objectStorage: configSecret: name: milvus-storage @@ -149,6 +159,20 @@ be restored. `lsn` is the last archived etcd revision. The Milvus conditions `Lo Changes to the archive policy do not interrupt the database: `spec.archiver.pause: true` on the Milvus (or `spec.pause` on the archiver) stops the sidekick and pauses the schedules. +To stop archiving, remove the archiver label and `spec.archiver` from the Milvus. KubeDB then deletes the sidekick +and the retention CronJob and pauses the `BackupConfiguration`; the repositories and Snapshots stay, so the existing +backups can still be restored. Attaching an archiver again resumes the schedules. Deleting the `MilvusArchiver` +itself instead removes the `BackupConfiguration`, and the archiver's `deletionPolicy` then decides whether the +backups are deleted. + +## Write fence safety + +A full backup briefly denies writes (the fence). The backup lifts it itself, also when its pod is terminated +(SIGTERM) or when `fenceTimeout` expires. If the pod is killed without a chance to clean up, the fence keeps a +marker with a deadline (`fenceTimeout` plus one minute): the sidekick lifts an expired fence within 30 seconds, and +the next backup lifts it before it starts. Databases without a sidekick (VolumeSnapshotter archivers) recover at +the next scheduled backup. + ## Restore Create a new Milvus from the archive. The object storage of the new Milvus must be **empty** and use the same @@ -164,8 +188,6 @@ spec: version: "2.6.11" topology: mode: Standalone - wal: - type: Woodpecker objectStorage: configSecret: name: milvus-restore-storage # empty bucket, same rootPath as the source @@ -174,6 +196,7 @@ spec: recoveryTimestamp: "2026-09-30T16:22:29Z" encryptionSecret: {name: encrypt-secret, namespace: demo} fullDBRepository: {name: milvus-full, namespace: demo} + manifestRepository: {name: milvus-manifest, namespace: demo} # optional: keep the source's credentials storageType: Durable storage: accessModes: [ReadWriteOnce] @@ -187,6 +210,13 @@ KubeDB picks the newest full backup at or before `recoveryTimestamp`, replays th and starts Milvus on the result. The condition `ArchiverRecoveryPlanned` names the chosen base. Distributed targets use `topology.mode: Distributed` with the same sizes of the source. +With `manifestRepository`, KubeDB first restores the source's auth secret as the new Milvus's own auth secret +(`milvus-restored-auth` above, or the name set in `spec.authSecret.name`), before it would generate one. The +restored Milvus therefore accepts the source's credentials, and the source's secret is left untouched, so the +restore can run in the source's namespace. If that auth secret already exists when the restore starts, it is +kept and the source's credentials are not restored. Only the auth secret is restored. The rendered configuration is +generated for the new Milvus, and a custom configuration secret must be referenced in the new Milvus's spec. + What the operator refuses, with a clear message on the `SuccessfullyDataRestored` condition: | Situation | Behaviour | @@ -197,6 +227,7 @@ What the operator refuses, with a clear message on the `SuccessfullyDataRestored | target bucket or etcd not empty | rejected, nothing is overwritten | | target `rootPath`, `dmlChannelNum` or channel prefix differs from the source | rejected | | target Milvus of another minor version, or an older patch, than the backup | rejected | +| target Milvus runs another Woodpecker version than the backup | rejected | Restored Milvus nodes register themselves again: the session keys of the source are not carried over and the streaming node assignment of every channel is reset, so the restored Milvus never talks to the old pods. @@ -218,10 +249,24 @@ gap cannot be restored. Increase the archiver resources or etcd's `--auto-compac ## VolumeSnapshot backups -Set `fullBackup.driver: VolumeSnapshotter` and `fullBackup.task.params.volumeSnapshotClassName` to snapshot -the data PVC of a RocksMQ standalone Milvus inside the write fence, instead of copying the RocksMQ directories. -Metadata and objects are archived exactly as with the Restic driver. A restore creates the data PVC from the -VolumeSnapshot. +Set `fullBackup.driver: VolumeSnapshotter` and `fullBackup.task.params.volumeSnapshotClassName` to also take a +CSI `VolumeSnapshot` of the data PVC of a **Standalone** Milvus. The snapshot is taken inside the same write +fence, right after the metadata dump. Metadata and objects are archived exactly as with the Restic driver, so +the archive alone is a complete backup. + +The data PVC holds only the local cache of Milvus (loaded segments, mmap files and disk indexes). A restore from +such a backup creates the data PVC of the new Milvus from the VolumeSnapshot, so it starts with a **warm cache** +instead of downloading everything again. The condition `WarmCacheRestored` reports whether the snapshot was used. +If it cannot be used (deleted, not ready, or in another namespace than the restored Milvus) the restore still +succeeds and Milvus starts with a cold cache. + +Requirements: + +- the Milvus data PVC uses a CSI `StorageClass` (`spec.storage.storageClassName`) and a matching + `VolumeSnapshotClass` exists; +- a failed snapshot fails the full backup session, so the backup you asked for is never silently incomplete; +- a Distributed Milvus has no PVC worth snapshotting: attached to an archiver with the VolumeSnapshotter driver it + reports the condition `ArchiverUnsupported` and is not backed up; use the Restic driver. ## Limitations @@ -231,6 +276,8 @@ VolumeSnapshot. - Backing up a Milvus that uses an external etcd with TLS/authentication requires the client secrets to be readable by the archiver service account. - The write fence denies writes for about ten seconds per full backup. +- A Standalone Milvus created by a KubeDB release before the archiver (RocksMQ write-ahead log) cannot be + archived; migrate it with a logical backup. ## Cleanup diff --git a/docs/guides/milvus/quickstart/standalone.md b/docs/guides/milvus/quickstart/standalone.md index 9a5236fea..87512b7df 100644 --- a/docs/guides/milvus/quickstart/standalone.md +++ b/docs/guides/milvus/quickstart/standalone.md @@ -138,6 +138,10 @@ data-milvus-standalone-0 Bound pvc-a6333ee2-f0ab-4ec2-8437-599d270b9ed0 1 The internal etcd metadata store provisions its own PVCs (`etcd-data--etcd-*`), and MinIO has separate PVCs as well. +The data PVC holds Milvus' local cache (loaded segments, mmap files, disk indexes). The write-ahead log is not on +it: KubeDB runs every Milvus with the Woodpecker write-ahead log, which is stored in the object storage bucket +together with the data files. + ### Auth Secret Milvus authentication is enabled by default. Because `spec.authSecret` was not provided, KubeDB auto-generates a basic-auth secret named `-auth` with a `root` user and a random password: From 7adac3afcaea9ed53e757400821d5642d6baa1fd Mon Sep 17 00:00:00 2001 From: sabnaj Date: Thu, 8 Oct 2026 11:30:34 +0600 Subject: [PATCH 3/6] Milvus archiver guide: supported Milvus versions and restore rules Signed-off-by: sabnaj --- docs/guides/milvus/backup/archiver.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/guides/milvus/backup/archiver.md b/docs/guides/milvus/backup/archiver.md index b2b5f91ae..8f4bc4112 100644 --- a/docs/guides/milvus/backup/archiver.md +++ b/docs/guides/milvus/backup/archiver.md @@ -112,6 +112,21 @@ spec: If `quotaAndLimits.enabled` is `false` the full backup fails with an explanatory message unless `allowQuiesceDowngrade: "true"` is set, which falls back to `Flush`. +## Supported Milvus versions + +| Milvus | Status | +|--------|--------| +| 2.6.11 | Verified: Standalone, Distributed, VolumeSnapshotter, point-in-time recovery, manifest restore | +| 2.6.9 | Verified: Standalone and Distributed (one replica per role), point-in-time recovery | +| 2.6.7 | Verified: Standalone, point-in-time recovery. VolumeSnapshotter and Distributed were not tested | +| 3.x | Not supported: the webhook rejects an archiver on Milvus 3.x until it is validated | + +A backup restores into a Milvus of the same minor version and the same or a newer patch, and only into a Milvus that +runs the same Woodpecker library (the metadata layout in etcd belongs to it). Milvus 2.6.7 ships Woodpecker v0.1.12, +2.6.9 and later ship v0.1.13, so a 2.6.7 backup cannot be restored into 2.6.9 or later and the restore is refused with +that message. A version that is missing from the table in `milvus-archiver` (`pkg/physical/meta/woodpecker.go`) can +only be restored into the same Milvus version. + ## Opt a Milvus in A database opts in by carrying the archiver's label (double opt-in). From a2eaf3e8f85449c21b9f9f86ce54bd5fce2e45e5 Mon Sep 17 00:00:00 2001 From: sabnaj Date: Wed, 28 Oct 2026 17:03:34 +0600 Subject: [PATCH 4/6] Milvus archiver guide: add backend prerequisites, 2.6.15/3.0.0, Running incremental snapshot Signed-off-by: sabnaj --- docs/guides/milvus/backup/archiver.md | 107 +++++++++++++++++++++++++- 1 file changed, 105 insertions(+), 2 deletions(-) diff --git a/docs/guides/milvus/backup/archiver.md b/docs/guides/milvus/backup/archiver.md index 8f4bc4112..92fac4881 100644 --- a/docs/guides/milvus/backup/archiver.md +++ b/docs/guides/milvus/backup/archiver.md @@ -63,6 +63,98 @@ logical backup (KubeStash `milvus-addon`, task `logical-backup`) and restore it - The Milvus must run with `quotaAndLimits.enabled: true` (KubeDB renders this by default): the write fence relies on it. - `kubectl create ns demo` +## Prepare the backend + +The archiver needs an S3 compatible `BackupStorage` for the archive, a `RetentionPolicy` and an encryption key. The +Milvus itself reads its own object storage (bucket and credentials) from a secret. This guide uses the MinIO of +the [quickstart](/docs/guides/milvus/quickstart/standalone.md); replace the endpoint, bucket names and credentials +with yours. Create three buckets first (any S3 client works): `kubestash` for the archive, `milvus-src` for the +Milvus and `milvus-restore` for the restored Milvus. The restore bucket must be **empty**. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: s3-secret + namespace: demo +stringData: + AWS_ACCESS_KEY_ID: minio + AWS_SECRET_ACCESS_KEY: minio123 +--- +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: s3-storage + namespace: demo +spec: + storage: + provider: s3 + s3: + bucket: kubestash + endpoint: http://minio.demo.svc.cluster.local:80 + region: us-east-1 + prefix: milvus-backup + secretName: s3-secret + usagePolicy: + allowedNamespaces: + from: Same + default: false + deletionPolicy: WipeOut +--- +apiVersion: storage.kubestash.com/v1alpha1 +kind: RetentionPolicy +metadata: + name: milvus-retention + namespace: demo +spec: + maxRetentionPeriod: 7d + successfulSnapshots: + last: 5 + failedSnapshots: + last: 2 + usagePolicy: + allowedNamespaces: + from: Same +--- +apiVersion: v1 +kind: Secret +metadata: + name: encrypt-secret + namespace: demo +stringData: + RESTIC_PASSWORD: changeme-restic-password +--- +apiVersion: v1 +kind: Secret +metadata: + name: milvus-storage + namespace: demo +stringData: + address: minio.demo.svc.cluster.local + port: "80" + accessKeyID: minio + secretAccessKey: minio123 + useSSL: "false" + bucketName: milvus-src + rootPath: milvus +--- +apiVersion: v1 +kind: Secret +metadata: + name: milvus-restore-storage + namespace: demo +stringData: + address: minio.demo.svc.cluster.local + port: "80" + accessKeyID: minio + secretAccessKey: minio123 + useSSL: "false" + bucketName: milvus-restore + rootPath: milvus +``` + +`rootPath` is the prefix inside the Milvus bucket; the restored Milvus must use the same one. + ## Deploy the archiver ```yaml @@ -116,10 +208,11 @@ If `quotaAndLimits.enabled` is `false` the full backup fails with an explanatory | Milvus | Status | |--------|--------| +| 3.0.0 | Verified: Standalone, point-in-time recovery, manifest restore | +| 2.6.15 | Verified: Standalone, point-in-time recovery, manifest restore | | 2.6.11 | Verified: Standalone, Distributed, VolumeSnapshotter, point-in-time recovery, manifest restore | | 2.6.9 | Verified: Standalone and Distributed (one replica per role), point-in-time recovery | | 2.6.7 | Verified: Standalone, point-in-time recovery. VolumeSnapshotter and Distributed were not tested | -| 3.x | Not supported: the webhook rejects an archiver on Milvus 3.x until it is validated | A backup restores into a Milvus of the same minor version and the same or a newer patch, and only into a Milvus that runs the same Woodpecker library (the metadata layout in etcd belongs to it). Milvus 2.6.7 ships Woodpecker v0.1.12, @@ -162,6 +255,15 @@ backup**, the `Sidekick` and the incremental Snapshot: $ kubectl get backupconfiguration,sidekick,snapshot -n demo ``` +The incremental Snapshot stays in phase `Running` for as long as the archiver ships the change log, and turns +`Succeeded` once the archiving is stopped: + +```bash +$ kubectl get snapshot -n demo milvus-incremental-snapshot +NAME REPOSITORY SESSION SNAPSHOT-TIME DELETION-POLICY PHASE AGE +milvus-incremental-snapshot milvus-full Delete Running 2m +``` + ```bash $ kubectl get snapshot -n demo milvus-incremental-snapshot -o jsonpath='{.status.components.wal.logStats}' {"end":"2026-09-30T16:23:04Z","lsn":"rev:552","start":"2026-09-30T16:17:37Z"} @@ -190,7 +292,8 @@ the next scheduled backup. ## Restore -Create a new Milvus from the archive. The object storage of the new Milvus must be **empty** and use the same +Create a new Milvus from the archive. Pick `recoveryTimestamp` inside the recoverable window of +`milvus-incremental-snapshot` (`logStats.start` to `logStats.end`, UTC). The object storage of the new Milvus must be **empty** and use the same `rootPath` as the source (the statistics of text indexes embed full object keys); a different bucket is fine. ```yaml From 2df8c4b5dfd1f92302fd57d77647920982e8df88 Mon Sep 17 00:00:00 2001 From: sabnaj Date: Thu, 29 Oct 2026 15:32:43 +0600 Subject: [PATCH 5/6] Milvus archiver guide: document the write fence length and settings for large databases Signed-off-by: sabnaj --- docs/guides/milvus/backup/archiver.md | 34 ++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/docs/guides/milvus/backup/archiver.md b/docs/guides/milvus/backup/archiver.md index 92fac4881..060326a59 100644 --- a/docs/guides/milvus/backup/archiver.md +++ b/docs/guides/milvus/backup/archiver.md @@ -24,7 +24,7 @@ Milvus alike: KubeDB always runs Milvus with the Woodpecker write-ahead log, whi | Step | What happens | |------|--------------| -| Full backup | KubeStash runs the `physical-backup` task. The plugin pauses Milvus garbage collection, denies writes (`forceDeny` in the etcd dynamic config, for at most `fenceTimeout`), flushes, dumps the whole etcd keyspace at **one revision** and copies every new object version into the archive. The write fence is lifted right after the dump. | +| Full backup | KubeStash runs the `physical-backup` task. The plugin pauses Milvus garbage collection, denies writes (`forceDeny` in the etcd dynamic config, for at most `fenceTimeout`), flushes, dumps the whole etcd keyspace at **one revision** and copies the new write-ahead log objects (`wp/`) into the archive. The write fence is lifted after that copy; the data objects are copied afterwards, with writes allowed again. | | Continuous archiving | A `Sidekick` runs `milvus-archiver archive` next to the database. It records every etcd change and every new object version (versioned object pool, tombstones for deletions) and publishes the recoverable window on the `-incremental-snapshot` Snapshot (`status.components.wal.logStats.start/end`). | | Restore | A new Milvus with `spec.init.archiver` is created. Before any Milvus process starts, a `RestoreSession` rebuilds the etcd image (base dump plus replay to the requested revision) and the objects as of that time. | @@ -197,7 +197,7 @@ spec: `quiesce` selects how writes are frozen for the crash-consistent capture: -- `DenyWrites` (default): writes and DDL are rejected for the length of the dump (about ten seconds including the settle time). +- `DenyWrites` (default): writes and DDL are rejected from the settle time until the metadata dump and the copy of the write-ahead log objects are done. That is about ten seconds for a small database and grows with the amount of write-ahead log that is not yet in the archive (see [Large databases](#large-databases)). - `Flush`: no fence. The capture is only as consistent as a flush; choose it when `quotaAndLimits` cannot be enabled. - `None`: no quiescing at all. Not recommended. @@ -350,6 +350,34 @@ What the operator refuses, with a clear message on the `SuccessfullyDataRestored Restored Milvus nodes register themselves again: the session keys of the source are not carried over and the streaming node assignment of every channel is reset, so the restored Milvus never talks to the old pods. +## Large databases + +The write fence is held until the new write-ahead log objects are copied into the archive. After a big ingest the +write-ahead log can be several GiB, so the fence lasts as long as that copy takes. If `fenceTimeout` (default `60s`) +expires first, the full backup aborts, lifts the fence and fails, and nothing is archived. In a test with about 1.3 +million 768-dimensional vectors (about 3.7 GiB of vectors, several GiB of write-ahead log) the default aborted the +backup, while `fenceTimeout: 10m` let it finish (the whole full backup took about three minutes). + +For databases that take in a lot of data, set a larger timeout on the archiver before the first full backup: + +```yaml +spec: + fullBackup: + task: + params: + fenceTimeout: 10m +``` + +Writes are rejected while the fence is up, so clients see errors such as `quota exceeded ... force deny writing` +until the backup lifts it; retry them or schedule full backups for quiet hours. Changing `fenceTimeout` on an +archiver that already created its `BackupConfiguration` did not update that `BackupConfiguration` in our test; edit +the `fenceTimeout` parameter of the `full-backup` session in the `BackupConfiguration` as well. + +Plan the object storage too. The archive holds the objects of the database plus its write-ahead log, and a restore +writes them again into the new bucket, so the free space needed is several times the size of the vectors. In the +test above, about 4.3 GB of vectors used 23 GB in the Milvus bucket, 24 GB in the archive and 23 GB in the restored +bucket. + ## Retention `logBackup.retentionPeriod` (`[dwmy]`) bounds the archive. KubeDB runs it as the CronJob `-log-retention` on `logBackup.retentionSchedule`. The retention job keeps the newest full backup @@ -393,7 +421,7 @@ Requirements: - Collections, indexes and users, roles and privileges are restored as they were; nothing is selectable. - Backing up a Milvus that uses an external etcd with TLS/authentication requires the client secrets to be readable by the archiver service account. -- The write fence denies writes for about ten seconds per full backup. +- The write fence denies writes for the length of the metadata dump and the copy of the write-ahead log, about ten seconds for a small database and longer for a large one (see [Large databases](#large-databases)). - A Standalone Milvus created by a KubeDB release before the archiver (RocksMQ write-ahead log) cannot be archived; migrate it with a logical backup. From 91bd3ab1ec9ecfab84127c55b15f24db7bc7e988 Mon Sep 17 00:00:00 2001 From: sabnaj Date: Sun, 1 Nov 2026 13:14:21 +0600 Subject: [PATCH 6/6] Milvus archiver guide: document the dedicated external etcd Signed-off-by: sabnaj --- docs/guides/milvus/backup/archiver.md | 36 +++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/docs/guides/milvus/backup/archiver.md b/docs/guides/milvus/backup/archiver.md index 060326a59..a228e89ab 100644 --- a/docs/guides/milvus/backup/archiver.md +++ b/docs/guides/milvus/backup/archiver.md @@ -378,6 +378,39 @@ writes them again into the new bucket, so the free space needed is several times test above, about 4.3 GB of vectors used 23 GB in the Milvus bucket, 24 GB in the archive and 23 GB in the restored bucket. +## External etcd + +By default KubeDB creates the metadata etcd of a Milvus itself. A Milvus with `spec.metaStorage.externallyManaged: true` +can use the archiver too, if the etcd is **dedicated to that Milvus**: the backup dumps the whole etcd keyspace, +Woodpecker keeps its metadata at the etcd root, and a restore needs a completely empty etcd. An etcd that other +applications or other Milvus instances use cannot be backed up or restored; the operator cannot detect this, so it is up to you. + +```yaml +spec: + metaStorage: + externallyManaged: true + endpoints: # list every member: the operator checks the member count against the endpoints + - http://etcd-a-0.etcd-a-pods.demo.svc.cluster.local:2379 + - http://etcd-a-1.etcd-a-pods.demo.svc.cluster.local:2379 + - http://etcd-a-2.etcd-a-pods.demo.svc.cluster.local:2379 + rootPath: milvus-etcd # optional, the etcd key prefix of Milvus + authSecret: {name: etcd-a-auth} # optional basic-auth Secret (username, password) in the Milvus namespace + clientTLSSecret: {name: etcd-a-client-tls} # optional ca.crt, tls.crt, tls.key +``` + +The archiver reads `authSecret` and `clientTLSSecret` from the Milvus namespace, like Milvus does. Two things the +operator does for its own etcd must be done by you on an external one: + +- Keep revision history for the change-log archiver: set periodic auto-compaction with a retention of about one hour + (`--auto-compaction-mode=periodic --auto-compaction-retention=1h`; for a KubeDB `Etcd`, + `spec.configuration.tuning.autoCompactionMode: periodic` and `autoCompactionRetention: "1h"`). +- Give a restore its own, empty etcd: the restored Milvus points `spec.metaStorage` at a second dedicated etcd (its + etcd `rootPath` may differ from the source's; the object storage `rootPath` may not). Restoring into the etcd of a running Milvus is refused. + +This was verified with a password-protected 3-member KubeDB `Etcd` created separately from the Milvus (Milvus 2.6.15, +point-in-time restore into a second empty etcd, manifest restore included). Client TLS to an external etcd is +implemented but has not been tested. + ## Retention `logBackup.retentionPeriod` (`[dwmy]`) bounds the archive. KubeDB runs it as the CronJob `-log-retention` on `logBackup.retentionSchedule`. The retention job keeps the newest full backup @@ -419,8 +452,7 @@ Requirements: - Only S3 compatible `BackupStorage` providers can hold the archive. - The restored Milvus must use the same `rootPath` and an empty bucket. - Collections, indexes and users, roles and privileges are restored as they were; nothing is selectable. -- Backing up a Milvus that uses an external etcd with TLS/authentication requires the client secrets to be - readable by the archiver service account. +- An external etcd must be dedicated to one Milvus (see [External etcd](#external-etcd)); a shared etcd cannot be backed up or restored. - The write fence denies writes for the length of the metadata dump and the copy of the write-ahead log, about ten seconds for a small database and longer for a large one (see [Large databases](#large-databases)). - A Standalone Milvus created by a KubeDB release before the archiver (RocksMQ write-ahead log) cannot be archived; migrate it with a logical backup.