diff --git a/docs/guides/kubevirt/_index.md b/docs/guides/kubevirt/_index.md new file mode 100644 index 0000000..ac5d4d9 --- /dev/null +++ b/docs/guides/kubevirt/_index.md @@ -0,0 +1,10 @@ +--- +title: Backup & Restore KubeVirt VirtualMachines | KubeStash +menu: + docs_{{ .version }}: + identifier: kubevirt + name: KubeVirt VirtualMachines + parent: guides + weight: 25 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/kubevirt/overview/images/backup_overview.svg b/docs/guides/kubevirt/overview/images/backup_overview.svg new file mode 100644 index 0000000..20cdcc5 --- /dev/null +++ b/docs/guides/kubevirt/overview/images/backup_overview.svg @@ -0,0 +1,112 @@ + + + + + + + + + + + + + + + + User + KubeStash Operator + Backup Job (kubevirt-plugin) + Backend + + + + + BackupConfiguration + target: kubevirt.io/VirtualMachine + + + BackupStorage + + credential Secret + + + RetentionPolicy + + + + + Repository (initialized) + + CronJob per session + + BackupSession + + Snapshot + one per Repository; + components registered up front + + + + + 1. Walk the VM graph + VM, DataVolumes, PVCs, Secrets, + ConfigMaps, ServiceAccounts, + ControllerRevisions + sanitize + CDI annotations + + + 2. Cut the point-in-time + stopped: mount disk PVCs directly + + + 3. One worker Job per disk + streams the disk with restic + + + + Snapshot components + + manifest + + volume-<pvc-1> + + volume-<pvc-2> + encrypted, deduplicated + restic repository + Succeeded only if every + component succeeded + + + + Running VirtualMachine — KubeVirt owns the freeze + + + VirtualMachine + + VirtualMachineInstance + holds the disks (RWO) + + + VirtualMachineSnapshot + freeze guest FS → + snapshot → unfreeze + + + VolumeSnapshot + one per disk → + temporary PVC clone + + qemu-guest-agent present → filesystem-consistent · absent → crash-consistent (warning) + Temporary PVCs and the VirtualMachineSnapshot are deleted after the upload. + A stopped VirtualMachine skips this lane entirely — its disks are already at rest. + + + + + + + creates + + + + + + diff --git a/docs/guides/kubevirt/overview/images/restore_overview.svg b/docs/guides/kubevirt/overview/images/restore_overview.svg new file mode 100644 index 0000000..641f9bb --- /dev/null +++ b/docs/guides/kubevirt/overview/images/restore_overview.svg @@ -0,0 +1,101 @@ + + + + + + + + + + + + + + + + + + User + KubeStash Operator + Restore Job (kubevirt-plugin) + Result + + + + + RestoreSession + dataSource: repository + + snapshot + no spec.target + + + Backend + manifest + volume-<pvc> + components + + + + + resolve Addon + and Function + + create restore Job + a nil target is treated as found + + + + + + 1. Restore the disks first + create destination PVCs from the + backed up manifests + one worker Job per disk streams + its data back with restic + + + 2. Then apply the manifests + ServiceAccount → Secret → ConfigMap + → ControllerRevision → PVC + → DataVolume → VirtualMachine + VirtualMachineInstance and + virt-launcher Pods are never applied + a failed disk stops the VM being applied + + + + CDI adopts the PVCs + populatedFor annotation + on each restored PVC + DataVolumes go Succeeded + without re-importing + + + VirtualMachine + restored, Stopped + start it to boot + cold boot + + + On start, KubeVirt creates + the VirtualMachineInstance + and its virt-launcher Pod + + + + Why the disks are restored before the manifests + When a DataVolume is applied, CDI decides whether to import its contents. A PVC that already exists and carries + cdi.kubevirt.io/storage.populatedFor is adopted as-is, so the restored disk data is kept. + Applying the DataVolume first would let CDI start importing from the original source and overwrite what was restored, + so the restore Job never applies the VirtualMachine until every disk component has succeeded. + A disk whose DataVolume was still importing at backup time has no volume component — CDI simply imports it again. + + + + + reads the Snapshot components + + + + + + diff --git a/docs/guides/kubevirt/overview/index.md b/docs/guides/kubevirt/overview/index.md new file mode 100644 index 0000000..c524f66 --- /dev/null +++ b/docs/guides/kubevirt/overview/index.md @@ -0,0 +1,135 @@ +--- +title: Backup and Restore KubeVirt VirtualMachine Overview | KubeStash +description: An overview on how backup and restore of a KubeVirt VirtualMachine works in KubeStash. +menu: + docs_{{ .version }}: + identifier: kubevirt-overview + name: How Backup and Restore works? + parent: kubevirt + weight: 10 +product_name: kubestash +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Backup and Restore KubeVirt VirtualMachines using KubeStash + +This guide will show you how KubeStash backs up and restores a KubeVirt `VirtualMachine`, including both its definition and the data on its disks. + +## Before You Begin + +- You should be familiar with the following `KubeStash` concepts: + - [BackupConfiguration](/docs/concepts/crds/backupconfiguration/index.md) + - [BackupSession](/docs/concepts/crds/backupsession/index.md) + - [RestoreSession](/docs/concepts/crds/restoresession/index.md) + - [BackupStorage](/docs/concepts/crds/backupstorage/index.md) + - [Snapshot](/docs/concepts/crds/snapshot/index.md) + +## What Gets Backed Up + +A `VirtualMachine` is not a single object. Running one depends on a graph of other objects, and restoring it requires all of them. KubeStash walks that graph and captures: + +- The `VirtualMachine` itself +- The `DataVolume`s and `PersistentVolumeClaim`s its disks are backed by +- `Secret`s, `ConfigMap`s and `ServiceAccount`s referenced as volumes, including cloud-init `userDataSecretRef` / `networkDataSecretRef` and access-credential secrets +- The `ControllerRevision`s that pin an instancetype or preference +- Backend-state PVCs for a persistent TPM or EFI, and memory-dump PVCs + +Alongside those manifests, the data on every disk PVC is copied to the backend. Runtime objects are deliberately **not** captured: the `VirtualMachineInstance` and its `virt-launcher` Pod are recreated by KubeVirt, so a restored VM cold-boots. + +This produces a single `Snapshot` per `Repository` whose `status.components` holds one entry per unit of work: + +| Component | Content | +|---|---| +| `manifest` | The sanitized manifests of the whole dependency graph | +| `volume-` (one per disk) | The disk data — a file tree for a `Filesystem` PVC, or a raw `disk.img` stream for a `Block` PVC | + +Because each disk is its own component, you can restore a single disk or only the manifests. + +## Stopped and Running VirtualMachines + +KubeStash picks the backup path from the state of the VM. You do not configure this. + +**Stopped VM.** The disks are at rest, so the backup Job mounts the disk PVCs directly and copies them. No snapshot machinery is involved. + +**Running VM.** The disks are held by `virt-launcher`, so KubeStash creates a single `VirtualMachineSnapshot` and lets KubeVirt cut the point-in-time: KubeVirt freezes the guest filesystem, takes one CSI `VolumeSnapshot` per disk, and unfreezes. KubeStash then provisions a temporary PVC from each `VolumeSnapshot`, copies those to the backend, and deletes the temporary PVCs and the `VirtualMachineSnapshot` afterwards. Delegating the freeze to KubeVirt means the guest is never left frozen if the backup Job dies. + +The consistency of a running-VM backup depends on the guest agent. With `qemu-guest-agent` installed, KubeVirt freezes the filesystem and the backup is filesystem-consistent. Without it, the snapshot is crash-consistent — the backup still succeeds and records a warning. + +Backing up a running VM therefore requires CSI snapshot support with a default `VolumeSnapshotClass` for the disks' `StorageClass`, and the KubeVirt `Snapshot` feature gate. A stopped VM needs none of these. + +## How Backup Process Works + +The following diagram shows how KubeStash takes backup of a `VirtualMachine`. Open the image in a new tab to see the enlarged version. + +
+ KubeStash VirtualMachine Backup Flow +
Fig: Backup process of a VirtualMachine in KubeStash
+
+ +The backup process consists of the following steps: + +1. At first, a user creates a `Secret`. This secret holds the credentials to access the backend where the backed up data will be stored. + +2. Then, she creates a `BackupStorage` custom resource that specifies the backend information, along with the `Secret` containing the credentials needed to access the backend. + +3. KubeStash operator watches for `BackupStorage` custom resources. When it finds a `BackupStorage` object, it initializes the `BackupStorage` by uploading the `metadata.yaml` file into the target storage. + +4. Then, she creates a `BackupConfiguration` custom resource that specifies the targeted `VirtualMachine`, the Addon info with a specified task, etc. It also provides information about one or more repositories, each indicating a path and a `BackupStorage` for storing the backed-up data. + +5. KubeStash operator watches for `BackupConfiguration` objects. It resolves the target to confirm the `VirtualMachine` exists. + +6. Once the KubeStash operator finds a `BackupConfiguration` object, it creates `Repository` with the information specified in the `BackupConfiguration`. + +7. KubeStash operator watches for `Repository` custom resources. When it finds the `Repository` object, it Initializes `Repository` by uploading `repository.yaml` file into the `spec.sessions[*].repositories[*].directory` path specified in `BackupConfiguration`. + +8. Then, it creates a `CronJob` for each session with the schedule specified in `BackupConfiguration` to trigger backup periodically. + +9. On the next scheduled slot, the `CronJob` triggers a backup by creating a `BackupSession` custom resource. + +10. KubeStash operator watches for `BackupSession` custom resources. + +11. When it finds a `BackupSession` object, it creates a `Snapshot` custom resource for each `Repository` specified in the `BackupConfiguration`. + +12. Then it resolves the respective `Addon` and `Function` and creates the backup `Job`. + +13. The backup `Job` walks the VM's dependency graph, sanitizes the manifests, and uploads them as the `manifest` component. + +14. Then it enumerates the VM's disk PVCs and registers a `volume-` component for each one, before any data is moved. + +15. For a running VM, the `Job` creates a `VirtualMachineSnapshot`, waits for it to become ready, and provisions a temporary PVC from each per-disk `VolumeSnapshot`. For a stopped VM, it uses the disk PVCs directly. + +16. The `Job` then creates one worker `Job` per disk. Each worker mounts a single disk and uploads it to the backend. + +17. After the backup process is completed, the `Job`(s) update the `status.components[*]` field of the `Snapshot` resource with backup information of the target components, and the temporary PVCs, the `VirtualMachineSnapshot` and the worker `Job`s are cleaned up. + +## How Restore Process Works + +The following diagram shows how KubeStash restores a backed up `VirtualMachine`. Open the image in a new tab to see the enlarged version. + +
+ KubeStash VirtualMachine Restore Flow +
Fig: Restore process of a VirtualMachine in KubeStash
+
+ +The restore process consists of the following steps: + +1. At first, the user creates a `RestoreSession` custom resource that specifies the `Repository` object that points to a `BackupStorage` that holds backend information, and the target `Snapshot`, which will be restored. It also specifies the `Addon` info with the task to use. + +2. KubeStash operator watches for `RestoreSession` custom resources. + +3. When it finds a `RestoreSession` custom resource, it resolves the respective `Addon` and `Function` and creates a restore `Job`. + +4. The restore `Job` creates the destination PVCs from the backed up manifests and creates one worker `Job` per disk to restore the disk data into them. + +5. Only after every disk has been restored does the `Job` apply the manifests, in dependency order: `ServiceAccount`s, `Secret`s, `ConfigMap`s, `ControllerRevision`s, `PersistentVolumeClaim`s, `DataVolume`s, and finally the `VirtualMachine`. If any disk fails to restore, the `VirtualMachine` is never applied. + +6. Because the restored PVCs carry the `cdi.kubevirt.io/storage.populatedFor` annotation, CDI adopts them instead of importing their contents again. + +7. KubeVirt reconciles the restored `VirtualMachine` and, when it is started, creates the `VirtualMachineInstance` and its `virt-launcher` Pod. + +8. Finally, when the restore process is completed, the `Job`(s) update the `status.components[*]` field of the `RestoreSession` with restore information of the target components. + +## Next Steps + +1. See a step by step guide to backup/restore a KubeVirt `VirtualMachine` [here](/docs/guides/kubevirt/virtualmachine/index.md). diff --git a/docs/guides/kubevirt/virtualmachine/examples/backupconfiguration.yaml b/docs/guides/kubevirt/virtualmachine/examples/backupconfiguration.yaml new file mode 100644 index 0000000..bf858f5 --- /dev/null +++ b/docs/guides/kubevirt/virtualmachine/examples/backupconfiguration.yaml @@ -0,0 +1,39 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: BackupConfiguration +metadata: + name: sample-vm-backup + namespace: demo +spec: + target: + apiGroup: kubevirt.io + kind: VirtualMachine + name: kubestash-demo-vm + namespace: demo + backends: + - name: gcs-backend + storageRef: + name: gcs-storage + namespace: demo + retentionPolicy: + name: demo-retention + namespace: demo + sessions: + - name: demo-session + scheduler: + schedule: "*/5 * * * *" + jobTemplate: + backoffLimit: 1 + repositories: + - name: gcs-vm-repo + backend: gcs-backend + directory: /vm + encryptionSecret: + name: encrypt-secret + namespace: demo + addon: + name: kubevirt + tasks: + - name: logical-backup + retryConfig: + maxRetry: 2 + delay: 1m diff --git a/docs/guides/kubevirt/virtualmachine/examples/backupstorage.yaml b/docs/guides/kubevirt/virtualmachine/examples/backupstorage.yaml new file mode 100644 index 0000000..ab94037 --- /dev/null +++ b/docs/guides/kubevirt/virtualmachine/examples/backupstorage.yaml @@ -0,0 +1,17 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: gcs-storage + namespace: demo +spec: + storage: + provider: gcs + gcs: + bucket: kubestash-qa + prefix: demo + secretName: gcs-secret + usagePolicy: + allowedNamespaces: + from: All + default: true + deletionPolicy: WipeOut diff --git a/docs/guides/kubevirt/virtualmachine/examples/restoresession.yaml b/docs/guides/kubevirt/virtualmachine/examples/restoresession.yaml new file mode 100644 index 0000000..cbe54d5 --- /dev/null +++ b/docs/guides/kubevirt/virtualmachine/examples/restoresession.yaml @@ -0,0 +1,16 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: RestoreSession +metadata: + name: sample-vm-restore + namespace: demo +spec: + dataSource: + repository: gcs-vm-repo + snapshot: latest + encryptionSecret: + name: encrypt-secret + namespace: demo + addon: + name: kubevirt + tasks: + - name: logical-restore diff --git a/docs/guides/kubevirt/virtualmachine/examples/retentionpolicy.yaml b/docs/guides/kubevirt/virtualmachine/examples/retentionpolicy.yaml new file mode 100644 index 0000000..9e2758c --- /dev/null +++ b/docs/guides/kubevirt/virtualmachine/examples/retentionpolicy.yaml @@ -0,0 +1,15 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: RetentionPolicy +metadata: + name: demo-retention + namespace: demo +spec: + default: true + failedSnapshots: + last: 2 + maxRetentionPeriod: 2mo + successfulSnapshots: + last: 5 + usagePolicy: + allowedNamespaces: + from: All diff --git a/docs/guides/kubevirt/virtualmachine/examples/sample-data.yaml b/docs/guides/kubevirt/virtualmachine/examples/sample-data.yaml new file mode 100644 index 0000000..fa856cd --- /dev/null +++ b/docs/guides/kubevirt/virtualmachine/examples/sample-data.yaml @@ -0,0 +1,32 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: kubestash-demo-data-writer + namespace: demo +spec: + backoffLimit: 1 + template: + spec: + restartPolicy: Never + containers: + - name: writer + image: busybox:1.36 + command: + - sh + - -c + - | + set -e + echo "kubestash-root-sample-data" > /root-disk/sample.txt + echo "kubestash-data-sample-data" > /data-disk/sample.txt + sha256sum /root-disk/sample.txt /data-disk/sample.txt + sync + volumeMounts: + - { name: root, mountPath: /root-disk } + - { name: data, mountPath: /data-disk } + volumes: + - name: root + persistentVolumeClaim: + claimName: kubestash-demo-vm-root + - name: data + persistentVolumeClaim: + claimName: kubestash-demo-vm-data diff --git a/docs/guides/kubevirt/virtualmachine/examples/verify-data.yaml b/docs/guides/kubevirt/virtualmachine/examples/verify-data.yaml new file mode 100644 index 0000000..b800233 --- /dev/null +++ b/docs/guides/kubevirt/virtualmachine/examples/verify-data.yaml @@ -0,0 +1,29 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: kubestash-demo-data-verify + namespace: demo +spec: + backoffLimit: 1 + template: + spec: + restartPolicy: Never + containers: + - name: verify + image: busybox:1.36 + command: + - sh + - -c + - | + cat /root-disk/sample.txt /data-disk/sample.txt + sha256sum /root-disk/sample.txt /data-disk/sample.txt + volumeMounts: + - { name: root, mountPath: /root-disk } + - { name: data, mountPath: /data-disk } + volumes: + - name: root + persistentVolumeClaim: + claimName: kubestash-demo-vm-root + - name: data + persistentVolumeClaim: + claimName: kubestash-demo-vm-data diff --git a/docs/guides/kubevirt/virtualmachine/examples/vm.yaml b/docs/guides/kubevirt/virtualmachine/examples/vm.yaml new file mode 100644 index 0000000..0488231 --- /dev/null +++ b/docs/guides/kubevirt/virtualmachine/examples/vm.yaml @@ -0,0 +1,131 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: kubestash-demo-sa + namespace: demo +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: kubestash-demo-config + namespace: demo +data: + app.conf: | + greeting=hello-from-configmap +--- +apiVersion: v1 +kind: Secret +metadata: + name: kubestash-demo-cloudinit + namespace: demo +type: Opaque +stringData: + userdata: | + #cloud-config + password: kubestash + chpasswd: { expire: False } +--- +apiVersion: v1 +kind: Secret +metadata: + name: kubestash-demo-sshkey + namespace: demo +type: Opaque +stringData: + key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABexamplekeyforkubestashdemo demo@kubestash" +--- +apiVersion: cdi.kubevirt.io/v1beta1 +kind: DataVolume +metadata: + name: kubestash-demo-vm-root + namespace: demo + annotations: + cdi.kubevirt.io/storage.bind.immediate.requested: "true" +spec: + source: + blank: {} + storage: + accessModes: ["ReadWriteOnce"] + volumeMode: Filesystem + resources: + requests: + storage: 1Gi +--- +apiVersion: cdi.kubevirt.io/v1beta1 +kind: DataVolume +metadata: + name: kubestash-demo-vm-data + namespace: demo + annotations: + cdi.kubevirt.io/storage.bind.immediate.requested: "true" +spec: + source: + blank: {} + storage: + accessModes: ["ReadWriteOnce"] + volumeMode: Filesystem + resources: + requests: + storage: 1Gi +--- +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: kubestash-demo-vm + namespace: demo +spec: + runStrategy: Halted + template: + metadata: + labels: + kubevirt.io/vm: kubestash-demo-vm + spec: + accessCredentials: + - sshPublicKey: + source: + secret: + secretName: kubestash-demo-sshkey + propagationMethod: + noCloud: {} + domain: + cpu: + cores: 1 + resources: + requests: + memory: 256Mi + devices: + disks: + - name: rootdisk + disk: + bus: virtio + - name: datadisk + disk: + bus: virtio + - name: cloudinitdisk + disk: + bus: virtio + - name: configdisk + serial: CONFIG + disk: + bus: virtio + - name: sadisk + serial: SA + disk: + bus: virtio + volumes: + - name: rootdisk + dataVolume: + name: kubestash-demo-vm-root + - name: datadisk + dataVolume: + name: kubestash-demo-vm-data + - name: cloudinitdisk + cloudInitNoCloud: + secretRef: + name: kubestash-demo-cloudinit + - name: configdisk + configMap: + name: kubestash-demo-config + - name: sadisk + serviceAccount: + serviceAccountName: kubestash-demo-sa diff --git a/docs/guides/kubevirt/virtualmachine/index.md b/docs/guides/kubevirt/virtualmachine/index.md new file mode 100644 index 0000000..87654d0 --- /dev/null +++ b/docs/guides/kubevirt/virtualmachine/index.md @@ -0,0 +1,586 @@ +--- +title: Backup and Restore a KubeVirt VirtualMachine | KubeStash +description: A step by step guide showing how to backup and restore a KubeVirt VirtualMachine. +menu: + docs_{{ .version }}: + identifier: kubevirt-virtualmachine + name: Backup & Restore a VirtualMachine + parent: kubevirt + weight: 20 +product_name: kubestash +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Backup and Restore a KubeVirt VirtualMachine + +This guide will show you how to use KubeStash to backup and restore a KubeVirt `VirtualMachine`, along with the data on its disks. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install `KubeStash` in your cluster following the steps [here](/docs/setup/README.md). + +- Install [KubeVirt](https://kubevirt.io/user-guide/cluster_workloads/installation/) and [CDI](https://github.com/kubevirt/containerized-data-importer) in your cluster. + +- You should be familiar with the following `KubeStash` concepts: + - [BackupConfiguration](/docs/concepts/crds/backupconfiguration/index.md) + - [BackupSession](/docs/concepts/crds/backupsession/index.md) + - [RestoreSession](/docs/concepts/crds/restoresession/index.md) + - [BackupStorage](/docs/concepts/crds/backupstorage/index.md) + - [Snapshot](/docs/concepts/crds/snapshot/index.md) + +- If you want to backup a **running** `VirtualMachine`, your cluster additionally needs a CSI driver with snapshot support, a default `VolumeSnapshotClass` for the disks' `StorageClass`, and the KubeVirt `Snapshot` feature gate enabled. A stopped `VirtualMachine` needs none of these. See [How Backup and Restore works?](/docs/guides/kubevirt/overview/index.md) for the details. + +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> **Note:** YAML files used in this tutorial are stored in [docs/guides/kubevirt/virtualmachine/examples](/docs/guides/kubevirt/virtualmachine/examples) directory of [kubestash/docs](https://github.com/kubestash/docs) repository. + +## Backup a VirtualMachine + +This section will show you how to use KubeStash to backup a `VirtualMachine`. Here, we are going to create a `VirtualMachine` with two disks and some sample data on them. Then, we are going to backup the VM and its disks using KubeStash. + +### Prepare VirtualMachine + +At first, we are going to create a `VirtualMachine` along with the objects it depends on. The sample VM has two `DataVolume`-backed disks, a cloud-init `Secret`, an SSH access-credential `Secret`, a `ConfigMap` and a `ServiceAccount`, so that we can see the whole dependency graph being captured. + +Below is a part of the YAML of the `VirtualMachine` that we are going to create, + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: kubestash-demo-vm + namespace: demo +spec: + runStrategy: Halted + template: + spec: + accessCredentials: + - sshPublicKey: + source: + secret: + secretName: kubestash-demo-sshkey + propagationMethod: + noCloud: {} + domain: + cpu: + cores: 1 + resources: + requests: + memory: 256Mi + devices: + disks: + - name: rootdisk + disk: + bus: virtio + - name: datadisk + disk: + bus: virtio + ... + volumes: + - name: rootdisk + dataVolume: + name: kubestash-demo-vm-root + - name: datadisk + dataVolume: + name: kubestash-demo-vm-data + - name: cloudinitdisk + cloudInitNoCloud: + secretRef: + name: kubestash-demo-cloudinit + - name: configdisk + configMap: + name: kubestash-demo-config + - name: sadisk + serviceAccount: + serviceAccountName: kubestash-demo-sa +``` + +Let's create the `VirtualMachine` and its dependencies we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubestash/docs/raw/{{< param "info.version" >}}/docs/guides/kubevirt/virtualmachine/examples/vm.yaml +serviceaccount/kubestash-demo-sa created +configmap/kubestash-demo-config created +secret/kubestash-demo-cloudinit created +secret/kubestash-demo-sshkey created +datavolume.cdi.kubevirt.io/kubestash-demo-vm-root created +datavolume.cdi.kubevirt.io/kubestash-demo-vm-data created +virtualmachine.kubevirt.io/kubestash-demo-vm created +``` + +Now, wait for the `DataVolume`s to go into the `Succeeded` phase, which means CDI has finished provisioning the disks. + +```bash +$ kubectl get datavolume -n demo +NAME PHASE PROGRESS RESTARTS AGE +kubestash-demo-vm-data Succeeded N/A 62s +kubestash-demo-vm-root Succeeded N/A 62s +``` + +**Generate Sample Data:** + +Now, let's write some sample data onto both disks so that we can verify the restore later. The VM is stopped, so its disk PVCs are free and a `Job` can mount them. + +```bash +$ kubectl apply -f https://github.com/kubestash/docs/raw/{{< param "info.version" >}}/docs/guides/kubevirt/virtualmachine/examples/sample-data.yaml +job.batch/kubestash-demo-data-writer created +``` + +Let's note down the checksums of the sample data, so that we can compare them after the restore, + +```bash +$ kubectl logs -n demo job/kubestash-demo-data-writer +bf07cb11e6dea27eb81b15480bbd093983cf68f8c619a2b32a3f07beab3d821e /root-disk/sample.txt +117aeee770d409f8503f87a594cd2a7dc10b0b00aa26dcf481b5fcf5b7e45309 /data-disk/sample.txt +``` + +Once the `Job` has completed, delete it so that it does not hold a reference to the disk PVCs, + +```bash +$ kubectl delete job -n demo kubestash-demo-data-writer +job.batch "kubestash-demo-data-writer" deleted +``` + +### Prepare Backend + +We are going to store our backed up data into a GCS bucket. We have to create a Secret with necessary credentials and a `BackupStorage` CR to use this backend. If you want to use a different backend, please read the respective backend configuration doc from [here](/docs/guides/backends/overview/index.md). + +**Create Secret:** + +Let's create a secret called `gcs-secret` with access credentials to our desired GCS bucket, + +```bash +$ echo -n '' > GOOGLE_PROJECT_ID +$ cat /path/to/downloaded-sa-key.json > GOOGLE_SERVICE_ACCOUNT_JSON_KEY +$ kubectl create secret generic -n demo gcs-secret \ + --from-file=./GOOGLE_PROJECT_ID \ + --from-file=./GOOGLE_SERVICE_ACCOUNT_JSON_KEY +secret/gcs-secret created +``` + +**Create BackupStorage:** + +Now, create a `BackupStorage` using this secret. Below is the YAML of `BackupStorage` CR we are going to create, + +```yaml +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: gcs-storage + namespace: demo +spec: + storage: + provider: gcs + gcs: + bucket: kubestash-qa + prefix: demo + secretName: gcs-secret + usagePolicy: + allowedNamespaces: + from: All + default: true + deletionPolicy: WipeOut +``` + +Let's create the BackupStorage we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubestash/docs/raw/{{< param "info.version" >}}/docs/guides/kubevirt/virtualmachine/examples/backupstorage.yaml +backupstorage.storage.kubestash.com/gcs-storage created +``` + +**Create RetentionPolicy:** + +Now, let's create a `RetentionPolicy` to specify how the old Snapshots should be cleaned up. + +Below is the YAML of the `RetentionPolicy` object that we are going to create, + +```yaml +apiVersion: storage.kubestash.com/v1alpha1 +kind: RetentionPolicy +metadata: + name: demo-retention + namespace: demo +spec: + default: true + failedSnapshots: + last: 2 + maxRetentionPeriod: 2mo + successfulSnapshots: + last: 5 + usagePolicy: + allowedNamespaces: + from: All +``` + +Let’s create the above `RetentionPolicy`, + +```bash +$ kubectl apply -f https://github.com/kubestash/docs/raw/{{< param "info.version" >}}/docs/guides/kubevirt/virtualmachine/examples/retentionpolicy.yaml +retentionpolicy.storage.kubestash.com/demo-retention created +``` + +### Backup + +We have to create a `BackupConfiguration` CR targeting the `kubestash-demo-vm` `VirtualMachine` that we have created earlier. Then, KubeStash will create a `CronJob` for each session to take periodic backup of the VM. + +At first, we need to create a secret with a Restic password for backup data encryption. + +**Create Secret:** + +Let's create a secret called `encrypt-secret` with the Restic password, + +```bash +$ echo -n 'changeit' > RESTIC_PASSWORD +$ kubectl create secret generic -n demo encrypt-secret \ + --from-file=./RESTIC_PASSWORD \ +secret "encrypt-secret" created +``` + +**Create BackupConfiguration:** + +Below is the YAML of the `BackupConfiguration` CR that we are going to create, + +```yaml +apiVersion: core.kubestash.com/v1alpha1 +kind: BackupConfiguration +metadata: + name: sample-vm-backup + namespace: demo +spec: + target: + apiGroup: kubevirt.io + kind: VirtualMachine + name: kubestash-demo-vm + namespace: demo + backends: + - name: gcs-backend + storageRef: + name: gcs-storage + namespace: demo + retentionPolicy: + name: demo-retention + namespace: demo + sessions: + - name: demo-session + scheduler: + schedule: "*/5 * * * *" + jobTemplate: + backoffLimit: 1 + repositories: + - name: gcs-vm-repo + backend: gcs-backend + directory: /vm + encryptionSecret: + name: encrypt-secret + namespace: demo + addon: + name: kubevirt + tasks: + - name: logical-backup + retryConfig: + maxRetry: 2 + delay: 1m +``` + +Here, + +- `spec.target` refers to the `VirtualMachine` we are backing up. A `VirtualMachine` is a first-class KubeStash target, identified by the `kubevirt.io` API group. +- `spec.sessions[*].addon.tasks[*].name` is set to `logical-backup`, which backs up the VM's manifests and all of its disks in one task. + +> The addon also exposes `manifest-backup` and `volume-backup` as separate tasks if you would rather run the two halves as separate task containers. The resulting `Snapshot` is the same either way. + +Let's create the `BackupConfiguration` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubestash/docs/raw/{{< param "info.version" >}}/docs/guides/kubevirt/virtualmachine/examples/backupconfiguration.yaml +backupconfiguration.core.kubestash.com/sample-vm-backup created +``` + +**Verify Backup Setup Successful** + +If everything goes well, the phase of the `BackupConfiguration` should be `Ready`. The `Ready` phase indicates that the backup setup is successful. Let's verify the `Phase` of the BackupConfiguration, + +```bash +$ kubectl get backupconfiguration -n demo +NAME PHASE PAUSED AGE +sample-vm-backup Ready 21s +``` + +Additionally, we can verify that the `Repository` specified in the `BackupConfiguration` has been created using the following command, + +```bash +$ kubectl get repo -n demo +NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE +gcs-vm-repo 0 0 B Ready 51s +``` + +**Verify CronJob:** + +It will also create a `CronJob` with the schedule specified in `spec.sessions[*].scheduler.schedule` field of `BackupConfiguration` CR. + +Verify that the `CronJob` has been created using the following command, + +```bash +$ kubectl get cronjob -n demo +NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE +trigger-sample-vm-backup-demo-session */5 * * * * False 0 79s +``` + +**Wait for BackupSession:** + +Wait for the next schedule for backup. Run the following command to watch `BackupSession` CR, + +```bash +$ kubectl get backupsession -n demo -w +NAME INVOKER-TYPE INVOKER-NAME PHASE DURATION AGE +sample-vm-backup-demo-session-1787216260 BackupConfiguration sample-vm-backup Succeeded 87s 3m +``` + +We can see from the above output that the backup session has succeeded. Now, we are going to verify whether the backed up data has been stored in the backend. + +**Verify Backup:** + +Once a backup is complete, KubeStash will update the respective `Repository` CR to reflect the backup. Check that the repository `gcs-vm-repo` has been updated by the following command, + +```bash +$ kubectl get repository -n demo gcs-vm-repo +NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE +gcs-vm-repo true 1 47.374 KiB Ready 2m11s 6m +``` + +At this moment we have one `Snapshot`. Run the following command to check the respective `Snapshot` which represents the state of a backup run for an application. + +```bash +$ kubectl get snapshots -n demo -l=kubestash.com/repo-name=gcs-vm-repo +NAME REPOSITORY SESSION SNAPSHOT-TIME DELETION-POLICY PHASE AGE +gcs-vm-repo-sample-vm-backup-demo-session-1787216260 gcs-vm-repo demo-session 2026-08-20T14:19:23Z Delete Succeeded 2m35s +``` + +> Note: KubeStash creates a `Snapshot` with the following labels: +> - `kubestash.com/app-ref-kind: ` +> - `kubestash.com/app-ref-name: ` +> - `kubestash.com/app-ref-namespace: ` +> - `kubestash.com/repo-name: ` +> +> These labels can be used to watch only the `Snapshot`s related to our desired `VirtualMachine` or `Repository`. + +If we check the YAML of the `Snapshot`, we can find the information about the backed up components of the `VirtualMachine`. + +```bash +$ kubectl get snapshots -n demo gcs-vm-repo-sample-vm-backup-demo-session-1787216260 -oyaml +``` + +```yaml +apiVersion: storage.kubestash.com/v1alpha1 +kind: Snapshot +metadata: + labels: + kubestash.com/app-ref-kind: VirtualMachine + kubestash.com/app-ref-name: kubestash-demo-vm + kubestash.com/app-ref-namespace: demo + kubestash.com/repo-name: gcs-vm-repo + name: gcs-vm-repo-sample-vm-backup-demo-session-1787216260 + namespace: demo +spec: + ... +status: + components: + manifest: + driver: Restic + integrity: true + path: repository/v1/demo-session/manifest + phase: Succeeded + ... + volume-kubestash-demo-vm-data: + driver: Restic + integrity: true + path: repository/v1/demo-session/volume-kubestash-demo-vm-data + phase: Succeeded + ... + volume-kubestash-demo-vm-root: + driver: Restic + integrity: true + path: repository/v1/demo-session/volume-kubestash-demo-vm-root + phase: Succeeded + ... + integrity: true + phase: Succeeded + totalComponents: 3 + ... +``` + +Here, the `manifest` component holds the manifests of the VM's dependency graph, and each `volume-` component holds the data of one disk. The `Snapshot` reaches the `Succeeded` phase only when every component has succeeded, so a `Succeeded` `Snapshot` can never be missing a disk. + +Now, if we navigate to the GCS bucket, we will see the backed up data stored in the `demo/vm/repository/v1/demo-session` directory. KubeStash also keeps the backup for `Snapshot` YAMLs, which can be found in the `demo/vm/snapshots` directory. + +> Note: KubeStash stores all backed up data encrypted in the backup directory, meaning it remains unreadable until decrypted. + +## Restore + +In this section, we are going to show you how to restore the `VirtualMachine` from the backup we have just taken. + +**Simulate Disaster:** + +Now, let's simulate a disaster scenario. Here, we are going to delete the `VirtualMachine` along with everything it depends on. + +```bash +$ kubectl delete -f https://github.com/kubestash/docs/raw/{{< param "info.version" >}}/docs/guides/kubevirt/virtualmachine/examples/vm.yaml +serviceaccount "kubestash-demo-sa" deleted +configmap "kubestash-demo-config" deleted +secret "kubestash-demo-cloudinit" deleted +secret "kubestash-demo-sshkey" deleted +datavolume.cdi.kubevirt.io "kubestash-demo-vm-root" deleted +datavolume.cdi.kubevirt.io "kubestash-demo-vm-data" deleted +virtualmachine.kubevirt.io "kubestash-demo-vm" deleted +``` + +Verify that the `VirtualMachine` and its disks are gone, + +```bash +$ kubectl get vm,datavolume,pvc -n demo +No resources found in demo namespace. +``` + +**Create RestoreSession:** + +To restore the `VirtualMachine`, you have to create a `RestoreSession` object pointing to the `Repository` and `Snapshot` you want to restore from. + +Here, is the YAML of the `RestoreSession` object that we are going to use for restoring our `kubestash-demo-vm` `VirtualMachine`. + +```yaml +apiVersion: core.kubestash.com/v1alpha1 +kind: RestoreSession +metadata: + name: sample-vm-restore + namespace: demo +spec: + dataSource: + repository: gcs-vm-repo + snapshot: latest + encryptionSecret: + name: encrypt-secret + namespace: demo + addon: + name: kubevirt + tasks: + - name: logical-restore +``` + +Here, + +- `spec.dataSource.snapshot` specifies to restore from latest `Snapshot`. +- `spec.addon.tasks[*].name` is set to `logical-restore`, which restores the disks first and then applies the manifests, in a single task. + +> **Note:** A `RestoreSession` for a `VirtualMachine` must **not** set `spec.target`. The `VirtualMachine` does not exist yet at restore time, so a target pointing at a missing object would never be found. KubeStash learns the VM's identity from the backed up manifests instead. + +Let's create the `RestoreSession` object we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubestash/docs/raw/{{< param "info.version" >}}/docs/guides/kubevirt/virtualmachine/examples/restoresession.yaml +restoresession.core.kubestash.com/sample-vm-restore created +``` + +Once, you have created the `RestoreSession` object, KubeStash will create restore Job(s). Run the following command to watch the phase of the `RestoreSession` object, + +```bash +$ watch kubectl get restoresession -n demo +NAME REPOSITORY FAILURE-POLICY PHASE DURATION AGE +sample-vm-restore gcs-vm-repo Succeeded 51s 79s +``` + +The `Succeeded` phase means that the restore process has been completed successfully. + +**Verify Restored VirtualMachine:** + +Let's verify that the `VirtualMachine` and every object it depends on have come back, + +```bash +$ kubectl get vm,datavolume,pvc,secret,configmap,serviceaccount -n demo | grep kubestash-demo +virtualmachine.kubevirt.io/kubestash-demo-vm 25s Stopped False +datavolume.cdi.kubevirt.io/kubestash-demo-vm-data Succeeded N/A 25s +datavolume.cdi.kubevirt.io/kubestash-demo-vm-root Succeeded N/A 25s +persistentvolumeclaim/kubestash-demo-vm-data Bound pvc-39036687-7270-4521-9207-ba5b113bbecc 1Gi RWO +persistentvolumeclaim/kubestash-demo-vm-root Bound pvc-a1fb889b-6d30-4846-a0b0-7685070e3019 1Gi RWO +secret/kubestash-demo-cloudinit Opaque 1 25s +secret/kubestash-demo-sshkey Opaque 1 25s +configmap/kubestash-demo-config 1 25s +serviceaccount/kubestash-demo-sa 25s +``` + +The `DataVolume`s are already in the `Succeeded` phase because KubeStash restored the disk data before applying them, and CDI adopted the populated PVCs instead of importing their contents again. + +**Verify Restored Data:** + +Now, let's verify that the data on the disks was restored, by comparing the checksums against the ones we noted before the backup, + +```bash +$ kubectl apply -f https://github.com/kubestash/docs/raw/{{< param "info.version" >}}/docs/guides/kubevirt/virtualmachine/examples/verify-data.yaml +job.batch/kubestash-demo-data-verify created + +$ kubectl logs -n demo job/kubestash-demo-data-verify +kubestash-root-sample-data +kubestash-data-sample-data +bf07cb11e6dea27eb81b15480bbd093983cf68f8c619a2b32a3f07beab3d821e /root-disk/sample.txt +117aeee770d409f8503f87a594cd2a7dc10b0b00aa26dcf481b5fcf5b7e45309 /data-disk/sample.txt +``` + +Hence, we can see from the above output that the checksums match the ones we noted before the backup, so the disk data has been restored successfully. + +The restored `VirtualMachine` is stopped. Start it whenever you are ready, + +```bash +$ kubectl patch vm -n demo kubestash-demo-vm --type merge -p '{"spec":{"runStrategy":"Always"}}' +virtualmachine.kubevirt.io/kubestash-demo-vm patched +``` + +> A restored `VirtualMachine` always cold-boots. KubeStash does not capture the guest's RAM or the `VirtualMachineInstance`, because KubeVirt recreates those from the `VirtualMachine` itself. + +## Backup a Running VirtualMachine + +You do not have to stop a `VirtualMachine` to back it up, and you do not have to configure anything differently — the same `BackupConfiguration` works. When the VM is running, KubeStash creates a `VirtualMachineSnapshot` and lets KubeVirt freeze the guest filesystem, take one CSI `VolumeSnapshot` per disk and unfreeze, then copies those snapshots to the backend and cleans them up. + +You can watch that happen during a backup of a running VM, + +```bash +$ kubectl get virtualmachinesnapshot,volumesnapshot -n demo +``` + +Install `qemu-guest-agent` in the guest to get a filesystem-consistent backup. Without it, KubeVirt cannot freeze the filesystem and the backup is crash-consistent — it still succeeds, and records a warning. + +## Restore Selected Components + +Since each disk is a separate component, you can restore a subset instead of the whole VM. Pass the component names to the `components` parameter of the restore task, + +```yaml + addon: + name: kubevirt + tasks: + - name: logical-restore + params: + components: volume-kubestash-demo-vm-root +``` + +Restoring only the `manifest` component recreates the `VirtualMachine` and its `DataVolume`s without their data, letting CDI import the disks from their original sources again. + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete -n demo backupconfiguration sample-vm-backup +kubectl delete -n demo restoresession sample-vm-restore +kubectl delete -n demo vm kubestash-demo-vm +kubectl delete -n demo datavolume --all +kubectl delete -n demo backupstorage gcs-storage +kubectl delete -n demo retentionpolicy demo-retention +kubectl delete -n demo secret gcs-secret encrypt-secret +```