Skip to content

linstor: grow cloned volumes inside the clone to fix resize race with BalanceAfterClone - #13998

Open
rp- wants to merge 3 commits into
apache:4.22from
LINBIT:linstor-4.22-clone-resize-race
Open

linstor: grow cloned volumes inside the clone to fix resize race with BalanceAfterClone#13998
rp- wants to merge 3 commits into
apache:4.22from
LINBIT:linstor-4.22-clone-resize-race

Conversation

@rp-

@rp- rp- commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Description

This PR fixes deploying a VM on LINSTOR primary storage with a root disk larger than its template
when the LINSTOR resource group has Clone/BalanceAfterClone=true.

The plugin created such a volume as: clone the template resource-definition, wait for the clone
to report COMPLETE, then volumeDefinitionModify to the requested size. LINSTOR reports a clone
COMPLETE as soon as every replica can access UpToDate data - with BalanceAfterClone that is
while the additional balance replica is still syncing - and the resize is rejected because it
requires all diskful replicas to be UpToDate:

Linstor: ApiEx - [{"ret_code":-4611686018390686770,
  "message":"Cannot resize volume, because we have a non-UpToDate DRBD device.", ...}]

The deploy then fails with a bare errorcode 530 Unable to orchestrate the start of VM instance
and the instance is left in Error state. Waiting client-side is not an option, it would block the
deploy for the whole initial sync of the template size.

LINSTOR 1.35.0 (REST API 1.29.1) accepts volume_sizes on the clone request and grows the
cloned volume inside the clone, before the balance placement. The plugin now checks the
controller's REST API version, passes the size in the clone request on >= 1.29.1 and skips the
post-clone resize; older controllers keep the previous clone-then-resize sequence unchanged.

java-linstor is updated to 0.8.1 for the new request field; resourceSnapshotDelete gained an
optional delete_empty_resource_definition parameter which is passed as null.

Also includes a small marvin fix: test_07_detach_volume_reboot_vm used the bare reboot()
instead of the _reboot_vm helper that waits for the guest, which made test_08 detach a volume
from a guest that had not enumerated it yet when a fast-booting template is used.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Unit tests for the version comparison, the version probe and the clone-size decision
(LinstorUtilTest, LinstorPrimaryDataStoreDriverImplTest).

3-node KVM cluster, LINSTOR primary storage (LVM-thin, place-count 2):

  • LINSTOR 1.34.2 (REST API 1.29.0): test_linstor_volumes.py and test_linstor_encrypted_snapshots.py
    pass (18/18), the log shows the unchanged clone-then-resize sequence.
  • LINSTOR 1.35.0 (REST API 1.29.1) with Clone/BalanceAfterClone=true on the resource group:
    deploying VMs with rootdisksize larger than the template (512 MiB template, 12/16/20 GiB
    root disks, several in parallel) succeeds every time. No resize call in the log, the
    volume-definition has the requested size and the balance replica is still SyncTarget while the
    VM is already running - the state in which the previous code failed.

Before the fix, on the same cluster with Clone/BalanceAfterClone=true, 9/9 such deploys failed
with the error above.

How did you try to break this feature and the system with this change?

  • Old controller (1.34.2): fallback path is the previous sequence, volume_sizes is never sent.
  • Controller unreachable during the version probe: falls back to clone-then-resize, the clone
    then fails with the controller's error as before.
  • Several deploys in parallel while balance replicas were syncing.
  • Size equal to the template size (no-op in LINSTOR), smaller sizes are rejected at clone start
    and CloudStack enforces root size >= template size anyway.
  • Version strings with two/three components, empty and garbage input.

rp- added 3 commits August 27, 2026 09:15
CloudStack reports the VM Running as soon as the domain exists, so the bare
reboot() let the next test attach and detach a volume against a guest that
had not enumerated its disks yet. Use the existing _reboot_vm helper, which
waits for the guest to come back.
Generated from LINSTOR REST API 1.29.1. resourceSnapshotDelete gained an
optional delete_empty_resource_definition flag; pass null to keep the
previous behaviour.
…erwards

Deploying a VM with a root disk larger than its template failed with

  Cannot resize volume, because we have a non-UpToDate DRBD device.

whenever the resource group has Clone/BalanceAfterClone=true. The clone
reports COMPLETE as soon as every replica can access UpToDate data, which is
while the additional balance replica is still doing its initial sync; the
resize that followed CloneWaiter.waitFor() was rejected by LINSTOR's
all-replicas-UpToDate precheck. Waiting for the sync client-side would block
the deploy for the whole sync of the template size, so the fix is in LINSTOR:
REST API 1.29.1 (LINSTOR 1.35.0) accepts volume_sizes on the clone request and
grows the volume inside the clone flux before the balance placement.

Pass the requested size that way when the controller supports it, detected
once per controller URL via its REST API version, and skip the post-clone
resize. Older controllers keep the clone-then-resize sequence unchanged.
@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.17949% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.80%. Comparing base (7ea1dca) to head (d301fe2).

Files with missing lines Patch % Lines
...tore/driver/LinstorPrimaryDataStoreDriverImpl.java 72.72% 3 Missing ⚠️
...cloudstack/storage/datastore/util/LinstorUtil.java 96.29% 0 Missing and 1 partial ⚠️
...ck/storage/snapshot/LinstorVMSnapshotStrategy.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               4.22   #13998   +/-   ##
=========================================
  Coverage     17.79%   17.80%           
- Complexity    15995    16011   +16     
=========================================
  Files          5928     5928           
  Lines        534306   534342   +36     
  Branches      65383    65391    +8     
=========================================
+ Hits          95069    95117   +48     
+ Misses       428467   428453   -14     
- Partials      10770    10772    +2     
Flag Coverage Δ
uitests 3.69% <ø> (ø)
unittests 18.88% <87.17%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant