linstor: grow cloned volumes inside the clone to fix resize race with BalanceAfterClone - #13998
Open
rp- wants to merge 3 commits into
Open
linstor: grow cloned volumes inside the clone to fix resize race with BalanceAfterClone#13998rp- wants to merge 3 commits into
rp- wants to merge 3 commits into
Conversation
CloudStack reports the VM Running as soon as the domain exists, so the bare reboot() let the next test attach and detach a volume against a guest that had not enumerated its disks yet. Use the existing _reboot_vm helper, which waits for the guest to come back.
Generated from LINSTOR REST API 1.29.1. resourceSnapshotDelete gained an optional delete_empty_resource_definition flag; pass null to keep the previous behaviour.
…erwards Deploying a VM with a root disk larger than its template failed with Cannot resize volume, because we have a non-UpToDate DRBD device. whenever the resource group has Clone/BalanceAfterClone=true. The clone reports COMPLETE as soon as every replica can access UpToDate data, which is while the additional balance replica is still doing its initial sync; the resize that followed CloneWaiter.waitFor() was rejected by LINSTOR's all-replicas-UpToDate precheck. Waiting for the sync client-side would block the deploy for the whole sync of the template size, so the fix is in LINSTOR: REST API 1.29.1 (LINSTOR 1.35.0) accepts volume_sizes on the clone request and grows the volume inside the clone flux before the balance placement. Pass the requested size that way when the controller supports it, detected once per controller URL via its REST API version, and skip the post-clone resize. Older controllers keep the clone-then-resize sequence unchanged.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 4.22 #13998 +/- ##
=========================================
Coverage 17.79% 17.80%
- Complexity 15995 16011 +16
=========================================
Files 5928 5928
Lines 534306 534342 +36
Branches 65383 65391 +8
=========================================
+ Hits 95069 95117 +48
+ Misses 428467 428453 -14
- Partials 10770 10772 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes deploying a VM on LINSTOR primary storage with a root disk larger than its template
when the LINSTOR resource group has
Clone/BalanceAfterClone=true.The plugin created such a volume as: clone the template resource-definition, wait for the clone
to report COMPLETE, then
volumeDefinitionModifyto the requested size. LINSTOR reports a cloneCOMPLETE as soon as every replica can access UpToDate data - with BalanceAfterClone that is
while the additional balance replica is still syncing - and the resize is rejected because it
requires all diskful replicas to be UpToDate:
The deploy then fails with a bare
errorcode 530 Unable to orchestrate the start of VM instanceand the instance is left in Error state. Waiting client-side is not an option, it would block the
deploy for the whole initial sync of the template size.
LINSTOR 1.35.0 (REST API 1.29.1) accepts
volume_sizeson the clone request and grows thecloned volume inside the clone, before the balance placement. The plugin now checks the
controller's REST API version, passes the size in the clone request on >= 1.29.1 and skips the
post-clone resize; older controllers keep the previous clone-then-resize sequence unchanged.
java-linstor is updated to 0.8.1 for the new request field;
resourceSnapshotDeletegained anoptional
delete_empty_resource_definitionparameter which is passed asnull.Also includes a small marvin fix:
test_07_detach_volume_reboot_vmused the barereboot()instead of the
_reboot_vmhelper that waits for the guest, which madetest_08detach a volumefrom a guest that had not enumerated it yet when a fast-booting template is used.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Unit tests for the version comparison, the version probe and the clone-size decision
(
LinstorUtilTest,LinstorPrimaryDataStoreDriverImplTest).3-node KVM cluster, LINSTOR primary storage (LVM-thin, place-count 2):
test_linstor_volumes.pyandtest_linstor_encrypted_snapshots.pypass (18/18), the log shows the unchanged clone-then-resize sequence.
Clone/BalanceAfterClone=trueon the resource group:deploying VMs with
rootdisksizelarger than the template (512 MiB template, 12/16/20 GiBroot disks, several in parallel) succeeds every time. No resize call in the log, the
volume-definition has the requested size and the balance replica is still SyncTarget while the
VM is already running - the state in which the previous code failed.
Before the fix, on the same cluster with
Clone/BalanceAfterClone=true, 9/9 such deploys failedwith the error above.
How did you try to break this feature and the system with this change?
volume_sizesis never sent.then fails with the controller's error as before.
and CloudStack enforces root size >= template size anyway.