Backup related upgrade failure from 4.20.1/4.20.2 to 4.21.0/4.22.0

[shwstppr]

problem

When upgrading from either 4.20.1 or 4.20.2 to 4.21.0 or 4.22.0, upgrade could fail when backup framework is in use.
An error like the following is seen,

2026-03-09 01:56:55,403 DEBUG [c.c.u.d.ScriptRunner] (main:[]) (logid:) -- Add diskOfferingId, deviceId, minIops and maxIops to backed_volumes in backups table
2026-03-09 01:56:55,403 DEBUG [c.c.u.d.ScriptRunner] (main:[]) (logid:) UPDATE `cloud`.`backups` b INNER JOIN `cloud`.`vm_instance` vm ON b.vm_id = vm.id SET b.backed_volumes = (     SELECT CONCAT("[",         GROUP_CONCAT(             CONCAT(                 "{\"uuid\":\"", v.uuid, "\",",                 "\"type\":\"", v.volume_type, "\",",                 "\"size\":", v.`size`, ",",                 "\"path\":\"", IFNULL(v.path, 'null'), "\",",                 "\"deviceId\":", IFNULL(v.device_id, 'null'), ",",                 "\"diskOfferingId\":\"", doff.uuid, "\",",                 "\"minIops\":", IFNULL(v.min_iops, 'null'), ",",                 "\"maxIops\":", IFNULL(v.max_iops, 'null'),                 "}"             )             SEPARATOR ","         ),     "]")     FROM `cloud`.`volumes` v     LEFT JOIN `cloud`.`disk_offering` doff ON v.disk_offering_id = doff.id     WHERE v.instance_id = vm.id )
2026-03-09 01:56:55,458 ERROR [c.c.u.d.ScriptRunner] (main:[]) (logid:) Error executing: UPDATE `cloud`.`backups` b INNER JOIN `cloud`.`vm_instance` vm ON b.vm_id = vm.id SET b.backed_volumes = (     SELECT CONCAT("[",         GROUP_CONCAT(             CONCAT(                 "{\"uuid\":\"", v.uuid, "\",",                 "\"type\":\"", v.volume_type, "\",",                 "\"size\":", v.`size`, ",",                 "\"path\":\"", IFNULL(v.path, 'null'), "\",",                 "\"deviceId\":", IFNULL(v.device_id, 'null'), ",",                 "\"diskOfferingId\":\"", doff.uuid, "\",",                 "\"minIops\":", IFNULL(v.min_iops, 'null'), ",",                 "\"maxIops\":", IFNULL(v.max_iops, 'null'),                 "}"             )             SEPARATOR ","         ),     "]")     FROM `cloud`.`volumes` v     LEFT JOIN `cloud`.`disk_offering` doff ON v.disk_offering_id = doff.id     WHERE v.instance_id = vm.id )
2026-03-09 01:56:55,475 ERROR [c.c.u.d.ScriptRunner] (main:[]) (logid:) java.sql.SQLException: Row 19 was cut by GROUP_CONCAT()
2026-03-09 01:56:55,495 ERROR [c.c.u.DatabaseUpgradeChecker] (main:[]) (logid:) Unable to execute upgrade script java.sql.SQLException: Row 19 was cut by GROUP_CONCAT()
        at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:186)
        at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:88)

This is due to the following DB changes in the 4.20.1to4.21.0 upgrade path,
https://github.com/apache/cloudstack/blob/main/engine/schema/src/main/resources/META-INF/db/schema-42010to42100.sql#L686-L709

The error is due to the length limit for group_concat_max_len in MySQL and it could be seen if there are backups for VMs having multiple volumes (possible 3 or more).

Workaround is setting value for group_concat_max_len:
SET GLOBAL group_concat_max_len = 1000000;

versions

Upgrade 4.20.x to 4.21/4.22

[rajujith]

@blueorangutan package

[DaanHoogland]

@blueorangutan package

did this on the linked PR @rajujith ;)

[daviftorres]

Hey Team,

Here is our Dev environment:

And this is our Prod environment:

Is it safe so say that our Prod went out of tree for upgrade paths?

How could we upgrade to 4.22.0.0? Is there any way we can craft an upgrade path to get out of the 4.20.2.0, back to the rolling path of upgrades?

Regards,

[rajujith]

@daviftorres I would suggest upgrading to 4.22.1 once it is available, expecting to release in 1-2 weeks.

[sureshanaparti]

Fixed in #12777