Skip to content

kvm: do not fail a NAS backup on non-numeric output from nasbackup.sh - #9

Open
calvix wants to merge 1 commit into
mainfrom
fix/nasbackup-size-parse-tolerant
Open

kvm: do not fail a NAS backup on non-numeric output from nasbackup.sh#9
calvix wants to merge 1 commit into
mainfrom
fix/nasbackup-size-parse-tolerant

Conversation

@calvix

@calvix calvix commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Description

LibvirtTakeBackupCommandWrapper.parseBackupSize() parsed every line of nasbackup.sh's stdout as a byte count. The script is not the only writer on that stream — mount helpers and storage clients emit warnings of their own — and a single such line threw out of the wrapper:

java.lang.NumberFormatException: For input string: "2026-08-25T17:38:38.403+0000"

That line came from the Ceph client:

2026-08-25T17:38:38.403+0000 7f... -1 auth: unable to find a keyring on /etc/ceph/ceph.keyring

The cost is out of proportion to the cause. The exception escapes after the backup has already been written, so the data sits on the repository, but the record stays in BackingUp — there is no timeout and no transition to Failed — and it blocks every later restore of that instance until the row is fixed by hand.

This ignores lines that do not begin with a byte count, logging them at debug, instead of parsing them.

Why it went unnoticed

Only the multi-volume branch (diskPaths non-empty) was reachable in practice, and that is the branch taken for a stopped instance. Running instances back up through the single-volume branch and were unaffected, so the bug only surfaces on the stopped-instance path.

Reproduction

On a KVM host with Ceph RBD primary storage and a NAS backup repository, where the agent's environment has no readable /etc/ceph/ceph.keyring: back up a stopped instance. The backup lands on the repository; the record never leaves BackingUp.

Fix

parseSizeLine() returns the leading byte count of a line, or null when the line is not a size line:

  • multi-volume: non-numeric lines are skipped, remaining sizes still summed;
  • single-volume: scans back to the last numeric line rather than assuming the final line is one;
  • no numeric line at all yields 0 rather than throwing.

Trailing tokens after the byte count are still honoured, and blank lines are ignored.

Relationship to other PRs

Neither apache#12843 nor apache#12898 touches parseBackupSize. apache#13877 refactors this class but keeps the unguarded Long.parseLong(line.split(" ")[0].trim()), so it does not fix this. I found no open PR or issue covering it.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Adds LibvirtTakeBackupCommandWrapperTest (7 cases). Verified both ways in a 4.23 reactor build:

  • with the fix: Tests run: 7, Failures: 0, Errors: 0 — BUILD SUCCESS
  • with the fix reverted: Tests run: 7, Failures: 0, Errors: 4, each Caused by: java.lang.NumberFormatException, including the literal production string "2026-08-25T17:38:38.403+0000"

The three cases covering existing behaviour (multi-volume sum, single-volume last line, trailing tokens) pass both with and without the fix, so current semantics are unchanged.

Also confirmed in practice on a KVM/Ceph dev host: with the keyring absent, a stopped-instance backup wedged in BackingUp; with this change on the agent, the same backup completed at 2.55 GB.

parseBackupSize() parsed every line of the backup script's stdout as a byte
count. The script is not the only writer on that stream: mount helpers and
storage clients emit warnings of their own, and one such line was enough to
throw NumberFormatException out of the wrapper, e.g.

  java.lang.NumberFormatException: For input string: "2026-08-25T17:38:38.403+0000"

emitted by a Ceph client as "unable to find a keyring on /etc/ceph/ceph.keyring".

The cost of that is out of proportion to the cause. The exception escapes after
the backup has already been written, so the data is on the repository but the
record stays in BackingUp - there is no timeout and no transition to Failed -
and it blocks every later restore of that instance until the row is corrected
by hand.

Ignore lines that do not begin with a byte count instead, logging them at debug.
Only the multi-volume branch was reachable in practice, which is the branch
taken for a stopped instance, so running instances backed up normally and the
failure went unnoticed.

Adds LibvirtTakeBackupCommandWrapperTest; four of its seven cases fail without
this change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant