Skip to content

[blk-threaded - 2/5] block: add worker thread lifecycle - #6128

Merged
PierreBertholom merged 8 commits into
feature/virtio-blk-threadedfrom
pbertho/blk-threaded-04
Sep 30, 2026
Merged

[blk-threaded - 2/5] block: add worker thread lifecycle#6128
PierreBertholom merged 8 commits into
feature/virtio-blk-threadedfrom
pbertho/blk-threaded-04

Conversation

@PierreBertholom

@PierreBertholom PierreBertholom commented Aug 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

[2/5] part of the PR stack starting with #6123

Add the complete lifecycle for an optional dedicated block worker thread.

Changes

  • Add WorkerHandle and ThreadedWorker.
  • Use a channel and an eventfd to control each worker.
  • Transfer data-path resources to the worker during activation.
  • Return resources during reset and park the worker for reuse.
  • Join active worker threads when the device is dropped.
  • Forward drive updates to the worker.
  • Wake the worker after rate-limiter events.
  • Pause the worker and collect queue state during snapshot save.
  • Mark queue memory dirty and resume the worker after snapshot save.

A later pull request exposes threaded mode through the public API.

@codecov

codecov Bot commented Aug 18, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 59.24370% with 194 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.43%. Comparing base (16e564e) to head (661c427).

Files with missing lines Patch % Lines
src/vmm/src/devices/virtio/block/virtio/worker.rs 55.36% 129 Missing ⚠️
src/vmm/src/devices/virtio/block/virtio/device.rs 65.97% 49 Missing ⚠️
src/vmm/src/devices/virtio/block/virtio/persist.rs 41.66% 7 Missing ⚠️
src/vmm/src/devices/virtio/block/device.rs 0.00% 5 Missing ⚠️
...m/src/devices/virtio/block/virtio/event_handler.rs 50.00% 3 Missing ⚠️
src/vmm/src/io_uring/mod.rs 87.50% 1 Missing ⚠️
Additional details and impacted files 
@@                       Coverage Diff                       @@
##           feature/virtio-blk-threaded    #6128      +/-   ##
===============================================================
- Coverage                        82.73%   82.43%   -0.31%     
===============================================================
  Files                              278      278              
  Lines                            31562    31973     +411     
===============================================================
+ Hits                             26114    26356     +242     
- Misses                            5448     5617     +169
Flag Coverage Δ
5.10-m5n.metal 82.59% <59.24%> (-0.35%) ⬇️
5.10-m6a.metal 81.96% <59.24%> (-0.35%) ⬇️
5.10-m6g.metal 79.48% <59.24%> (-0.31%) ⬇️
5.10-m6i.metal 82.60% <59.24%> (-0.34%) ⬇️
5.10-m7a.metal-48xl 81.95% <59.24%> (-0.34%) ⬇️
5.10-m7g.metal 79.48% <59.24%> (-0.31%) ⬇️
5.10-m7i.metal-24xl 82.57% <59.24%> (-0.34%) ⬇️
5.10-m7i.metal-48xl 82.57% <59.24%> (-0.34%) ⬇️
5.10-m8g.metal-24xl 79.48% <59.24%> (-0.31%) ⬇️
5.10-m8g.metal-48xl 79.48% <59.24%> (-0.31%) ⬇️
5.10-m8i.metal-48xl 82.57% <59.24%> (-0.34%) ⬇️
5.10-m8i.metal-96xl 82.57% <59.24%> (-0.35%) ⬇️
5.10-m9g.metal-48xl 79.48% <59.24%> (-0.30%) ⬇️
6.1-m5n.metal 82.62% <59.24%> (-0.34%) ⬇️
6.1-m6a.metal 81.98% <59.24%> (-0.34%) ⬇️
6.1-m6g.metal 79.48% <59.24%> (-0.31%) ⬇️
6.1-m6i.metal 82.62% <59.24%> (-0.34%) ⬇️
6.1-m7a.metal-48xl 81.97% <59.24%> (-0.34%) ⬇️
6.1-m7g.metal 79.48% <59.24%> (-0.31%) ⬇️
6.1-m7i.metal-24xl 82.63% <59.24%> (-0.34%) ⬇️
6.1-m7i.metal-48xl 82.64% <59.24%> (-0.34%) ⬇️
6.1-m8g.metal-24xl 79.47% <59.24%> (-0.31%) ⬇️
6.1-m8g.metal-48xl 79.47% <59.24%> (-0.31%) ⬇️
6.1-m8i.metal-48xl 82.63% <59.24%> (-0.34%) ⬇️
6.1-m8i.metal-96xl 82.63% <59.24%> (-0.34%) ⬇️
6.1-m9g.metal-48xl 79.48% <59.24%> (-0.31%) ⬇️
6.18-m5n.metal 82.61% <59.24%> (-0.34%) ⬇️
6.18-m6a.metal 81.98% <59.24%> (-0.35%) ⬇️
6.18-m6g.metal 79.58% <59.24%> (-0.31%) ⬇️
6.18-m6i.metal 82.62% <59.24%> (-0.34%) ⬇️
6.18-m7a.metal-48xl 81.97% <59.24%> (-0.34%) ⬇️
6.18-m7g.metal 79.58% <59.24%> (-0.31%) ⬇️
6.18-m7i.metal-24xl 82.63% <59.24%> (-0.34%) ⬇️
6.18-m7i.metal-48xl 82.63% <59.24%> (-0.34%) ⬇️
6.18-m8g.metal-24xl 79.58% <59.24%> (-0.31%) ⬇️
6.18-m8g.metal-48xl 79.58% <59.24%> (-0.31%) ⬇️
6.18-m8i.metal-48xl 82.63% <59.24%> (-0.34%) ⬇️
6.18-m8i.metal-96xl 82.63% <59.24%> (-0.35%) ⬇️
6.18-m9g.metal-48xl 79.58% <59.24%> (-0.30%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@PierreBertholom
PierreBertholom removed this pull request from stack #6135 September 9, 2026 10:07
@PierreBertholom
PierreBertholom changed the base branch from pbertho/blk-threaded-03 to pbertho/blk-threaded-01 September 9, 2026 10:07
@PierreBertholom
PierreBertholom added this pull request to stack #6198 September 9, 2026 10:08
@PierreBertholom PierreBertholom changed the title [blk-threaded - 4/12] block: add basic worker thread structures [blk-threaded - 2/5] block: add worker thread lifecycle Sep 9, 2026
This was referenced Sep 9, 2026
Closed
Closed
Closed
Closed
@PierreBertholom
PierreBertholom force-pushed the pbertho/blk-threaded-04 branch 2 times, most recently from 0f96cb0 to 5183429 Compare September 14, 2026 18:29
Base automatically changed from pbertho/blk-threaded-01 to feature/virtio-blk-threaded September 15, 2026 14:35
ShadowCurse
ShadowCurse reviewed Sep 15, 2026
View reviewed changes
Comment thread src/vmm/src/devices/virtio/block/virtio/worker.rs Outdated
Comment thread src/vmm/src/devices/virtio/block/virtio/worker.rs Outdated
Comment thread src/vmm/src/devices/virtio/block/virtio/worker.rs Outdated
Comment thread src/vmm/src/devices/virtio/block/virtio/worker.rs Outdated
Comment thread src/vmm/src/devices/virtio/block/virtio/device.rs Outdated
Comment thread src/vmm/src/devices/virtio/block/virtio/worker.rs
@PierreBertholom
PierreBertholom force-pushed the pbertho/blk-threaded-04 branch 3 times, most recently from 96137d2 to f28dcc4 Compare September 16, 2026 15:57
ShadowCurse
ShadowCurse reviewed Sep 18, 2026
View reviewed changes
Comment thread src/vmm/src/devices/virtio/block/virtio/device.rs Outdated
Comment thread src/vmm/src/devices/virtio/block/virtio/worker.rs Outdated
@PierreBertholom
block: borrow path in DiskProperties::new
47c179c 
Use &str instead of String in DiskProperties::new because
DiskProperties no longer stores the file path.

Signed-off-by: Pierre Bertholom <pbertho@amazon.com>
ShadowCurse
ShadowCurse previously approved these changes Sep 21, 2026
View reviewed changes
not4s
not4s reviewed Sep 22, 2026
View reviewed changes
Comment thread src/vmm/src/devices/virtio/block/virtio/io/async_io.rs Outdated
not4s
not4s reviewed Sep 22, 2026
View reviewed changes

@not4s not4s left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any tests coming in later? The tests should be included in the relevant commits ideally, but if they're coming in later I'm fine with it

JackThomson2
JackThomson2 reviewed Sep 28, 2026
View reviewed changes
Comment thread src/vmm/src/devices/virtio/block/virtio/device.rs
JackThomson2
JackThomson2 reviewed Sep 28, 2026
View reviewed changes
Comment thread src/vmm/src/devices/virtio/block/virtio/worker.rs Outdated
JackThomson2
JackThomson2 reviewed Sep 28, 2026
View reviewed changes
Comment thread src/vmm/src/devices/virtio/block/virtio/worker.rs
JackThomson2
JackThomson2 reviewed Sep 28, 2026
View reviewed changes
Comment thread src/vmm/src/devices/virtio/block/virtio/worker.rs Outdated
JackThomson2
JackThomson2 reviewed Sep 28, 2026
View reviewed changes
Comment thread src/vmm/src/devices/virtio/block/virtio/worker.rs
@JackThomson2

JackThomson2 commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Overall looks great, just a few minor comments

@PierreBertholom
block: add basic worker thread structures
65d6865 
Add WorkerHandle and ThreadedWorker to manage a block worker thread
and its event loop.

Use a channel and eventfd to start and finish the worker. Treat
control send and notification failures as invariants.

Keep the structures disconnected from VirtioBlock until later commits
add lifecycle wiring.

Signed-off-by: Pierre Bertholom <pbertho@amazon.com>
@PierreBertholom
block: add worker thread activation
b94fb3b 
Add inline and threaded active states and transfer runtime resources to
the worker thread during activation.

Keep queue config and interrupt state on the VMM thread and register
runtime events on the thread that owns the data path.

Signed-off-by: Pierre Bertholom <pbertho@amazon.com>
@PierreBertholom
io_uring: retry submit_and_wait_all until all ops complete
d26c3b5 
A signal interrupting io_uring_enter makes it return early, with the
submitted count if it also submitted, else with EINTR. Loop until the
CQ holds every operation and retry on EINTR. Other errors are still
returned to the caller.

Signed-off-by: Pierre Bertholom <pbertho@amazon.com>
@PierreBertholom
block: add worker thread reset and teardown
b219540 
Add a reset path that returns data-path resources to the VMM thread
and parks the worker for reactivation.

Move ownership transitions into `_reset()`. Drain and join an active
worker thread when the device is dropped.

Signed-off-by: Pierre Bertholom <pbertho@amazon.com>
@PierreBertholom
block: add runtime updates on worker thread
1302a71 
Forward disk image updates to the worker.
Return the updated disk capacity required by the VMM to update config
space.
Wake the worker after the VMM handles a rate limiter timer event.

Signed-off-by: Pierre Bertholom <pbertho@amazon.com>
@PierreBertholom
snapshot: require paused VM to save state
75f2fd0 
Only allow snapshots to be saved while the VM is paused.

Signed-off-by: Pierre Bertholom <pbertho@amazon.com>
@PierreBertholom
block: add worker thread snapshot save support
661c427 
Pause the worker and collect its queue state when saving a snapshot.
Mark queue memory dirty after saving and use kick to resume the worker.

Keep fsync failures non-fatal. Treat other drain failures as
unrecoverable state.

Signed-off-by: Pierre Bertholom <pbertho@amazon.com>
@PierreBertholom

PierreBertholom commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Are there any tests coming in later? The tests should be included in the relevant commits ideally, but if they're coming in later I'm fine with it

They come later in #6132 (4/5 PR)

@PierreBertholom
PierreBertholom merged commit 8c0d752 into feature/virtio-blk-threaded Sep 30, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@not4s not4s not4s left review comments
@JackThomson2 JackThomson2 JackThomson2 approved these changes
@ShadowCurse ShadowCurse ShadowCurse approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@PierreBertholom @JackThomson2 @ShadowCurse @not4s