[blk-threaded - 5/5] block: add virtio-blk multiqueue by PierreBertholom · Pull Request #6197 · firecracker-microvm/firecracker

PierreBertholom · 2026-09-09T10:08:12Z

[5/5] part of the PR stack starting with #6123

Extend threaded virtio-block from one queue to multiple queues.

Changes

Add num_queues to the API configuration and virtio config space.
Keep 1 queue as the default.
Advertise VIRTIO_BLK_F_MQ when the device has multiple queues.
Allow multiqueue only in threaded mode.
Check the queue count against the supported limit and configured vCPU count.
Prevent vCPU updates that leave a drive with too many queues.
Create one BlockResources value and one worker for each configured queue.
Share the drive rate limiter across all queues.
Allow guest drivers to use fewer queues than Firecracker exposes.
Extend activation, reset, snapshots, and drive updates to all queues.
Reject multiqueue for vhost-user block devices.
Add documentation for threaded mode and multiqueue.

codecov · 2026-09-09T10:14:06Z

Codecov Report

❌ Patch coverage is 87.87879% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.86%. Comparing base (8b93126) to head (4bdcf5c).

Files with missing lines	Patch %	Lines
src/vmm/src/devices/virtio/block/virtio/device.rs	85.42%	29 Missing ⚠️
...m/src/devices/virtio/block/virtio/event_handler.rs	57.14%	3 Missing ⚠️
src/vmm/src/devices/virtio/block/virtio/persist.rs	94.11%	2 Missing ⚠️
.../vmm/src/devices/virtio/block/vhost_user/device.rs	66.66%	1 Missing ⚠️
src/vmm/src/devices/virtio/block/virtio/worker.rs	94.73%	1 Missing ⚠️

Additional details and impacted files

@@                       Coverage Diff                       @@
##           feature/virtio-blk-threaded    #6197      +/-   ##
===============================================================
+ Coverage                        82.84%   82.86%   +0.01%     
===============================================================
  Files                              278      278              
  Lines                            32028    32180     +152     
===============================================================
+ Hits                             26535    26665     +130     
- Misses                            5493     5515      +22

Flag	Coverage Δ
5.10-m5n.metal	`83.08% <87.87%> (+0.01%)`	⬆️
5.10-m6a.metal	`82.46% <87.87%> (+0.01%)`	⬆️
5.10-m6g.metal	`79.98% <87.87%> (+0.02%)`	⬆️
5.10-m6i.metal	`83.07% <87.87%> (+<0.01%)`	⬆️
5.10-m7a.metal-48xl	`82.45% <87.87%> (+0.02%)`	⬆️
5.10-m7g.metal	`79.98% <87.87%> (+0.02%)`	⬆️
5.10-m7i.metal-24xl	`83.05% <87.87%> (+<0.01%)`	⬆️
5.10-m7i.metal-48xl	`83.05% <87.87%> (+<0.01%)`	⬆️
5.10-m8g.metal-24xl	`79.98% <87.87%> (+0.02%)`	⬆️
5.10-m8g.metal-48xl	`79.98% <87.87%> (+0.02%)`	⬆️
5.10-m8i.metal-48xl	`83.05% <87.87%> (+<0.01%)`	⬆️
5.10-m8i.metal-96xl	`83.05% <87.87%> (+<0.01%)`	⬆️
5.10-m9g.metal-48xl	`79.98% <87.87%> (+0.02%)`	⬆️
6.1-m5n.metal	`83.10% <87.87%> (+0.01%)`	⬆️
6.1-m6a.metal	`82.48% <87.87%> (+0.01%)`	⬆️
6.1-m6g.metal	`79.98% <87.87%> (+0.02%)`	⬆️
6.1-m6i.metal	`83.09% <87.87%> (+0.01%)`	⬆️
6.1-m7a.metal-48xl	`82.47% <87.87%> (+0.02%)`	⬆️
6.1-m7g.metal	`79.98% <87.87%> (+0.02%)`	⬆️
6.1-m7i.metal-24xl	`83.11% <87.87%> (+0.01%)`	⬆️
6.1-m7i.metal-48xl	`83.11% <87.87%> (+<0.01%)`	⬆️
6.1-m8g.metal-24xl	`79.98% <87.87%> (+0.02%)`	⬆️
6.1-m8g.metal-48xl	`79.98% <87.87%> (+0.02%)`	⬆️
6.1-m8i.metal-48xl	`83.11% <87.87%> (+<0.01%)`	⬆️
6.1-m8i.metal-96xl	`83.11% <87.87%> (+0.01%)`	⬆️
6.1-m9g.metal-48xl	`79.98% <87.87%> (+0.02%)`	⬆️
6.18-m5n.metal	`83.09% <87.87%> (+0.01%)`	⬆️
6.18-m6a.metal	`82.48% <87.87%> (+0.01%)`	⬆️
6.18-m6g.metal	`80.08% <87.87%> (+0.02%)`	⬆️
6.18-m6i.metal	`83.10% <87.87%> (+0.01%)`	⬆️
6.18-m7a.metal-48xl	`82.47% <87.87%> (+0.01%)`	⬆️
6.18-m7g.metal	`80.09% <87.87%> (+0.02%)`	⬆️
6.18-m7i.metal-24xl	`83.11% <87.87%> (+<0.01%)`	⬆️
6.18-m7i.metal-48xl	`83.11% <87.87%> (+0.01%)`	⬆️
6.18-m8g.metal-24xl	`80.08% <87.87%> (+0.02%)`	⬆️
6.18-m8g.metal-48xl	`80.09% <87.87%> (+0.03%)`	⬆️
6.18-m8i.metal-48xl	`83.11% <87.87%> (+0.01%)`	⬆️
6.18-m8i.metal-96xl	`83.11% <87.87%> (+0.01%)`	⬆️
6.18-m9g.metal-48xl	`80.09% <87.87%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ShadowCurse · 2026-10-01T15:02:57Z

+                resources.len(),
+                "one worker must be spawned for each block queue"
+            );
+            let queue_config = resources.iter().map(|r| r.queue.config.clone()).collect();


queue_configs to make it clear that this is an array

ShadowCurse · 2026-10-01T15:16:24Z

+                for handle in active.worker_handles {
+                    handle.finish(flush_mode);
+                }


All workers of the same block share the underlying file (they dupe the fd, but the file is same), so you only need to flush once and it can be done from the main thread (or just first worker thread)

applied, now n-1 workers drain and the last one drains and flush, also applied to prepare_save()

ShadowCurse · 2026-10-01T15:22:36Z

    fn register_runtime_events(&self, ops: &mut EventOps) {
        if let Err(err) = ops.add(Events::with_data(
-            &self.resources().queue_evt,
+            &self.resources()[0].queue_evt,


I think it is worth adding an assert here as well to ensure resources().len() == 1 (and in couple other places where [0] is used assuming a inline threaded version)

ShadowCurse · 2026-10-01T15:45:07Z

+        if num_queues > 1 {
+            avail_features |= 1u64 << VIRTIO_BLK_F_MQ;
+        }


The spec says that VIRTIO_BLK_F_MQ just enables num_queues field and since it always exist now, we can enable this feature unconditionally

The block `ConfigSpace` already has a `num_queues` field, but it is always 0. Default it to 1 and make `num_queues()` report its value. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

Store runtime resources and parked workers in vectors before activation. After activation, keep one worker handle and queue config per queue. Update reset, disk updates, rate-limit kicks, teardown, and persistence to iterate over every queue. All queues share the same backing file, so only the last worker flushes it on snapshot and teardown while the others only drain their pending I/O. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

Virtio drivers can use fewer queues than the device exposes. Keep unused queues uninitialized while their workers remain available for renegotiation. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

Replace the `threaded` drive option with an optional `num_queues`. When it is set, the drive is processed on dedicated worker threads. When it is unset, the drive keeps using the VMM thread. Use the same representation in the virtio-block configuration so threading is derived from `num_queues`, and reject `num_queues` for vhost-user block devices. The device keeps exposing a single queue until multiqueue is enabled. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

Reject a queue count greater than the vCPU count when a drive is configured, loaded from the config file or hotplugged, and reject a vCPU count update below an existing drive's queue count. The check lives in `MachineConfig` so other multiqueue devices can reuse it. A queue count of zero is rejected when parsing the `NonZeroU16` field. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

Create one queue with its own resources for each configured queue. Advertise VIRTIO_BLK_F_MQ unconditionally, since the config space always reports the queue count. Restore rebuilds one slot per saved queue and checks that multi-queue snapshots offer VIRTIO_BLK_F_MQ and that threaded mode matches the queue count. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

Cover API validation, block hotplug, guest I/O, and snapshot restore. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

Describe threaded processing, multiqueue limits, and worker resources. Document seccomp requirements and partial updates across queues. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

Document the new `num_queues` virtio-block drive option. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

PierreBertholom added this pull request to stack #6198 September 9, 2026 10:08

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch from b8f2eff to dbbe406 Compare September 10, 2026 15:24

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch 2 times, most recently from 189c34d to 9b42f24 Compare September 14, 2026 20:36

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch from 9b42f24 to a3ada00 Compare September 15, 2026 14:35

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch from a3ada00 to 152320f Compare September 15, 2026 22:51

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch from 152320f to 31afecd Compare September 16, 2026 15:05

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch from 31afecd to 90dfb97 Compare September 16, 2026 15:57

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch from 90dfb97 to 7859850 Compare September 18, 2026 14:46

ShadowCurse reviewed Sep 25, 2026

View reviewed changes

Comment thread src/vmm/src/devices/virtio/block/virtio/device.rs

Comment thread src/vmm/src/devices/virtio/block/virtio/device.rs Outdated

Comment thread src/firecracker/swagger/firecracker.yaml

Comment thread src/vmm/src/vmm_config/drive.rs Outdated

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch from 7859850 to 7937846 Compare September 29, 2026 14:18

dangra mentioned this pull request Sep 29, 2026

vmm: add a Threaded block IO engine, with a worker that serves the virtqueue superfly/firecracker#20

Merged

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch from 7937846 to c69fe28 Compare September 30, 2026 09:49

PierreBertholom mentioned this pull request Sep 30, 2026

[blk-threaded - 2/5] block: add worker thread lifecycle #6128

Merged

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch 2 times, most recently from 3123f2f to 1629165 Compare October 1, 2026 12:40

ShadowCurse force-pushed the pbertho/blk-threaded-multiqueue branch from 1629165 to 47e3311 Compare October 1, 2026 12:55

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch from 47e3311 to c1ae0ce Compare October 1, 2026 13:01

Base automatically changed from pbertho/blk-threaded-10 to feature/virtio-blk-threaded October 1, 2026 15:30

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch from c1ae0ce to b51fb22 Compare October 1, 2026 15:30

ShadowCurse reviewed Oct 1, 2026

View reviewed changes

PierreBertholom added 9 commits October 2, 2026 14:43

block: report num_queues from ConfigSpace

3c88322

The block `ConfigSpace` already has a `num_queues` field, but it is always 0. Default it to 1 and make `num_queues()` report its value. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

block: allow drivers to use fewer queues than exposed

a9e67d3

Virtio drivers can use fewer queues than the device exposes. Keep unused queues uninitialized while their workers remain available for renegotiation. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

tests: cover virtio-blk multiqueue

9d0123a

Cover API validation, block hotplug, guest I/O, and snapshot restore. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

docs: document threaded block and multiqueue

b215f41

Describe threaded processing, multiqueue limits, and worker resources. Document seccomp requirements and partial updates across queues. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

docs(changelog): note virtio-block worker threads and multiqueue

4bdcf5c

Document the new `num_queues` virtio-block drive option. Signed-off-by: Pierre Bertholom <pbertho@amazon.com>

PierreBertholom force-pushed the pbertho/blk-threaded-multiqueue branch from b51fb22 to 4bdcf5c Compare October 2, 2026 13:45

Navigation Menu

[blk-threaded - 5/5] block: add virtio-blk multiqueue - #6197

[blk-threaded - 5/5] block: add virtio-blk multiqueue#6197
PierreBertholom wants to merge 9 commits into
feature/virtio-blk-threadedfrom
pbertho/blk-threaded-multiqueue

PierreBertholom commented Sep 9, 2026

Uh oh!

codecov Bot commented Sep 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ShadowCurse Oct 1, 2026

Uh oh!

PierreBertholom Oct 2, 2026

Uh oh!

ShadowCurse Oct 1, 2026

Uh oh!

PierreBertholom Oct 2, 2026

Uh oh!

ShadowCurse Oct 1, 2026

Uh oh!

PierreBertholom Oct 2, 2026

Uh oh!

ShadowCurse Oct 1, 2026

Uh oh!

PierreBertholom Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

PierreBertholom commented Sep 9, 2026

Changes

Uh oh!

codecov Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Sep 9, 2026 •

edited

Loading