[Roadmap] Payload Transfer in TransferQueue

[Ginray]

This roadmap tracks the development of Payload Transfer in TransferQueue. Feedback, discussion, and contributions are welcome.

If you plan to work on an item, please comment below before starting to help coordinate the work and avoid duplication.

Foundation

• Payload Transfer abstraction and NIXL-UCX backend
  Introduce the Payload Transfer abstraction and the initial NIXL-UCX PUT/GET data path, while keeping ZMQ as the control plane and default payload path.

  • [PR #163]([feat] introduce payload transfer abstraction and NIXL-UCX backend #163)

Reliability

• Improve transfer lifecycle management
  Safely manage MR, buffers, pending state, and peer metadata across PREPARE, CANCEL, transfer failure, retry, and cleanup paths.

Performance

• Optimize buffer pooling and memory registration
  Reduce buffer allocation and memory registration overhead through better reuse.

• Reduce unnecessary data copies
  Remove avoidable copies in the receive path and process data directly from registered buffers where possible.

• Improve payload transfer concurrency
  Avoid serializing independent transfers through a single send executor (for example, max_workers=1) when one manager communicates with multiple StorageUnits, and avoid blocking the StorageUnit control worker while waiting for long-running NIXL GET transfers to complete. Preserve transfer lifecycle correctness while allowing concurrent data-plane progress and reducing head-of-line blocking in RL workloads.

Backend Expansion

• Add more Payload Transfer backends
  Extend the abstraction to support additional backends such as HIXL.

Validation

• Expand correctness, stability, and performance validation
  Cover larger-scale, sustained, and multi-node workloads.

Documentation

• Complete user-facing tutorials and documentation
  Add usage guides, configuration examples, and related feature documentation.

Mainline Integration

• Prepare for integration into main
  Move toward mainline integration once the feature is sufficiently stable and validated.

[xupinjie]

It is a clear technical roadmap. May I ask whether anyone is currently working on these three performance areas? I’d be happy to help with the optimization if that would be useful.

[Ginray]

It is a clear technical roadmap. May I ask whether anyone is currently working on these three performance areas? I’d be happy to help with the optimization if that would be useful.

1. Sorry that I didn’t see your reply earlier. We’ve also been working on another project in parallel recently, so I missed your message. We already have some work in progress on our side, but it is still under validation. To avoid duplicated work, we submitted PR [WIP][feat] optimize NIXL-UCX payload transfer for SimpleStorage #176 with our current implementation. We are still evaluating whether the current design is reasonable enough. You likely have much more experience with performance optimization in this area, so if you have time, we’d really appreciate it if you could take a look. If there are no major issues with the overall architecture, you are also welcome to submit PRs directly to continue improving it.

2. Our follow-up work will mainly focus on larger-scale, sustained, and multi-node workloads, and another colleague will also be involved. Apart from that, we currently do not plan to integrate more NIXL backends or capabilities, such as GDR. I heard that you may be interested in this area. If so, we can sync on what each side plans to work on in advance, so that we can avoid duplicated work as much as possible.

3. Again, sorry that I didn’t see your reply earlier. I hope this hasn’t caused any trouble with the work you may have already started.