Skip to content

interop.rst

Latest commit

 

History

History
225 lines (168 loc) · 8.67 KB

interop.rst

File metadata and controls

225 lines (168 loc) · 8.67 KB
Raw

hipObject cuObject Interoperability

This guide summarizes compatibility with NVIDIA cuObject v1.2.0 and documents how to test hipObject on lab hardware.

Compatibility matrix

Feature cuObject v1.2.0 hipObject
S3 x-amz-rdma-token Yes Yes
S3 x-amz-rdma-reply Yes Yes
GET / PUT / multipart Yes Yes (via SDK)
RDMA transport DC (ConnectX) RC (bnxt/ionic)
Direct cuObject server N/A adapter

Control plane

hipObject is wire-compatible with cuObject at the S3 header level. Applications embed a hex-encoded RDMA token in x-amz-rdma-token and read x-amz-rdma-reply from the HTTP response.

x-amz-rdma-token

The header value is exactly one fixed-width RDMA token encoded as 88 hex characters (44 binary bytes). No suffix or additional :-separated component is permitted in the header value.

The binary payload layout is:

Byte Width Type Meaning
0 1 byte u8 transport
1 4 bytes u32 LE QP number
5 16 bytes bytes GID, copied verbatim
21 4 bytes u32 LE rkey
25 8 bytes u64 LE remote address
33 8 bytes u64 LE length
41 1 byte u8 port number
42 2 bytes u16 LE LID

The fields are packed with no padding between them. The integers are little-endian whatever the byte order of the host that encodes or decodes the token.

Enumerated values:

  • transport = 0x00: DC
  • transport = 0x01: RC

encodeRdmaToken() emits lowercase hex, but receivers accept either hex case. A peer that validates the token as a fixed-length hex string will reject any value longer than 88 hex characters.

x-amz-rdma-reply

Clients must accept these forms:

  • ok — legacy success marker, deprecated
  • err — legacy error marker, deprecated
  • <http-status> — decimal HTTP status code such as 200 or 501
  • 200:<server-token-hex> — RC peer handshake reply carrying one 88-hex RDMA token

Only 200:<server-token-hex> carries a peer token. Legacy ok / err remain accepted for cuObject compatibility but new implementations should prefer numeric status codes, and RC peer handshakes should use the 200:<server-token-hex> form.

The peer token has the same layout as the client's token, but only names the server's QP, which the client connects its own QP to. The client uses these fields:

  • transport, which must be RC (0x01); a reply with a DC peer token fails the transfer
  • QP number
  • GID
  • LID

The rkey, remote address, length, and port number are reserved: servers must set them to zero, and clients ignore them. The client never reads or writes the server's memory, since the server performs the transfer in both directions, with an RDMA WRITE for a GET and an RDMA READ for a PUT, so a reply doesn't advertise any server memory. The client connects its QP on its own port, not the one in the peer token.

Data plane

cuObject v1.2.0 requires Dynamic Connection (DC) transport on NVIDIA ConnectX NICs. hipObject uses Reliable Connection (RC) on Broadcom Thor-2 and AMD Pensando Pollara NICs.

A stock libcuobjserver stack cannot serve RC clients directly. An RC-to-DC adapter bridges AMD RC clients to cuObject-equipped storage gateways in production deployments.

The client and the server don't negotiate a path MTU, and the token doesn't carry one, so both ends' MTUs must match. hipObject sets its connections' path MTU to 4096 bytes, so configure the client's and the server's ports, and the network between them, for 4096-byte RDMA packets. With RoCE, that takes jumbo frames, usually an Ethernet MTU of 9000. If the MTUs don't match, transfers hang until the server times out, without an error on either end.

Testing paths

Layer 1 — Unit tests (CI)

test-rdma-token tests token encoding and decoding and reply parsing. It's part of the ctest suite, which runs in GitHub Actions without a GPU, an RDMA NIC, or an S3 server:

cmake -B build -DBUILD_TESTING=ON
cmake --build build
ctest --test-dir build -R test-rdma-token --output-on-failure

Leave out -R test-rdma-token to run the whole suite.

Layer 2 — RC integration test server

hipobj-rdma-test-server is an in-repo S3+RDMA harness that speaks RC and returns peer tokens for the client-side QP handshake:

cmake -B build -DBUILD_TESTING=ON
cmake --build build --target hipobj-rdma-test-server
./build/test/integration/rdma-test-server/hipobj-rdma-test-server 9000

On a GPU client node with libcurl installed:

cmake -B build -DBUILD_TESTING=ON
cmake --build build --target get-object
./build/examples/get-object 4096 --live http://127.0.0.1:9000

Layer 3 — MinIO AIStor bridge

Build the minio-cpp bridge for production-style S3 signing and retries:

cmake -B build -DHIPOBJ_MINIO_CLIENT=ON
cmake --build build --target minio-getput-rdma

See integrations/minio-cpp/TESTING.md for AIStor + cuObjServer setup.

Layer 4 — cuObject server probe (optional)

When libcuobjserver is installed locally:

cmake -B build \
  -DHIPOBJ_FIND_CUOBJECT_SERVER=ON \
  -DCUOBJSERVER_ROOT=/opt/nvidia/cuobjserver \
  -DBUILD_TESTING=ON
cmake --build build --target hipobj-cuobject-probe

This verifies linkage only. End-to-end cuObject interop still requires ConnectX hardware and the RC adapter on the storage gateway.

Emulated hardware (CI)

The hipobject-hardware-test-* workflows run the RDMA data path under QEMU, on rocm-ernic's emulated ionic NIC and, for GPU buffers, rocjitsu's emulated GPU. They move data between minio-getput-rdma (see Layer 3) and hipobj-rdma-test-server or rocm-ernic's own S3 store, in host and GPU memory.

Hardware topology

AMD GPU client                Storage server
+----------------+            +------------------+
| MI300 + hipObj |  RoCEv2 RC  | RC adapter       |
| bnxt / ionic   | ----------> | cuObjServer (DC) |
+----------------+            | MinIO AIStor     |
                              +------------------+

CMake options

HIPOBJ_FETCH_CUOBJECT_CLIENT
Download libcuobjclient v1.2.0.59 from the NVIDIA CUDA CDN.
HIPOBJ_FIND_CUOBJECT_SERVER
Locate a system libcuobjserver and build hipobj-cuobject-probe.
HIPOBJ_INTEGRATION_TESTS
Build hipobj-rdma-test-server (default ON when testing is enabled).
HIPOBJ_MINIO_CLIENT
Build the minio-cpp RDMA bridge and minio-getput-rdma example.