rdma: single PUT for oversize buffers + fix >2 GiB upload truncation - #241
Merged
harshavardhana merged 5 commits intoAug 4, 2026
Merged
Conversation
|
Warning Review limit reached
Next review available in: 27 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThe client adds a 4 GiB cuObject registration limit, ranged RDMA GET support, HTTP fallback for oversized buffers, 64-bit upload sizing, and updated CI tooling. ChangesLarge-buffer RDMA fallback
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant RDMA
participant HTTP
participant libcurl
Client->>Client: Compare buffer size with kCuObjMaxMemoryRegSize
alt Buffer is at most 4 GiB
Client->>RDMA: Register and transfer
RDMA-->>Client: Return transfer result
else Buffer exceeds 4 GiB
Client->>HTTP: Send complete buffer in one request
HTTP->>libcurl: Set InfileSizeLarge with curl_off_t
libcurl-->>HTTP: Upload request
end
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
A buffer larger than a single cuObject registration (cuMemObjGetDescriptor, ~4 GiB) cannot be pinned for RDMA. PutObject/GetObject already fell back to HTTP in that case, but PutObject re-chunked the buffer into a 16 MiB-part multipart upload. Since the buffer is already fully resident, that buys nothing: send it as a single PUT via the request body instead. AIStor accepts a single PUT up to 5 TiB — far beyond the registration ceiling and beyond anything a client can pin or allocate. - Add kCuObjMaxMemoryRegSize (4 GiB) and gate the RDMA attempt on it, so an oversized buffer skips a registration that is guaranteed to fail. - PutObject HTTP fallback now issues one BaseClient::PutObject from the buffer rather than a multipart stream upload. - Document the limit and the single-PUT behavior in the README; the SDK does not chunk registrations — sizing the buffer is the caller's responsibility.
Execute() set the upload size via curlpp::Options::InfileSize, i.e. CURLOPT_INFILESIZE (long), which curl documents as capped at 2 GiB — a larger single-request body was silently truncated. Any single PUT above 2 GiB was affected; it became reachable in practice once an oversized RDMA buffer (> 4 GiB) falls back to a single PUT. Switch to CURLOPT_INFILESIZE_LARGE (curl_off_t), which is correct for all sizes; Content-Length is already stamped from the full 64-bit body.size().
harshavardhana
force-pushed
the
fix/rdma-oversize-single-put
branch
from
August 4, 2026 03:47
b6df509 to
668888c
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/client.cc`:
- Around line 700-706: Preserve all caller request options in both HTTP fallback
paths: in the GET fallback, copy the original GET arguments before clearing buf
and size so version, range, conditionals, SSE-C, extra headers/query parameters,
and progress callbacks remain intact; in the PUT fallback, reuse the multipart
path’s header-building logic and copy extra_query_params, progressfunc, and
progress_userdata while retaining metadata, SSE, tags, retention, and legal
hold. Update the fallback logic near the RDMA selection and the corresponding
PUT path, and add regression coverage for version/range/SSE-C GETs and
metadata/SSE/progress PUTs.
- Around line 1221-1226: Apply the kCuObjMaxMemoryRegSize guard to the RDMA
registration checks in the multipart upload paths at both cuMemObjGetDescriptor
call sites using args.part_size. Ensure registration is attempted only when
args.part_size is within the limit; otherwise leave RDMA disabled so the HTTP
multipart path is selected, preserving the existing direct-buffer PutObject
guard.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 53444dd9-873e-4094-b0fe-2a92ef6f184f
📒 Files selected for processing (4)
README.mdinclude/miniocpp/rdma.hsrc/client.ccsrc/http.cc
The buffer/RDMA GetObject ignored GetObjectArgs::offset and always fetched the whole object into the buffer, so an object larger than one cuObject registration (> 4 GiB) had no ranged path — unlike PUT, which can be chunked via multipart. Honor the offset so a caller can stream a large object as a sequence of <= 4 GiB ranged GETs, each of which registers cleanly and rides the RDMA fast path. Per the AIStor server contract (rdmaTransferBounds): the object offset travels in a signed Range header (server derives rangeBase from it) and is independent of the buffer address carried in the RDMA token's StartAddr; the server replies 206 for the range, which the client already accepts. rdmaGet/rdmaGetWithRetry take an object range_offset (-1 = whole object); the HTTP fallback sets the same offset/length so it reads identical bytes.
The workflows floated on vcpkg's default branch and installed CMake 3.28 via apt. On arm64 Linux vcpkg has no prebuilt CMake and falls back to the system one; 3.28 is too old for vcpkg's SPDX generation (`string(JSON ... STRING_ENCODE)`), so any port built from source (e.g. openssl on a cache miss) fails with "invalid mode 'STRING_ENCODE'". amd64 uses vcpkg's own recent CMake and is unaffected. - Pin vcpkg to release 2026.07.29 for reproducible builds. - Use lukka/get-cmake to put a recent cmake+ninja on PATH (fixes arm64), and drop the apt cmake install.
harshavardhana
force-pushed
the
fix/rdma-oversize-single-put
branch
from
August 4, 2026 06:13
2a34de4 to
67a65ad
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@include/miniocpp/rdma.h`:
- Around line 257-265: Update the ranged-read logic around the range header
construction to reject or return early when range_offset is nonnegative and size
is zero, before formatting bytes=... Prevent generating a descending range such
as offset--1 while preserving valid nonzero range requests and the existing
GetObjectArgs::Validate() contract.
- Around line 226-232: Update rdmaGet and the related RDMA helper declarations
to use std::optional<int64_t> range_offset with a std::nullopt default instead
of the -1 sentinel. Adjust the internal RDMA caller to pass the optional value
and preserve whole-object behavior when it is unset, including the existing
offset behavior when a value is present.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: dd29f257-fd72-497b-b342-5f7e87b0addc
📒 Files selected for processing (4)
.github/workflows/ci-rdma.yml.github/workflows/ci.ymlinclude/miniocpp/rdma.hsrc/client.cc
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci-rdma.yml:
- Line 55: Replace the mutable lukka/get-cmake@latest reference at
.github/workflows/ci-rdma.yml:55 and .github/workflows/ci.yml:113 with the same
reviewed immutable commit SHA, preserving the existing action usage and adding
no other workflow changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: ac6ca90e-14df-40a7-a95c-387fd32aead4
📒 Files selected for processing (2)
.github/workflows/ci-rdma.yml.github/workflows/ci.yml
… action) - Apply the kCuObjMaxMemoryRegSize (4 GiB) guard to the two multipart RDMA registration sites (parallel pool + serial), matching the direct-buffer guard and the documented contract; a > 4 GiB part now skips the doomed registration and uses the HTTP multipart path. - rdmaGet: reject a zero-length ranged read before formatting the header, which would otherwise emit an invalid descending range (bytes=X-(X-1)). - ci: pin lukka/get-cmake to an immutable commit (v4.4.2) in both workflows instead of the mutable @latest.
Member
Author
CodeRabbit Autofix — AppliedAddressed 3 of 5 review items in Applied:
Deferred (with reason):
Files modified: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two related fixes for the RDMA direct-buffer path.
1. Oversize buffers → single PUT, not multipart
A buffer larger than a single cuObject registration (
cuMemObjGetDescriptor, ~4 GiB) cannot be pinned for RDMA.PutObjectalready fell back to HTTP, but re-chunked the buffer into a 16 MiB-part multipart upload. The buffer is already fully resident, so that buys nothing — send it as a single PUT via the request body instead. AIStor accepts a single PUT up to 5 TiB, far beyond the registration ceiling and beyond anything a client can pin or allocate.kCuObjMaxMemoryRegSize(4 GiB) and gate the RDMA attempt on it, so an oversized buffer skips a registration guaranteed to fail (both GET and PUT single-shot sites).PutObjectHTTP fallback now issues oneBaseClient::PutObjectfrom the buffer.2. Fix >2 GiB upload truncation (
CURLOPT_INFILESIZE_LARGE)Execute()set the upload size withcurlpp::Options::InfileSize→CURLOPT_INFILESIZE(long), which curl documents as capped at 2 GiB — larger single-request bodies were silently truncated. This became reachable once an oversized RDMA buffer falls back to a single PUT. Switch toCURLOPT_INFILESIZE_LARGE(curl_off_t), correct for all sizes.Content-Lengthwas already stamped from the full 64-bitbody.size().Testing
Relies on
ci-rdma.yml(builds-DMINIO_CPP_ENABLE_RDMA=ON) for compile coverage — I could not build locally (vcpkg deps absent). The single-PUT construction mirrors the existingpart_count == 1path; theInfileSizeLargechange is the 64-bit counterpart of the existing option.🤖 Generated with Claude Code
Summary by CodeRabbit