Skip to content

fix(downloader): keep the largest object size across 416 replica fallbacks - #154

Closed
detail-app[bot] wants to merge 1 commit into
mainfrom
detail/bug-fix/fix-downloader-keep-the-largest-object-size-across-407300
Closed

detail-app[bot] wants to merge 1 commit into
mainfrom
detail/bug-fix/fix-downloader-keep-the-largest-object-size-across-407300

Conversation

@detail-app

@detail-app detail-app Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Detail bug report: View on Detail

Closes #153

Bug

On a ranged read where every replica returns HTTP 416 for an out-of-range page, the 416 response could report a too-small object_size in its Content-Range: bytes */{size} header.

  • download_replicas falls back across replicas on RangeNotSatisfied (since a5321a9), accumulating the error to return via record_fallback_error.
  • The accumulator kept the last error within priority 3, so a truncated/shorter replica tried after the full primary could overwrite the primary's correct object_size with its own smaller one.
  • The HTTP layer (on_chunk_error) then emitted Content-Range: bytes */{truncated_size} — wrong for object-size discovery. In-range reads self-correct (206 carries the true size), so the impact is limited to clients relying on the 416 header without a follow-up in-range request.

Fix

record_fallback_error now keeps the largest object_size among accumulated RangeNotSatisfied errors, while preserving the existing latest-wins rule for every other priority-3 error. should_attempt_fallback_bucket is unchanged, so the cross-replica 416 fallback behavior from a5321a9 is retained.

Testing

  • Unit tests (cargo nextest run --locked --all-features --lib): 99 passed, 1 skipped (simulation::campaign, #[ignore] by design). Added range_not_satisfied_fallbacks_keep_the_largest_object_size and range_not_satisfied_survives_lower_priority_fallback_errors (replicas), and range_not_satisfied_emits_416_with_object_size_content_range (routes, asserting the 416 Content-Range contract for Some(N) and the None case). The largest-size test fails on the pre-fix latest-wins logic (object_size: Some(4)) and passes on the fix (Some(17)), in both fallback orderings.
  • Integration tests (testcontainers RustFS, Docker): downloader_integration_test (3/3, incl. unsatisfied_ranges_fall_back_to_other_replicas) and server_integration_test (10/10, incl. the out-of-range 416 case) pass.
  • Clippy (-D warnings --allow deprecated) and nightly fmt --check clean; cargo build --locked --all-features --all-targets succeeds.
  • End-to-end through the real SDK stack: a throwaway scripted S3 connector emitting Content-Range: bytes */N on 416 (mimicking AWS S3; the RustFS test backend omits it, confirmed object_size: None) drove Downloader::download with a full primary (size 17) before a truncated fallback (size 4); it returns Some(17) on the fix and Some(4) on the reverted buggy logic, in both orderings. Not versioned (one-off verification artifact).
  • Could not run against AWS S3 directly (no credentials) nor MinIO (docker pull denied); the scripted-connector check exercises the same download_replicas → fetch_piece → map_get_object_error → record_fallback_error path a real S3 backend would.

Automatic Fixes PRs can be configured here.

@detail-app
detail-app Bot requested a review from shikhar September 29, 2026 05:23
@greptile-apps

greptile-apps Bot commented Sep 29, 2026

Copy link
Copy Markdown

PR author is not in the allowed authors list.

@shikhar

shikhar commented Sep 29, 2026

Copy link
Copy Markdown
Member

Superseded by #155, which preserves the largest observed 416 size across responses without a size and intervening backend errors, while retaining existing error precedence. The replacement includes regression coverage through the SDK downloader path.

@shikhar shikhar closed this Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Detail Bug] HTTP 416 responses can report the wrong object size when replica fallback occurs

1 participant