Skip to content

feat: Add SetMaxResponseSize to limit response body size (#406) - #512

Open
ManuelReschke wants to merge 2 commits into
imroc:masterfrom
ManuelReschke:feat_406
Open

feat: Add SetMaxResponseSize to limit response body size (#406)#512
ManuelReschke wants to merge 2 commits into
imroc:masterfrom
ManuelReschke:feat_406

Conversation

@ManuelReschke

Copy link
Copy Markdown
Contributor

Add SetMaxResponseSize to limit response body size

Fixes #406

Summary

This change adds a built-in maximum response body size so callers no longer need to combine DisableAutoReadResponse() with a manual io.LimitReader / io.CopyN workaround.

  • Add Client.SetMaxResponseSize and Request.SetMaxResponseSize (request overrides client; 0 means unlimited).
  • When Content-Length is known and exceeds the limit, close the body without reading it and return ResponseBodyTooLargeError (saves bandwidth and memory).
  • When length is unknown (e.g. chunked), wrap the body with a limiting reader that stops at the configured size and returns a sticky *ResponseBodyTooLargeError.
  • Skip the Content-Length early reject for HEAD (advertised length, empty body) so ParallelDownload size probes keep working.
  • Support errors.Is(err, ErrResponseBodyTooLarge) and errors.As for structured inspection of Limit / ContentLength.

Motivation

Users making many requests reported unexpected bandwidth use and wanted a first-class size cap on response bodies. The previous workaround was:

client.DisableAutoReadResponse()
resp, err := client.R().Get(url)
// manually limit reads from resp.Body

A library-level option is more ergonomic and applies consistently to auto-read, manual body reads, downloads, and result unmarshalling.

Behavior notes

• The limit applies to bytes delivered to the application after transport-level Content-Encoding handling (e.g. gzip). For auto-decompressed responses, ContentLength is often -1, so only the streaming limit applies.
• Early rejection closes the body immediately and may prevent keep-alive reuse for that connection (intentional for oversized / untrusted payloads).
• Request.SetMaxResponseSize(0) disables the client-level limit for that request.
• Negative values are treated as unlimited (0).

Example

client := req.C().SetMaxResponseSize(1 << 20) // 1 MiB

resp, err := client.R().Get("https://example.com/large")
if errors.Is(err, req.ErrResponseBodyTooLarge) {
    var e *req.ResponseBodyTooLargeError
    if errors.As(err, &e) {
        // e.Limit, e.ContentLength (-1 if exceeded while reading)
    }
}

// Per-request override
resp, err = client.R().SetMaxResponseSize(64 << 10).Get(url)

Tests

Added coverage for:

• bodies within and exactly at the limit
• Content-Length early reject without buffering
• chunked bodies that exceed the limit while reading
• request-level override and disable of the client limit
• DisableAutoReadResponse + manual / ToBytes reads
• SetOutput / SetOutputFile download paths
• HEAD not failing on advertised Content-Length
• download still running after non-size errors (e.g. unmarshal failure)
• clone preservation and sticky reader errors

Allow clients and individual requests to cap how much response body data
is accepted. When Content-Length exceeds the limit the body is not read,
saving bandwidth; otherwise a limiting reader stops at the configured
size and returns ResponseBodyTooLargeError.
HEAD responses advertise Content-Length without a body; rejecting them
broke ParallelDownload size probes. Narrow the handleDownload skip to
Content-Length early rejects only, and document compression/keep-alive
behavior. Expand tests for HEAD and download-after-unmarshal.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support size limit of response body?

1 participant