feat: Add SetMaxResponseSize to limit response body size (#406) - #512
Open
ManuelReschke wants to merge 2 commits into
Open
feat: Add SetMaxResponseSize to limit response body size (#406)#512ManuelReschke wants to merge 2 commits into
ManuelReschke wants to merge 2 commits into
Conversation
Allow clients and individual requests to cap how much response body data is accepted. When Content-Length exceeds the limit the body is not read, saving bandwidth; otherwise a limiting reader stops at the configured size and returns ResponseBodyTooLargeError.
HEAD responses advertise Content-Length without a body; rejecting them broke ParallelDownload size probes. Narrow the handleDownload skip to Content-Length early rejects only, and document compression/keep-alive behavior. Expand tests for HEAD and download-after-unmarshal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add SetMaxResponseSize to limit response body size
Fixes #406
Summary
This change adds a built-in maximum response body size so callers no longer need to combine
DisableAutoReadResponse()with a manualio.LimitReader/io.CopyNworkaround.Client.SetMaxResponseSizeandRequest.SetMaxResponseSize(request overrides client;0means unlimited).Content-Lengthis known and exceeds the limit, close the body without reading it and returnResponseBodyTooLargeError(saves bandwidth and memory).*ResponseBodyTooLargeError.HEAD(advertised length, empty body) soParallelDownloadsize probes keep working.errors.Is(err, ErrResponseBodyTooLarge)anderrors.Asfor structured inspection ofLimit/ContentLength.Motivation
Users making many requests reported unexpected bandwidth use and wanted a first-class size cap on response bodies. The previous workaround was:
A library-level option is more ergonomic and applies consistently to auto-read, manual body reads, downloads, and result unmarshalling.
Behavior notes
• The limit applies to bytes delivered to the application after transport-level Content-Encoding handling (e.g. gzip). For auto-decompressed responses, ContentLength is often -1, so only the streaming limit applies.
• Early rejection closes the body immediately and may prevent keep-alive reuse for that connection (intentional for oversized / untrusted payloads).
• Request.SetMaxResponseSize(0) disables the client-level limit for that request.
• Negative values are treated as unlimited (0).
Example
Tests
Added coverage for:
• bodies within and exactly at the limit
• Content-Length early reject without buffering
• chunked bodies that exceed the limit while reading
• request-level override and disable of the client limit
• DisableAutoReadResponse + manual / ToBytes reads
• SetOutput / SetOutputFile download paths
• HEAD not failing on advertised Content-Length
• download still running after non-size errors (e.g. unmarshal failure)
• clone preservation and sticky reader errors