Skip to content

fix(gzip): reject invalid header before the full 10 bytes arrive - #489

Merged
NobodyXu merged 1 commit into
Nullus157:mainfrom
doxxx93:gzip-header-magic-early-reject
Sep 16, 2026
Merged

NobodyXu merged 1 commit into
Nullus157:mainfrom
doxxx93:gzip-header-magic-early-reject

Conversation

@doxxx93

@doxxx93 doxxx93 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Motivation

Fixes #488.

The gzip magic is decided by the first bytes of a member, but it is only checked in
Header::parse, which runs once all 10 bytes of the fixed header are buffered. A reader that
stalls part way through those 10 bytes never resolves, even when the bytes already seen cannot
begin a gzip member.

That is what makes multiple_members unusable over a live stream: once a member completes,
trailing bytes shorter than a header leave the decoder waiting rather than failing.
tower-rs/tower-http#621 disabled multiple_members for gzip over exactly this case, which is why
tower-http and reqwest (which delegates to it since 0.6.8) now reject concatenated gzip with
"there are extra bytes after body has been decompressed". Kubernetes 1.37 emits concatenated gzip
on watch streams, so this is reachable from ordinary HTTP clients today.

Solution

Check the magic against the bytes seen so far on every fill of State::Fixed, so an impossible
header fails immediately regardless of how it is chunked. Header::parse keeps its own check.

The new test mirrors bufread_multiple_members_with_invalid_padding from xz.rs, but reads from a
reader that yields the bytes and then stays pending, which is what a live HTTP body looks like.
Without the fix it hits the ntest timeout; with it, it errors immediately.

End to end, with this branch patched in and multiple_members(true) restored on tower-http's
GzipDecoder, reqwest's whole tests/gzip.rs passes, including
test_chunked_fragmented_response_with_extra_bytes which motivated tower-http#621, plus variants
carrying 2, 10 and 16 bytes of trailing garbage. Before the fix, the 2 and 5 byte cases hang and
only 10 bytes or more produce an error.

The magic is decided by the first bytes of a member, but it was only checked
in `Header::parse`, which runs once all 10 bytes of the fixed header are
buffered. A reader that stalls part way through those 10 bytes therefore never
resolves, even when the bytes already seen cannot begin a gzip member.

This matters for `multiple_members` over a live stream: after a member
completes, trailing bytes shorter than a header leave the decoder waiting
instead of failing. tower-rs/tower-http#621 disabled `multiple_members` for
gzip over exactly this, which is why tower-http and reqwest now reject
concatenated gzip responses.

Check the magic against the bytes seen so far on every fill, so an impossible
header fails immediately regardless of how it is chunked.
@doxxx93
doxxx93 force-pushed the gzip-header-magic-early-reject branch from 319dc05 to 76371aa Compare September 16, 2026 12:28
@doxxx93

doxxx93 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Two things worth calling out for review.

The error kind changes when a stream ends part way through a header. A member followed by two
garbage bytes and EOF used to give UnexpectedEof / "unexpected end of file"; it now gives
InvalidData / "Invalid gzip header". The decoded output is the same either way, and I think
InvalidData is the more accurate one since the EOF is not what is wrong, but anyone matching on
the kind would see the difference. Nothing in the test suite depends on it.

This does not remove stalls in general. Trailing bytes that do start with 1f 8b 08 and then stop
still wait, which is correct, since that is indistinguishable from a next member arriving slowly.
The guarantee here is only that bytes already proven not to begin a member fail immediately.

@NobodyXu NobodyXu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Will cut a new release now after merging

@NobodyXu
NobodyXu added this pull request to the merge queue Sep 16, 2026
Merged via the queue into Nullus157:main with commit 7f2e753 Sep 16, 2026
23 checks passed
@github-actions github-actions Bot mentioned this pull request Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gzip header magic is only checked once 10 bytes are buffered, stalling multi-member decoding

2 participants