fix(gzip): reject invalid header before the full 10 bytes arrive - #489
Conversation
The magic is decided by the first bytes of a member, but it was only checked in `Header::parse`, which runs once all 10 bytes of the fixed header are buffered. A reader that stalls part way through those 10 bytes therefore never resolves, even when the bytes already seen cannot begin a gzip member. This matters for `multiple_members` over a live stream: after a member completes, trailing bytes shorter than a header leave the decoder waiting instead of failing. tower-rs/tower-http#621 disabled `multiple_members` for gzip over exactly this, which is why tower-http and reqwest now reject concatenated gzip responses. Check the magic against the bytes seen so far on every fill, so an impossible header fails immediately regardless of how it is chunked.
319dc05 to
76371aa
Compare
|
Two things worth calling out for review. The error kind changes when a stream ends part way through a header. A member followed by two This does not remove stalls in general. Trailing bytes that do start with |
Motivation
Fixes #488.
The gzip magic is decided by the first bytes of a member, but it is only checked in
Header::parse, which runs once all 10 bytes of the fixed header are buffered. A reader thatstalls part way through those 10 bytes never resolves, even when the bytes already seen cannot
begin a gzip member.
That is what makes
multiple_membersunusable over a live stream: once a member completes,trailing bytes shorter than a header leave the decoder waiting rather than failing.
tower-rs/tower-http#621 disabled
multiple_membersfor gzip over exactly this case, which is whytower-http and reqwest (which delegates to it since 0.6.8) now reject concatenated gzip with
"there are extra bytes after body has been decompressed". Kubernetes 1.37 emits concatenated gzip
on watch streams, so this is reachable from ordinary HTTP clients today.
Solution
Check the magic against the bytes seen so far on every fill of
State::Fixed, so an impossibleheader fails immediately regardless of how it is chunked.
Header::parsekeeps its own check.The new test mirrors
bufread_multiple_members_with_invalid_paddingfromxz.rs, but reads from areader that yields the bytes and then stays pending, which is what a live HTTP body looks like.
Without the fix it hits the
ntesttimeout; with it, it errors immediately.End to end, with this branch patched in and
multiple_members(true)restored on tower-http'sGzipDecoder, reqwest's wholetests/gzip.rspasses, includingtest_chunked_fragmented_response_with_extra_byteswhich motivated tower-http#621, plus variantscarrying 2, 10 and 16 bytes of trailing garbage. Before the fix, the 2 and 5 byte cases hang and
only 10 bytes or more produce an error.