Skip to content

Fix GzipReader#gets(limit) splitting a multibyte character at the buffer end - #178

Open
segiddins wants to merge 1 commit into
ruby:masterfrom
segiddins:segiddins/gzreader-gets-limit-charboundary
Open

segiddins wants to merge 1 commit into
ruby:masterfrom
segiddins:segiddins/gzreader-gets-limit-charboundary

Conversation

@segiddins

Copy link
Copy Markdown
Contributor

gzreader_charboundary finds how many bytes of the character at the limit are missing from the inflated buffer, but asked gzfile_fill for only n + missing bytes. When the buffer already held more than n bytes nothing new was read, and the character was cut. The inflated buffer ends inside a character whenever a 2048-byte read of compressed data does.

content = (("\u3042\u3044\u3046" * 700) + "\n") * 3
r = Zlib::GzipReader.new(StringIO.new(Zlib.gzip(content, level: 0)))
lines = []
while (line = r.gets(1)) do lines << line end
lines.reject(&:valid_encoding?).size
# expected: 0
# actual:   2

Request the buffer length plus the missing bytes instead.

Fixes #92

…fer end

gzreader_charboundary finds how many bytes of the character at the limit are missing from the inflated buffer, but asked gzfile_fill for only n + missing bytes. When the buffer already held more than n bytes nothing new was read, and the character was cut. The inflated buffer ends inside a character whenever a 2048-byte read of compressed data does.

```ruby
content = (("\u3042\u3044\u3046" * 700) + "\n") * 3
r = Zlib::GzipReader.new(StringIO.new(Zlib.gzip(content, level: 0)))
lines = []
while (line = r.gets(1)) do lines << line end
lines.reject(&:valid_encoding?).size
# expected: 0
# actual:   2
```

Request the buffer length plus the missing bytes instead.

Fixes ruby#92

Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GzipReader: gets sometimes returns incomplete multi-byte characters

1 participant