feat(upload): Skip decompression - #6358
Draft
jjbayer wants to merge 4 commits into
Draft
Conversation
The `/upload` PATCH endpoint no longer decompresses request bodies. A `zstd` body is forwarded or stored verbatim and the algorithm is recorded on the object via objectstore's `precompressed` API. Any other content encoding is rejected with 415. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![sentry-warden[bot]](https://avatars.githubusercontent.com/in/2755838?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
So far, we've used tower's automatic decompression layer on all endpoints. This means that even for
/uploadrequests, we decompress everything in every relay, and then re-compress before sending the data to the upstream or objectstore.Not only is this a waste of CPU-cycles, it also causes a problem once objectstore supports resumable uploads, because the upload offset communicated by
Consequences:
Upload-Offsetcommunicated to the client (i.e. SDK) will be in compressed bytes, not uncompressed bytes. This means that the SDK either needs to have the entire file in compressed form on disk, or needs to compress the entire file on-the-fly and discard bytes upto the offset if it needs to resume from an offset > 0./uploadendpoint only supports zstandard compression or no compression, since that is what objectstore supports.Check before merging:
EventAttachmenttable will refer to compressed bytes. Does this break anything?ref: INGEST-1162