Skip to content

Expose set_pledged_src_size as a public API for zstd compression #329

Description

@shrekris-openai

Issue

async-compression offers no way to call set_pledged_src_size() on the zstd encoder. Could we provide a public API in async-compression to call this?

Context

zstd encoding can include an optional Frame_Content_Size value. Decompression libraries such as Python's zstandard provide APIs (e.g. decompress) that use this value to decompress the data in a single operation. Without it, these libraries must treat the encoded bytes as a stream.

Writing Frame_Content_Size requires two steps (from the zstd manual):

  1. Enable ZSTD_c_contentSizeFlag. async-compression already offers a CParameter method to do this.
  2. Call ZSTD_CCtx_setPledgedSrcSize() to set the decompressed data size. The zstd crate already exposes the public method set_pledged_src_size() for this, but it's not exposed by the async-compression crate.

When set_pledged_src_size is not called, the encoded bytes won't have a Frame_Content_Size value.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions