Test: read/write ALP encoded parquet files (apache/arrow-rs#9372) - #24373
Draft
alamb wants to merge 3 commits into
Draft
Test: read/write ALP encoded parquet files (apache/arrow-rs#9372)#24373alamb wants to merge 3 commits into
alamb wants to merge 3 commits into
Conversation
Pin arrow/parquet crates to arrow-rs rev 901e69f913bb8a1738009218898959e8f4cd7ec8 via [patch.crates-io] until arrow 60.0.0 is released. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tore 0.14) - Adapt to the new arrow_schema::Metadata type (was HashMap<String, String>): DFSchema::metadata()/ExprSchema::metadata() now return &Metadata, DFSchema constructors take impl Into<Metadata>, SchemaFieldMetadata is now an alias for Metadata, and check_metadata_with_storage_equal/format_type_and_metadata accept Option<&Metadata> - Add FieldMetadata <-> Metadata conversions and a DFHeapSize impl for Metadata - Replace deprecated fb_to_schema with try_fb_to_schema - Rework two benches that mixed DataFusion's rand 0.9 with arrow's rand 0.10 - Add new object_store 0.14 GetResult/ListResult extensions field - Update map field name expectations for new arrow spec defaults (key/value) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…che/arrow-rs#9372) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Test PR (not for merge) to try out the Parquet ALP encoder/decoder support proposed in apache/arrow-rs#9372 from DataFusion, so ALP encoded parquet files can be written and read with
datafusion-cli.Based on #24366 (update to arrow-rs main), with the
[patch.crates-io]section repointed at thealpbranch from https://github.com/sdf-jkl/arrow-rs.What changes are included in this PR?
sdf-jkl/arrow-rsbranchalp(currently commit5e9ad1c6)try_fb_to_schemaback tofb_to_schema(thealpbranch is based on an arrow-rs main commit that predates that API)alpas a recognized value for the parquetencodingwriter optionAre these changes tested?
Manually, with
datafusion-cli:Only
datafusion-cliis expected to compile/run; other tests are not expected to pass.Are there any user-facing changes?
No (test PR, not intended for merge).