Skip to content

Test: read/write ALP encoded parquet files (apache/arrow-rs#9372) - #24373

Draft
alamb wants to merge 3 commits into
apache:mainfrom
alamb:alamb/test_alp_encoding
Draft

Test: read/write ALP encoded parquet files (apache/arrow-rs#9372)#24373
alamb wants to merge 3 commits into
apache:mainfrom
alamb:alamb/test_alp_encoding

Conversation

@alamb

@alamb alamb commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Test PR (not for merge) to try out the Parquet ALP encoder/decoder support proposed in apache/arrow-rs#9372 from DataFusion, so ALP encoded parquet files can be written and read with datafusion-cli.

Based on #24366 (update to arrow-rs main), with the [patch.crates-io] section repointed at the alp branch from https://github.com/sdf-jkl/arrow-rs.

What changes are included in this PR?

  • Pin the arrow/parquet git patch to sdf-jkl/arrow-rs branch alp (currently commit 5e9ad1c6)
  • Adjust try_fb_to_schema back to fb_to_schema (the alp branch is based on an arrow-rs main commit that predates that API)
  • Add alp as a recognized value for the parquet encoding writer option

Are these changes tested?

Manually, with datafusion-cli:

> COPY (SELECT random() AS r, CAST(random() AS FLOAT) AS f FROM generate_series(1,10000))
  TO '/tmp/alp_test.parquet'
  OPTIONS ('format.encoding' 'alp', 'format.dictionary_enabled' 'false');

> SELECT path_in_schema, type, encodings FROM parquet_metadata('/tmp/alp_test.parquet');
+----------------+--------+------------+
| path_in_schema | type   | encodings  |
+----------------+--------+------------+
| "r"            | DOUBLE | [RLE, ALP] |
| "f"            | FLOAT  | [RLE, ALP] |
+----------------+--------+------------+

> SELECT count(*), min(r) >= 0, max(r) <= 1, min(f) >= 0 FROM '/tmp/alp_test.parquet';
-- 10000, true, true, true

Only datafusion-cli is expected to compile/run; other tests are not expected to pass.

Are there any user-facing changes?

No (test PR, not intended for merge).

alamb and others added 3 commits August 14, 2026 09:15
Pin arrow/parquet crates to arrow-rs rev 901e69f913bb8a1738009218898959e8f4cd7ec8
via [patch.crates-io] until arrow 60.0.0 is released.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tore 0.14)

- Adapt to the new arrow_schema::Metadata type (was HashMap<String, String>):
  DFSchema::metadata()/ExprSchema::metadata() now return &Metadata, DFSchema
  constructors take impl Into<Metadata>, SchemaFieldMetadata is now an alias
  for Metadata, and check_metadata_with_storage_equal/format_type_and_metadata
  accept Option<&Metadata>
- Add FieldMetadata <-> Metadata conversions and a DFHeapSize impl for Metadata
- Replace deprecated fb_to_schema with try_fb_to_schema
- Rework two benches that mixed DataFusion's rand 0.9 with arrow's rand 0.10
- Add new object_store 0.14 GetResult/ListResult extensions field
- Update map field name expectations for new arrow spec defaults (key/value)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added sql SQL Planner logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) catalog Related to the catalog crate common Related to common crate execution Related to the execution crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate ffi Changes to the ffi crate physical-plan Changes to the physical-plan crate labels Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

catalog Related to the catalog crate common Related to common crate core Core DataFusion crate datasource Changes to the datasource crate execution Related to the execution crate ffi Changes to the ffi crate functions Changes to functions implementation logical-expr Logical plan and expressions optimizer Optimizer rules physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate proto Related to proto crate sql SQL Planner sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant