Update to arrow-rs main (arrow 60 pre-release testing) by alamb · Pull Request #24366 · apache/datafusion

alamb · 2026-08-14T13:15:41Z

Which issue does this PR close?

Part of testing for the upcoming arrow 60 release: Release arrow-rs / parquet Major version 60.0.0 (August 2026) arrow-rs#9880

Rationale for this change

Test DataFusion against the latest arrow-rs main ahead of the arrow 60.0.0 release, so we can find and fix any issues before the release is published.

What changes are included in this PR?

Pin all arrow/parquet crates to upstream arrow
Rename the arrow-flight flight-sql-experimental feature to flight-sql (renamed upstream)
Update object_store to 0.14.1 (required by parquet on arrow-rs main)

We will comment inline where upstream changes required updates.

Are these changes tested?

By CI

Are there any user-facing changes?

No

codecov-commenter · 2026-08-14T15:28:15Z

Codecov Report

❌ Patch coverage is 80.68966% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.18%. Comparing base (ec110ce) to head (70bd16a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/common/src/metadata.rs	67.74%	10 Missing ⚠️
datafusion/proto-common/src/from_proto/mod.rs	0.00%	4 Missing and 1 partial ⚠️
datafusion/common/src/dfschema.rs	50.00%	3 Missing ⚠️
datafusion/common/src/heap_size.rs	57.14%	2 Missing and 1 partial ⚠️
datafusion/datasource-arrow/src/file_format.rs	0.00%	2 Missing and 1 partial ⚠️
datafusion/datasource-arrow/src/source.rs	25.00%	2 Missing and 1 partial ⚠️
datafusion/sql/src/statement.rs	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24366      +/-   ##
==========================================
- Coverage   81.19%   81.18%   -0.02%     
==========================================
  Files        1110     1110              
  Lines      388618   388612       -6     
  Branches   388618   388612       -6     
==========================================
- Hits       315531   315487      -44     
- Misses      54507    54534      +27     
- Partials    18580    18591      +11

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alamb · 2026-08-14T15:35:27Z

 use arrow::{
    array::AsArray,
-    datatypes::{DataType, Field},
+    datatypes::{DataType, Field, Metadata},


Arrow 60 introduces a new metadata type:

Use Metadata struct instead of HashMap<String, String> for metadata: Ordered; cheap to clone arrow-rs#10075

alamb · 2026-08-14T15:46:17Z

 arrow-data = { version = "59.2.0", default-features = false }
 arrow-flight = { version = "59.2.0", features = [
-    "flight-sql-experimental",
+    "flight-sql",


The flight-sql-experimental feature was renamed to flight-sql and the old alias has now been removed:

Fix: Converted feature flight-sql-experimental to flight-sql arrow-rs#7546

Remove flight-sql-experimental feature from arrow-flight arrow-rs#10335

alamb · 2026-08-14T15:46:18Z

 memchr = "2.8.1"
 num-traits = { version = "0.2" }
-object_store = { version = "0.13.2", default-features = false }
+object_store = { version = "0.14.1", default-features = false }


parquet on arrow-rs main now requires object_store 0.14:

chore: bump object_store to 0.14.1 arrow-rs#10375

alamb · 2026-08-14T15:46:19Z

 };
 use arrow::ipc::{
-    convert::fb_to_schema,
+    convert::try_fb_to_schema,


fb_to_schema is deprecated in favor of the fallible try_fb_to_schema (same for the datasource-arrow changes):

fix(arrow-ipc): return an error instead of panicking on malformed IPC schema arrow-rs#10647

alamb · 2026-08-14T15:46:19Z

 where
-    T: ArrowPrimitiveType,
-    StandardUniform: Distribution<T::Native>,
+    T: ArrowPrimitiveType<Native = i64>,


Arrow 60 upgraded to rand 0.10, so bench_util's StandardUniform bounds can no longer be stated from DataFusion's rand 0.9:

Upgrade rand to 0.10 arrow-rs#10566

Filed re-export arrow::util rand version, breaking downstreams on a different rand major arrow-rs#10685 upstream

alamb · 2026-08-14T15:46:25Z

 Plan with Metrics
 01)SortExec: TopK(fetch=1), expr=[b@1 DESC], preserve_partitioning=[false], filter=[b@1 IS NULL OR b@1 > bd], metrics=[output_rows=1, output_batches=1, row_replacements=1]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_single_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 IS NULL OR b@1 > bd ], sort_order_for_reorder=[b@1 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@0 > 0 OR b_null_count@0 != row_count@2 AND b_max@1 > bd, required_guarantees=[], metrics=[output_rows=4, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=0, predicate_cache_inner_records=4, predicate_cache_records=4, scan_efficiency_ratio=21.62% (222/1.03 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_single_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 IS NULL OR b@1 > bd ], sort_order_for_reorder=[b@1 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@0 > 0 OR b_null_count@0 != row_count@2 AND b_max@1 > bd, required_guarantees=[], metrics=[output_rows=4, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=0, predicate_cache_inner_records=4, predicate_cache_records=4, scan_efficiency_ratio=21.94% (222/1.01 K)]


The scan_efficiency_ratio denominators shifted because the parquet files written by this test got ~15 bytes smaller: since apache/arrow-rs#9619 float/double statistics use IEEE 754 total order, which is not backwards compatible with old readers' signed comparison, so the writer no longer duplicates min/max into the deprecated Statistics min/max fields for floating point columns.

Verified by writing the same table with arrow 59.2 vs the pinned rev: 1027 bytes (footer thrift 686) vs 1012 bytes (footer thrift 668).

alamb · 2026-08-14T15:51:37Z

 }

+impl DFHeapSize for Metadata {
+    fn heap_size(&self, ctx: &mut DFHeapSizeCtx) -> usize {


Could be simpler with nicer upstream API. Filed a ticket to track:

Expose Metadata's inner Arc (e.g. as_arc / into_arc) arrow-rs#10684

alamb · 2026-08-14T15:54:43Z

-                .into_iter()
-                .collect(),
-        ),
+        Field::new("with_metadata", DataType::UInt64, true)


the new Metadata api is nicer to work with I think 😍

alamb · 2026-08-14T15:57:59Z

        match &mut intersected {
            None => {
-                intersected = Some(metadata.clone());
+                intersected = Some(


This should be nicer with an API like

Add Metadata::retain to filter entries in place arrow-rs#10683

alamb · 2026-08-14T16:01:33Z

        +---------------------+-----------+-----------------+------+------+
-        | 212                 | 0.parquet | 3642            | 0    | 2    |
-        | 212                 | 1.parquet | 3642            | 1    | 2    |
+        | 216                 | 0.parquet | 3620            | "0"  | 2    |


"0" due to

fix: quote ETags per RFC 9110 in InMemory and LocalFileSystem arrow-rs-object-store#770

alamb · 2026-08-14T16:31:35Z

-    /// Note this is not a `HashMap` because `HashMap` does not provide
-    /// implementations for traits like `Debug` and `Hash`.
-    inner: Arc<BTreeMap<String, String>>,
+    inner: Metadata,


Changed the inner field to be Metadata (as it is now the same representation after apache/arrow-rs#10075)

Pin arrow/parquet crates to arrow-rs rev 901e69f913bb8a1738009218898959e8f4cd7ec8 via [patch.crates-io] until arrow 60.0.0 is released. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…tore 0.14, etc) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

alamb mentioned this pull request Aug 14, 2026

Test: read/write ALP encoded parquet files (apache/arrow-rs#9372) #24373

Draft

alamb commented Aug 14, 2026

View reviewed changes

alamb and others added 2 commits August 14, 2026 13:00

Update to arrow-rs main (arrow 60 pre-release)

035eb94

Pin arrow/parquet crates to arrow-rs rev 901e69f913bb8a1738009218898959e8f4cd7ec8 via [patch.crates-io] until arrow 60.0.0 is released. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fix compilation for arrow-rs main (Metadata type, rand 0.10, object_s…

70bd16a

…tore 0.14, etc) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

alamb force-pushed the alamb/update_arrow_60 branch from fd4a74f to 70bd16a Compare August 14, 2026 17:00

Navigation Menu

Uh oh!

Update to arrow-rs main (arrow 60 pre-release testing) - #24366

Update to arrow-rs main (arrow 60 pre-release testing)#24366
alamb wants to merge 2 commits into
apache:mainfrom
alamb:alamb/update_arrow_60

alamb commented Aug 14, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 14, 2026 •

edited

Loading

Uh oh!

alamb Aug 14, 2026

Uh oh!

alamb Aug 14, 2026

Uh oh!

alamb Aug 14, 2026

Uh oh!

alamb Aug 14, 2026

Uh oh!

alamb Aug 14, 2026

Uh oh!

alamb Aug 14, 2026

Uh oh!

alamb Aug 14, 2026

Uh oh!

alamb Aug 14, 2026

Uh oh!

alamb Aug 14, 2026

Uh oh!

alamb Aug 14, 2026

Uh oh!

alamb Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

alamb commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

alamb commented Aug 14, 2026 •

edited

Loading

codecov-commenter commented Aug 14, 2026 •

edited

Loading