Skip to content

Update to arrow-rs main (arrow 60 pre-release testing) #24366

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
alamb wants to merge 2 commits into
base: main
Choose a base branch
from
+478 −340
Draft

Update to arrow-rs main (arrow 60 pre-release testing) #24366

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
035eb94
Update to arrow-rs main (arrow 60 pre-release)
alamb Aug 14, 2026
70bd16a
Fix compilation for arrow-rs main (Metadata type, rand 0.10, object_s…
alamb Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
225 changes: 170 additions & 55 deletions Cargo.lock
View file
Open in desktop

Large diffs are not rendered by default.

21 changes: 19 additions & 2 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ arrow-avro = { version = "59.2.0", default-features = false, features = [
arrow-buffer = { version = "59.2.0", default-features = false }
arrow-data = { version = "59.2.0", default-features = false }
arrow-flight = { version = "59.2.0", features = [
"flight-sql-experimental",
"flight-sql",

@alamb alamb Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flight-sql-experimental feature was renamed to flight-sql and the old alias has now been removed:

] }
# Both codecs are required here to make sure that code paths like
# file-spilling have access to all compression codecs.
Expand Down Expand Up @@ -176,7 +176,7 @@ liblzma = { version = "0.4.6", features = ["static"] }
log = "^0.4"
memchr = "2.8.1"
num-traits = { version = "0.2" }
object_store = { version = "0.13.2", default-features = false }
object_store = { version = "0.14.1", default-features = false }

@alamb alamb Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parquet on arrow-rs main now requires object_store 0.14:

parking_lot = "0.12"
parquet = { version = "59.2.0", default-features = false, features = [
"arrow",
Expand Down Expand Up @@ -359,3 +359,20 @@ debug = false
debug-assertions = false
strip = "debuginfo"
incremental = false

## Temporary arrow-rs patch until 60.0.0 is released

[patch.crates-io]
arrow = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
arrow-array = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
arrow-avro = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
arrow-buffer = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
arrow-cast = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
arrow-data = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
arrow-ipc = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
arrow-schema = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
arrow-select = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
arrow-string = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
arrow-ord = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
arrow-flight = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
parquet = { git = "https://github.com/apache/arrow-rs.git", rev = "901e69f913bb8a1738009218898959e8f4cd7ec8" }
20 changes: 10 additions & 10 deletions datafusion-cli/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -662,9 +662,9 @@ mod tests {
+-----------------------------------+-----------------+---------------------+------+------------------+
| filename | file_size_bytes | metadata_size_bytes | hits | extra |
+-----------------------------------+-----------------+---------------------+------+------------------+
| alltypes_plain.parquet | 1851 | 8794 | 1 | page_index=false |
| alltypes_tiny_pages.parquet | 454233 | 268970 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 1331 | 1 | page_index=false |
| alltypes_plain.parquet | 1851 | 8970 | 1 | page_index=false |

@alamb alamb Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parquet metadata heap sizes shifted slightly:

| alltypes_tiny_pages.parquet | 454233 | 269562 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 1347 | 1 | page_index=false |
+-----------------------------------+-----------------+---------------------+------+------------------+
");

Expand Down Expand Up @@ -693,9 +693,9 @@ mod tests {
+-----------------------------------+-----------------+---------------------+------+------------------+
| filename | file_size_bytes | metadata_size_bytes | hits | extra |
+-----------------------------------+-----------------+---------------------+------+------------------+
| alltypes_plain.parquet | 1851 | 8794 | 4 | page_index=false |
| alltypes_tiny_pages.parquet | 454233 | 268970 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 1331 | 2 | page_index=false |
| alltypes_plain.parquet | 1851 | 8970 | 4 | page_index=false |
| alltypes_tiny_pages.parquet | 454233 | 269562 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 1347 | 2 | page_index=false |
+-----------------------------------+-----------------+---------------------+------+------------------+
");

Expand Down Expand Up @@ -862,14 +862,14 @@ mod tests {
])?
.sort(vec![col("filename").sort(true, false)])?;
let rbs = df.collect().await?;
assert_snapshot!(batches_to_string(&rbs),@r"
assert_snapshot!(batches_to_string(&rbs),@r#"
+---------------------+-----------+-----------------+------+------+
| metadata_size_bytes | filename | file_size_bytes | etag | hits |
+---------------------+-----------+-----------------+------+------+
| 212 | 0.parquet | 3642 | 0 | 2 |
| 212 | 1.parquet | 3642 | 1 | 2 |
| 216 | 0.parquet | 3620 | "0" | 2 |

@alamb alamb Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"0" due to

| 216 | 1.parquet | 3620 | "1" | 2 |
+---------------------+-----------+-----------------+------+------+
");
"#);

Ok(())
}
Expand Down
4 changes: 2 additions & 2 deletions datafusion/catalog-listing/src/helpers.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ use datafusion_expr::{BinaryExpr, Operator, lit, utils};

use arrow::{
array::AsArray,
datatypes::{DataType, Field},
datatypes::{DataType, Field, Metadata},

@alamb alamb Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow 60 introduces a new metadata type:

record_batch::RecordBatch,
};
use datafusion_expr::execution_props::ExecutionProps;
Expand Down Expand Up @@ -423,7 +423,7 @@ pub async fn pruned_partition_list<'a>(
.iter()
.map(|(n, d)| Field::new(n, d.clone(), true))
.collect(),
Default::default(),
Metadata::new(),
)?;

Ok(objects
Expand Down
4 changes: 2 additions & 2 deletions datafusion/catalog-listing/src/table.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ use crate::helpers::{
expr_applicable_for_cols, filter_partitioned_file, pruned_partition_list,
};
use crate::{ListingOptions, ListingTableConfig};
use arrow::datatypes::{Field, Schema, SchemaBuilder, SchemaRef};
use arrow::datatypes::{Field, Metadata, Schema, SchemaBuilder, SchemaRef};
use async_trait::async_trait;
use datafusion_catalog::{ScanArgs, ScanResult, Session, TableProvider};
use datafusion_common::stats::Precision;
Expand Down Expand Up @@ -1041,7 +1041,7 @@ impl ListingTable {
.iter()
.map(|(name, data_type)| Field::new(name, data_type.clone(), true))
.collect(),
Default::default(),
Metadata::new(),
)?;

file_groups
Expand Down
17 changes: 9 additions & 8 deletions datafusion/common/src/dfschema.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
//! DFSchema is an extended schema struct that DataFusion uses to provide support for
//! fields with optional relation names.

use std::collections::{BTreeSet, HashMap, HashSet};
use std::collections::{BTreeSet, HashSet};
use std::fmt::{Display, Formatter};
use std::hash::Hash;
use std::sync::{Arc, LazyLock};
Expand All @@ -31,7 +31,7 @@ use crate::{

use arrow::compute::can_cast_types;
use arrow::datatypes::{
DataType, Field, FieldRef, Fields, Schema, SchemaBuilder, SchemaRef,
DataType, Field, FieldRef, Fields, Metadata, Schema, SchemaBuilder, SchemaRef,
};

/// A reference-counted reference to a [DFSchema].
Expand Down Expand Up @@ -153,7 +153,7 @@ impl DFSchema {
/// Create a `DFSchema` from an Arrow schema where all the fields have a given qualifier
pub fn new_with_metadata(
qualified_fields: Vec<(Option<TableReference>, Arc<Field>)>,
metadata: HashMap<String, String>,
metadata: impl Into<Metadata>,
) -> Result<Self> {
let (qualifiers, fields): (Vec<Option<TableReference>>, Vec<Arc<Field>>) =
qualified_fields.into_iter().unzip();
Expand All @@ -172,7 +172,7 @@ impl DFSchema {
/// Create a new `DFSchema` from a list of Arrow [Field]s
pub fn from_unqualified_fields(
fields: Fields,
metadata: HashMap<String, String>,
metadata: impl Into<Metadata>,
) -> Result<Self> {
let field_count = fields.len();
let schema = Arc::new(Schema::new_with_metadata(fields, metadata));
Expand Down Expand Up @@ -864,7 +864,7 @@ impl DFSchema {
}

/// Get metadata of this schema
pub fn metadata(&self) -> &HashMap<String, String> {
pub fn metadata(&self) -> &Metadata {
&self.inner.metadata
}

Expand Down Expand Up @@ -1179,7 +1179,7 @@ impl ToDFSchema for Vec<Field> {
let field_count = self.len();
let schema = Schema {
fields: self.into(),
metadata: HashMap::new(),
metadata: Metadata::new(),
};
let dfschema = DFSchema {
inner: schema.into(),
Expand Down Expand Up @@ -1221,7 +1221,7 @@ pub trait ExprSchema: std::fmt::Debug {
}

/// Returns the column's optional metadata.
fn metadata(&self, col: &Column) -> Result<&HashMap<String, String>> {
fn metadata(&self, col: &Column) -> Result<&Metadata> {
Ok(self.field_from_column(col)?.metadata())
}

Expand All @@ -1245,7 +1245,7 @@ impl<P: AsRef<DFSchema> + std::fmt::Debug> ExprSchema for P {
self.as_ref().data_type(col)
}

fn metadata(&self, col: &Column) -> Result<&HashMap<String, String>> {
fn metadata(&self, col: &Column) -> Result<&Metadata> {
ExprSchema::metadata(self.as_ref(), col)
}

Expand Down Expand Up @@ -1379,6 +1379,7 @@ pub fn qualified_name(qualifier: Option<&TableReference>, name: &str) -> String
#[cfg(test)]
mod tests {
use crate::assert_contains;
use std::collections::HashMap;

use super::*;

Expand Down
15 changes: 14 additions & 1 deletion datafusion/common/src/heap_size.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ use arrow::array::{
};
use arrow::datatypes::{
DataType, Field, Fields, IntervalDayTime, IntervalMonthDayNano, IntervalUnit,
TimeUnit, UnionFields, UnionMode, i256,
Metadata, TimeUnit, UnionFields, UnionMode, i256,
};
use chrono::{DateTime, Utc};
use half::f16;
Expand Down Expand Up @@ -396,6 +396,19 @@ impl DFHeapSize for UnionFields {
}
}

impl DFHeapSize for Metadata {
fn heap_size(&self, ctx: &mut DFHeapSizeCtx) -> usize {

@alamb alamb Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be simpler with nicer upstream API. Filed a ticket to track:

// `Metadata` does not expose its underlying reference-counted map, so
// this approximates the `BTreeMap` entries' sizes and cannot dedupe
// instances that share the same allocation.
self.iter()
.map(|(k, v)| {
size_of::<(String, String)>() + k.heap_size(ctx) + v.heap_size(ctx)
})
.sum()
}
}

impl DFHeapSize for Field {
fn heap_size(&self, ctx: &mut DFHeapSizeCtx) -> usize {
self.name().heap_size(ctx)
Expand Down
Loading