Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
* `iceberg_catalog_loader::BoxedCatalogBuilder` has a new required method, `with_runtime`. Types implementing `CatalogBuilder` get it through the blanket impl; types implementing `BoxedCatalogBuilder` directly (for example, a wrapper around another `Box<dyn BoxedCatalogBuilder>`) must implement it, typically by forwarding the runtime to the builder they wrap.
* `Transaction::commit` now returns `ErrorKind::PreconditionFailed` for read-only tables (`Table::readonly()`, e.g. from `StaticTable::into_table` or `TableBuilder::readonly(true)`), including empty transactions, which previously returned the table unchanged. Java treats an empty transaction on a static table as a no-op.
* `ArrowReader` reads nested struct fields as null for Parquet files that have neither field ids nor a name mapping, as iceberg-java does. Earlier versions returned their stored values by position. Tables migrated without field ids need a name mapping (`schema.name-mapping.default`) that covers nested fields to read them.
* `ArrowReader` now fails with `ErrorKind::DataInvalid` instead of returning null when a promoted column holds a value that does not fit the target type, for example a `date` promoted to `timestamp_ns` beyond the year 2262 or a millisecond timestamp beyond about ±292,000 years.

## [v0.10.1] - 2026-07-28

Expand Down
6 changes: 4 additions & 2 deletions crates/iceberg/src/arrow/reader/pipeline.rs
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,11 @@ use super::{
ArrowFileReader, ArrowReader, ParquetReadOptions, add_fallback_field_ids_to_arrow_schema,
apply_name_mapping_to_arrow_schema, coerce_fixed_size_lists, find_leaf_by_field_id,
};
use crate::arrow::build_partition_constant;
use crate::arrow::caching_delete_file_loader::CachingDeleteFileLoader;
use crate::arrow::int96::coerce_int96_timestamps;
use crate::arrow::record_batch_transformer::RecordBatchTransformerBuilder;
use crate::arrow::scan_metrics::{CountingFileRead, ScanMetrics, ScanResult};
use crate::arrow::{build_partition_constant, parquet_timestamp_unit};
use crate::encryption::StandardKeyMetadata;
use crate::error::{Result, invalid_data};
use crate::expr::BoundPredicate;
Expand Down Expand Up @@ -784,6 +784,7 @@ impl FileScanTaskReader {

let physical_type = col_meta.column_type();
let type_length = col_meta.column_descr().type_length();
let timestamp_unit = parquet_timestamp_unit(col_meta.column_descr());

match builder
.get_row_group_column_bloom_filter(rg_idx, col_idx)
Expand All @@ -792,7 +793,8 @@ impl FileScanTaskReader {
Ok(Some(sbbf)) => {
bloom_filters.insert(
field_id,
ColumnBloomFilter::new(sbbf, physical_type, type_length),
ColumnBloomFilter::new(sbbf, physical_type, type_length)
.with_timestamp_unit(timestamp_unit),
);
}
Ok(None) => {}
Expand Down
125 changes: 114 additions & 11 deletions crates/iceberg/src/arrow/reader/predicate_visitor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ use arrow_array::cast::AsArray;
use arrow_array::types::{Float32Type, Float64Type};
use arrow_array::{Array, ArrayRef, BooleanArray, Datum as ArrowDatum, RecordBatch, Scalar};
use arrow_buffer::BooleanBuffer;
use arrow_cast::cast::cast;
use arrow_cast::cast::{CastOptions, cast, cast_with_options};
use arrow_ord::cmp::{eq, gt, gt_eq, lt, lt_eq, neq};
use arrow_schema::{ArrowError, DataType};
use arrow_schema::{ArrowError, DataType, TimeUnit};
use arrow_string::like::starts_with;
use fnv::FnvHashSet;
use parquet::schema::types::SchemaDescriptor;
Expand Down Expand Up @@ -278,6 +278,27 @@ fn project_column(
}
}

/// Like [`project_column`], but widens millisecond timestamps to microseconds so
/// comparisons with microsecond literals keep their precision.
fn project_comparison_column(
batch: &RecordBatch,
column_idx: usize,
) -> std::result::Result<ArrayRef, ArrowError> {
let column = project_column(batch, column_idx)?;

match column.data_type() {
DataType::Timestamp(TimeUnit::Millisecond, tz) => cast_with_options(
&column,
&DataType::Timestamp(TimeUnit::Microsecond, tz.clone()),
&CastOptions {
safe: false,
..Default::default()
},
),
_ => Ok(column),
}
}
Comment on lines +281 to +300

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a heads-up: I think this overlaps with #3540, which is approved. It converts the filter columns to the table schema with a RecordBatchTransformer in get_row_filter, so a millisecond column already reaches the predicate as Timestamp(µs). With that, I think this widening is no longer needed. #3540 also compares milliseconds under the nanosecond types exactly, while this path widens the column to µs and then truncates the ns literal. The two PRs conflict in predicate_visitor.rs and row_filter.rs. If #3540 lands first, I think this commit and its two unit tests could go, and the end-to-end test could stay. What do you think?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. If #3540 lands first, I'll drop the widening commit and its two unit tests and keep the end to end test.


fn compute_is_nan(array: &ArrayRef) -> std::result::Result<BooleanArray, ArrowError> {
// Compute NaN over the contiguous values slice, then fold the null bitmap
// in with a single bitwise AND so that null slots become false.
Expand Down Expand Up @@ -427,7 +448,7 @@ impl BoundPredicateVisitor for PredicateConverter<'_> {
let literal = get_arrow_datum(literal)?;

Ok(Box::new(move |batch| {
let left = project_column(&batch, idx)?;
let left = project_comparison_column(&batch, idx)?;
let literal = try_cast_literal(&literal, left.data_type())?;
lt(&left, literal.as_ref())
}))
Expand All @@ -447,7 +468,7 @@ impl BoundPredicateVisitor for PredicateConverter<'_> {
let literal = get_arrow_datum(literal)?;

Ok(Box::new(move |batch| {
let left = project_column(&batch, idx)?;
let left = project_comparison_column(&batch, idx)?;
let literal = try_cast_literal(&literal, left.data_type())?;
lt_eq(&left, literal.as_ref())
}))
Expand All @@ -467,7 +488,7 @@ impl BoundPredicateVisitor for PredicateConverter<'_> {
let literal = get_arrow_datum(literal)?;

Ok(Box::new(move |batch| {
let left = project_column(&batch, idx)?;
let left = project_comparison_column(&batch, idx)?;
let literal = try_cast_literal(&literal, left.data_type())?;
gt(&left, literal.as_ref())
}))
Expand All @@ -487,7 +508,7 @@ impl BoundPredicateVisitor for PredicateConverter<'_> {
let literal = get_arrow_datum(literal)?;

Ok(Box::new(move |batch| {
let left = project_column(&batch, idx)?;
let left = project_comparison_column(&batch, idx)?;
let literal = try_cast_literal(&literal, left.data_type())?;
gt_eq(&left, literal.as_ref())
}))
Expand All @@ -507,7 +528,7 @@ impl BoundPredicateVisitor for PredicateConverter<'_> {
let literal = get_arrow_datum(literal)?;

Ok(Box::new(move |batch| {
let left = project_column(&batch, idx)?;
let left = project_comparison_column(&batch, idx)?;
let literal = try_cast_literal(&literal, left.data_type())?;
eq(&left, literal.as_ref())
}))
Expand All @@ -527,7 +548,7 @@ impl BoundPredicateVisitor for PredicateConverter<'_> {
let literal = get_arrow_datum(literal)?;

Ok(Box::new(move |batch| {
let left = project_column(&batch, idx)?;
let left = project_comparison_column(&batch, idx)?;
let literal = try_cast_literal(&literal, left.data_type())?;
neq(&left, literal.as_ref())
}))
Expand Down Expand Up @@ -592,7 +613,7 @@ impl BoundPredicateVisitor for PredicateConverter<'_> {

Ok(Box::new(move |batch| {
// update this if arrow ever adds a native is_in kernel
let left = project_column(&batch, idx)?;
let left = project_comparison_column(&batch, idx)?;

let mut acc = constant_bool_array(false, batch.num_rows());
for literal in &literals {
Expand Down Expand Up @@ -622,7 +643,7 @@ impl BoundPredicateVisitor for PredicateConverter<'_> {

Ok(Box::new(move |batch| {
// update this if arrow ever adds a native not_in kernel
let left = project_column(&batch, idx)?;
let left = project_comparison_column(&batch, idx)?;
let mut acc = constant_bool_array(true, batch.num_rows());
for literal in &literals {
let literal = try_cast_literal(literal, left.data_type())?;
Expand Down Expand Up @@ -672,7 +693,9 @@ mod tests {
use super::{CollectFieldIdVisitor, PredicateConverter, constant_bool_array};
use crate::expr::visitors::bound_predicate_visitor::visit;
use crate::expr::{Bind, Predicate, Reference};
use crate::spec::{NestedField, PrimitiveType, Schema, SchemaRef, Type};
use crate::spec::{
Datum, NestedField, PrimitiveLiteral, PrimitiveType, Schema, SchemaRef, Type,
};

fn table_schema_simple() -> SchemaRef {
Arc::new(
Expand Down Expand Up @@ -836,4 +859,84 @@ mod tests {
);
assert!(!result.is_null(2));
}

#[test]
fn test_predicate_converter_compares_millisecond_timestamps_as_micros() {
use arrow_array::TimestampMillisecondArray;

for (primitive_type, tz) in [
(PrimitiveType::Timestamp, None),
(PrimitiveType::Timestamptz, Some("UTC")),
] {
let schema = Arc::new(
Schema::builder()
.with_fields(vec![
NestedField::required(4, "ts", Type::Primitive(primitive_type.clone()))
.into(),
])
.build()
.unwrap(),
);
let column =
TimestampMillisecondArray::from(vec![-1_500, 1_000, 2_000]).with_timezone_opt(tz);
let batch = RecordBatch::try_new(
Arc::new(ArrowSchema::new(vec![Field::new(
"ts",
column.data_type().clone(),
false,
)])),
vec![Arc::new(column)],
)
.unwrap();
let micros = |v| Datum::new(primitive_type.clone(), PrimitiveLiteral::Long(v));

for (predicate, expected) in [
(Reference::new("ts").less_than(micros(1_000_500)), [
true, true, false,
]),
(Reference::new("ts").equal_to(micros(1_000_500)), [
false, false, false,
]),
(Reference::new("ts").equal_to(micros(2_000_000)), [
false, false, true,
]),
] {
let result = apply_predicate_to_batch(predicate, schema.clone(), batch.clone());
assert_eq!(result, BooleanArray::from(expected.to_vec()), "{tz:?}");
}
}
}

#[test]
fn test_predicate_converter_null_checks_skip_millisecond_widening() {
use arrow_array::TimestampMillisecondArray;

let schema = Arc::new(
Schema::builder()
.with_fields(vec![
NestedField::optional(4, "ts", Type::Primitive(PrimitiveType::Timestamp))
.into(),
])
.build()
.unwrap(),
);
let column = TimestampMillisecondArray::from(vec![Some(i64::MAX), None]);
let batch = RecordBatch::try_new(
Arc::new(ArrowSchema::new(vec![Field::new(
"ts",
column.data_type().clone(),
true,
)])),
vec![Arc::new(column)],
)
.unwrap();

for (predicate, expected) in [
(Reference::new("ts").is_null(), [false, true]),
(Reference::new("ts").is_not_null(), [true, false]),
] {
let result = apply_predicate_to_batch(predicate, schema.clone(), batch.clone());
assert_eq!(result, BooleanArray::from(expected.to_vec()));
}
}
}
106 changes: 105 additions & 1 deletion crates/iceberg/src/arrow/reader/projection.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ use std::str::FromStr;
use std::sync::Arc;

use arrow_schema::{
DataType, Field, FieldRef, Fields, Schema as ArrowSchema, SchemaRef as ArrowSchemaRef,
DataType, Field, FieldRef, Fields, Schema as ArrowSchema, SchemaRef as ArrowSchemaRef, TimeUnit,
};
use parquet::arrow::{PARQUET_FIELD_ID_META_KEY, ProjectionMask};
use parquet::schema::types::{SchemaDescriptor, Type as ParquetType};
Expand Down Expand Up @@ -214,6 +214,24 @@ impl ArrowReader {
}),
arrow_schema.metadata().clone(),
);
// A millisecond column whose Iceberg type differs from the table's would otherwise be
// dropped and read as nulls.
for field in projected_arrow_schema.flattened_fields() {
if let DataType::Timestamp(TimeUnit::Millisecond, _) = field.data_type()
&& let Some(field_id) = parquet_field_id(field)
&& let Some(table_field) = iceberg_schema_of_task.field_by_id(field_id)
{
let file_type = arrow_type_to_type(field.data_type())?;
if file_type != *table_field.field_type {
return Err(invalid_data!(
"Cannot read column {} (field id {field_id}) as {}: the file stores {} which reads as {file_type}",
table_field.name,
table_field.field_type,
field.data_type()
));
}
}
}
let iceberg_schema = arrow_schema_to_schema(&projected_arrow_schema)?;

tagged_fields.filter_leaves(|idx, field| {
Expand Down Expand Up @@ -1011,6 +1029,7 @@ mod tests {
use parquet::file::properties::WriterProperties;
use parquet::schema::parser::parse_message_type;
use parquet::schema::types::SchemaDescriptor;
use rstest::rstest;
use tempfile::TempDir;

use crate::arrow::{ArrowReader, ArrowReaderBuilder, ArrowReaderOptions as ReaderOptions};
Expand Down Expand Up @@ -1143,6 +1162,91 @@ message schema {
assert_eq!(mask, ProjectionMask::leaves(&parquet_schema, vec![0]));
}

#[rstest]
#[case::naive_under_timestamp("TIMESTAMP(MILLIS,false)", PrimitiveType::Timestamp, None)]
#[case::utc_under_timestamptz("TIMESTAMP(MILLIS,true)", PrimitiveType::Timestamptz, None)]
#[case::legacy_under_timestamptz("TIMESTAMP_MILLIS", PrimitiveType::Timestamptz, None)]
#[case::naive_under_timestamptz(
"TIMESTAMP(MILLIS,false)",
PrimitiveType::Timestamptz,
Some("Timestamp(ms) which reads as timestamp")
)]
#[case::utc_under_timestamp(
"TIMESTAMP(MILLIS,true)",
PrimitiveType::Timestamp,
Some("Timestamp(ms, \"UTC\") which reads as timestamptz")
)]
#[case::legacy_under_timestamp(
"TIMESTAMP_MILLIS",
PrimitiveType::Timestamp,
Some("Timestamp(ms, \"UTC\") which reads as timestamptz")
)]
#[case::naive_under_timestamp_ns(
"TIMESTAMP(MILLIS,false)",
PrimitiveType::TimestampNs,
Some("Timestamp(ms) which reads as timestamp")
)]
#[case::naive_under_timestamptz_ns(
"TIMESTAMP(MILLIS,false)",
PrimitiveType::TimestamptzNs,
Some("Timestamp(ms) which reads as timestamp")
)]
#[case::utc_under_timestamp_ns(
"TIMESTAMP(MILLIS,true)",
PrimitiveType::TimestampNs,
Some("Timestamp(ms, \"UTC\") which reads as timestamptz")
)]
#[case::utc_under_timestamptz_ns(
"TIMESTAMP(MILLIS,true)",
PrimitiveType::TimestamptzNs,
Some("Timestamp(ms, \"UTC\") which reads as timestamptz")
)]
fn test_arrow_projection_mask_millisecond_column_must_match_table_type(
#[case] declared: &str,
#[case] table_type: PrimitiveType,
#[case] rejected: Option<&str>,
) {
let parquet_schema = SchemaDescriptor::new(Arc::new(
parse_message_type(&format!(
"message schema {{ required int32 id = 1; optional int64 ts ({declared}) = 2; }}"
))
.unwrap(),
));
let arrow_schema = Arc::new(parquet_to_arrow_schema(&parquet_schema, None).unwrap());
let schema = Schema::builder()
.with_fields(vec![
NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(),
NestedField::optional(2, "ts", Type::Primitive(table_type.clone())).into(),
])
.build()
.unwrap();

let result = ArrowReader::get_arrow_projection_mask(
&[1, 2],
&schema,
&parquet_schema,
&arrow_schema,
false,
);

match rejected {
None => assert_eq!(
result.unwrap(),
ProjectionMask::leaves(&parquet_schema, vec![0, 1])
),
Some(file_type) => {
let err = result.unwrap_err();
assert_eq!(err.kind(), ErrorKind::DataInvalid);
assert_eq!(
err.to_string(),
format!(
"DataInvalid => Cannot read column ts (field id 2) as {table_type}: the file stores {file_type}"
)
);
}
}
}

fn field_id_meta(id: i32) -> HashMap<String, String> {
HashMap::from([(PARQUET_FIELD_ID_META_KEY.to_string(), id.to_string())])
}
Expand Down
Loading
Loading