Skip to content

fix(arrow): preserve INT96 timestamptz values in projections - #3632

Open
sam-1112 wants to merge 7 commits into
apache:mainfrom
sam-1112:fix/int96-timestamptz-projection
Open

sam-1112 wants to merge 7 commits into
apache:mainfrom
sam-1112:fix/int96-timestamptz-projection

Conversation

@sam-1112

@sam-1112 sam-1112 commented Nov 5, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Closes #3611.
Closes #3616.

What changes are included in this PR?

INT96 timestamp coercion preserves a timezone-less Arrow type even when Iceberg requests timestamptz. Projection compatibility checks then exclude the column, causing multi-column reads to silently return NULL.

Coerce confirmed physical INT96 columns to the precision and timezone required by the Iceberg schema, including timezone-only adjustments for timestamptz_ns. Preserve field metadata and nested structures while leaving non-INT96 timestamp types unchanged.

Are these changes tested?

Regression tests compare complete batches and physical projection masks across projection orders, embedded field IDs, name mapping, and positional fallback. Coverage includes genuine NULLs, missing fields and defaults, mixed timestamp encodings, nested structs/lists/maps, and dates outside the nanosecond range.

Validation passed:

  • INT96 tests: 29 passed.
  • Full iceberg library suite: 2,131 passed, 1 ignored.
  • Doctests, all-targets/all-features Clippy, and formatting checks.
  • Public API generation: no snapshot changes.

Also validated alongside #3624 at 30b9c20. Its additional Name Mapping parameter required a one-line adjustment to the mask-test helper in the isolated checkout.

AI Disclosure

OpenAI Codex assisted with implementation and tests. Review should focus on the assumption that full Arrow primitive traversal matches physical Parquet column order, and on UTC schema-hint handling for INT96 timestamptz types.

@sam-1112
sam-1112 marked this pull request as ready for review November 7, 2026 00:25

@rich7420 rich7420 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the patch

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sam-1112 for tracking this down from the Comet report and for the thorough tests. I also ran a filtered scan the way Comet's native Iceberg scan reads these files: no field IDs, a name mapping, a predicate on the INT96 timestamptz column, and row selection on. On main the filter keeps the right rows but ts comes back NULL. With this PR the values are correct.

Comment thread crates/iceberg/src/arrow/int96.rs Outdated
Comment on lines 51 to 66
) -> Option<Arc<ArrowSchema>> {
let mut visitor = Int96CoercionVisitor::new(iceberg_schema);
if !parquet_schema
.columns()
.iter()
.any(|column| column.physical_type() == PhysicalType::INT96)
{
return None;
}
let mut visitor = Int96CoercionVisitor::new(iceberg_schema, parquet_schema);
let coerced = visit_schema(arrow_schema, &mut visitor).ok()?;
if visitor.changed {
if visitor.changed && visitor.columns.len() == 0 {
Some(Arc::new(coerced))
} else {
None
}
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the visitor fails, or doesn't consume every Parquet column, this returns None and the file is read without coercion. That brings back both bugs this function exists to fix. Dates outside ~1677-2262 overflow, and a timestamptz column fails the projection type check and reads as NULL (#3611).

One way to get here today is a file whose embedded ARROW:schema declares a ListView column. arrow-rs reads a Parquet list as ListView when the schema hint asks for it, and visit_schema rejects ListView. I called coerce_int96_timestamps with a ListView<Int32> column ahead of an INT96 column declared timestamptz, and it returned None.

Could this return a Result and fail when the Arrow leaves and Parquet columns don't line up? configure_arrow_reader_metadata already returns Result, so both callers only need a ?. A file like the one above would then fail to read instead of returning NULLs or overflowed values.

Suggested change
) -> Option<Arc<ArrowSchema>> {
let mut visitor = Int96CoercionVisitor::new(iceberg_schema);
if !parquet_schema
.columns()
.iter()
.any(|column| column.physical_type() == PhysicalType::INT96)
{
return None;
}
let mut visitor = Int96CoercionVisitor::new(iceberg_schema, parquet_schema);
let coerced = visit_schema(arrow_schema, &mut visitor).ok()?;
if visitor.changed {
if visitor.changed && visitor.columns.len() == 0 {
Some(Arc::new(coerced))
} else {
None
}
}
) -> Result<Option<Arc<ArrowSchema>>> {
if !parquet_schema
.columns()
.iter()
.any(|column| column.physical_type() == PhysicalType::INT96)
{
return Ok(None);
}
let mut visitor = Int96CoercionVisitor::new(iceberg_schema, parquet_schema);
let coerced = visit_schema(arrow_schema, &mut visitor)?;
if visitor.columns.len() != 0 {
return Err(Error::new(
ErrorKind::Unexpected,
"Arrow schema has fewer primitive leaves than the Parquet schema has columns",
));
}
Ok(visitor.changed.then(|| Arc::new(coerced)))
}

Comment on lines +991 to +1037
fn assert_int96_projection_mask(task: &FileScanTask, expected_indices: &[usize]) {
use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};
use parquet::file::reader::{FileReader, SerializedFileReader};

use super::{ArrowReader, FileScanTaskReader};

let reader = SerializedFileReader::new(File::open(task.data_file_path()).unwrap()).unwrap();
let metadata = ArrowReaderMetadata::try_new(
Arc::new(reader.metadata().clone()),
ArrowReaderOptions::default(),
)
.unwrap();
let missing_field_ids = metadata
.schema()
.fields()
.iter()
.next()
.is_some_and(|field| !field.metadata().contains_key(PARQUET_FIELD_ID_META_KEY));
let use_position_fallback = missing_field_ids && task.name_mapping().is_none();
let metadata = FileScanTaskReader::configure_arrow_reader_metadata(
metadata,
task,
missing_field_ids,
false,
)
.unwrap();
let mask = ArrowReader::get_arrow_projection_mask(
task.project_field_ids(),
task.schema(),
metadata.parquet_schema(),
metadata.schema(),
use_position_fallback,
)
.unwrap();
let actual: Vec<bool> = (0..metadata.parquet_schema().num_columns())
.map(|index| mask.leaf_included(index))
.collect();
let expected: Vec<bool> = (0..metadata.parquet_schema().num_columns())
.map(|index| expected_indices.contains(&index))
.collect();
assert_eq!(
actual,
expected,
"projection {:?}",
task.project_field_ids()
);
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helper copies the reader's field-ID detection and calls get_arrow_projection_mask directly. #3624 changes both: it detects field IDs from the Parquet schema, and it changes the signature of get_arrow_projection_mask. If the copy drifts from the reader, these assertions check a mask the reader no longer builds.

The batch comparisons already catch #3611, because the NULLs show up in the output. Could we drop the mask assertions and keep the batch comparisons?

Comment on lines +3012 to +3213
async fn assert_int96_timestamp_projections(
timestamp_type: PrimitiveType,
field_ids: Int96FieldIds,
with_nulls: bool,
) {
use arrow_array::{TimestampMicrosecondArray, TimestampNanosecondArray, new_null_array};
use parquet::basic::{LogicalType, Repetition, Type as PhysicalType};
use parquet::data_type::{ByteArray, ByteArrayType, Int64Type, Int96, Int96Type};
use parquet::file::writer::SerializedFileWriter;
use parquet::schema::types::Type as SchemaType;

use crate::arrow::schema::UTC_TIME_ZONE;
use crate::arrow::schema_to_arrow_schema;
use crate::spec::Literal;

let schema = Arc::new(
Schema::builder()
.with_fields(vec![
NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(),
NestedField::optional(2, "ts", Type::Primitive(timestamp_type.clone())).into(),
NestedField::required(3, "s", Type::Primitive(PrimitiveType::String)).into(),
NestedField::optional(4, "missing_ts", Type::Primitive(timestamp_type.clone()))
.into(),
NestedField::optional(5, "default_s", Type::Primitive(PrimitiveType::String))
.with_initial_default(Literal::string("default"))
.into(),
])
.build()
.unwrap(),
);
let name_mapping = matches!(field_ids, Int96FieldIds::NameMapping).then(|| {
Arc::new(NameMapping::new(vec![
MappedField::new(Some(1), vec!["id".to_string()], vec![]),
MappedField::new(Some(2), vec!["ts".to_string()], vec![]),
MappedField::new(Some(3), vec!["s".to_string()], vec![]),
]))
});
let with_field_ids = matches!(field_ids, Int96FieldIds::Embedded);
let parquet_schema = SchemaType::group_type_builder("schema")
.with_fields(vec![
Arc::new(
SchemaType::primitive_type_builder("id", PhysicalType::INT64)
.with_repetition(Repetition::REQUIRED)
.with_id(with_field_ids.then_some(1))
.build()
.unwrap(),
),
Arc::new(
SchemaType::primitive_type_builder("ts", PhysicalType::INT96)
.with_repetition(Repetition::OPTIONAL)
.with_id(with_field_ids.then_some(2))
.build()
.unwrap(),
),
Arc::new(
SchemaType::primitive_type_builder("s", PhysicalType::BYTE_ARRAY)
.with_repetition(Repetition::REQUIRED)
.with_logical_type(Some(LogicalType::String))
.with_id(with_field_ids.then_some(3))
.build()
.unwrap(),
),
])
.build()
.unwrap();
let micros = vec![0_i64, 1_000_003, 2_000_006, 3_000_009, 4_000_012];
let nanos: Vec<i64> = micros
.iter()
.map(|micros| {
micros * 1_000
+ i64::from(matches!(
timestamp_type,
PrimitiveType::TimestampNs | PrimitiveType::TimestamptzNs
))
})
.collect();
let ids = vec![0_i64, 1, 2, 3, 4];
let strings = vec!["0", "1", "2", "3", "4"];
let definition_levels: Vec<i16> = (0..5)
.map(|index| i16::from(!with_nulls || index % 2 == 0))
.collect();
let int96_values: Vec<Int96> = nanos
.iter()
.zip(&definition_levels)
.filter(|(_, level)| **level == 1)
.map(|(nanos, _)| {
let mut value = Int96::new();
value.set_data(
*nanos as u32,
(nanos >> 32) as u32,
UNIX_EPOCH_JULIAN as u32,
);
value
})
.collect();
let string_values: Vec<ByteArray> = strings
.iter()
.map(|value| ByteArray::from(*value))
.collect();

let tmp_dir = TempDir::new().unwrap();
let file_path = tmp_dir.path().join("timestamptz_name_mapping.parquet");
let mut writer = SerializedFileWriter::new(
File::create(&file_path).unwrap(),
Arc::new(parquet_schema),
Default::default(),
)
.unwrap();
let mut row_group = writer.next_row_group().unwrap();
let mut column = row_group.next_column().unwrap().unwrap();
column
.typed::<Int64Type>()
.write_batch(&ids, None, None)
.unwrap();
column.close().unwrap();
let mut column = row_group.next_column().unwrap().unwrap();
column
.typed::<Int96Type>()
.write_batch(&int96_values, Some(&definition_levels), None)
.unwrap();
column.close().unwrap();
let mut column = row_group.next_column().unwrap().unwrap();
column
.typed::<ByteArrayType>()
.write_batch(&string_values, None, None)
.unwrap();
column.close().unwrap();
row_group.close().unwrap();
writer.close().unwrap();

let timezone = matches!(
timestamp_type,
PrimitiveType::Timestamptz | PrimitiveType::TimestamptzNs
)
.then_some(UTC_TIME_ZONE);
let expected_timestamps: ArrayRef = match timestamp_type {
PrimitiveType::Timestamp | PrimitiveType::Timestamptz => {
let values: Vec<Option<i64>> = micros
.into_iter()
.zip(&definition_levels)
.map(|(value, level)| (*level == 1).then_some(value))
.collect();
Arc::new(TimestampMicrosecondArray::from(values).with_timezone_opt(timezone))
}
PrimitiveType::TimestampNs | PrimitiveType::TimestamptzNs => {
let values: Vec<Option<i64>> = nanos
.into_iter()
.zip(&definition_levels)
.map(|(value, level)| (*level == 1).then_some(value))
.collect();
Arc::new(TimestampNanosecondArray::from(values).with_timezone_opt(timezone))
}
other => panic!("Expected timestamp type, got {other}"),
};
let missing_timestamps = new_null_array(expected_timestamps.data_type(), 5);
let expected_batch =
RecordBatch::try_new(Arc::new(schema_to_arrow_schema(&schema).unwrap()), vec![
Arc::new(Int64Array::from(ids)),
expected_timestamps,
Arc::new(StringArray::from(strings)),
missing_timestamps,
Arc::new(StringArray::from(vec!["default"; 5])),
])
.unwrap();
let mut actual = Vec::new();
let mut expected = Vec::new();
for field_ids in [
vec![2],
vec![1, 2, 3],
vec![2, 1, 3],
vec![2, 4, 5, 1],
vec![4, 5],
] {
let column_indices: Vec<usize> =
field_ids.iter().map(|id| (*id - 1) as usize).collect();
expected.push(vec![expected_batch.project(&column_indices).unwrap()]);
if field_ids.iter().any(|id| *id <= 3) {
let task = int96_scan_task(
file_path.to_str().unwrap(),
schema.clone(),
field_ids.clone(),
name_mapping.clone(),
);
let physical_indices: Vec<usize> = column_indices
.iter()
.copied()
.filter(|index| *index < 3)
.collect();
assert_int96_projection_mask(&task, &physical_indices);
}
actual.push(
read_int96_batches_with_name_mapping(
file_path.to_str().unwrap(),
schema.clone(),
field_ids,
name_mapping.clone(),
)
.await,
);
}
assert_eq!(actual, expected);
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helper writes another INT96 file by hand next to write_int96_parquet_file. It also mixes in cases that aren't about INT96: a missing field, an initial default, column order, and a projection of only missing fields. The transformer tests already cover defaults and missing fields, for example test_all_four_spec_rules.

write_int96_parquet_file already writes an INT96 ts next to an id, with or without field IDs. Its year-3333 values also cover the overflow. I read that file with read_int96_batches_with_name_mapping, with ts declared timestamptz and projection [1, 2], once with embedded IDs and once with a name mapping. On main both read ts as NULL, and on this branch both match the expected batch.

Could the regression tests use that file? One way is to take the Iceberg timestamp type as a parameter in test_read_int96_timestamps_with_field_ids and test_read_int96_timestamps_without_field_ids, and add a name-mapping case. Then this helper only needs the nanosecond cases, which need values inside the nanosecond range. For that smaller file, parse_message_type (as in test_read_int96_with_mixed_timestamp_types_in_struct) is shorter than the schema builders.

Comment on lines +3368 to +3374
let incompatible_task = int96_scan_task(
file_path.to_str().unwrap(),
schema_with_micros_type(PrimitiveType::Timestamptz),
vec![1, 2, 6],
None,
);
assert_int96_projection_mask(&incompatible_task, &[0, 2, 4]);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assertion checks that an INT64 TIMESTAMP(MICROS,false) column is left out of the read when the table declares it timestamptz, so it reads as NULL. That's the same silent NULL as #3611, for a different encoding.

The spec only says what writers must produce. Iceberg Java returns the values in both Spark readers. The row reader dispatches on TIMESTAMP_MICROS without checking isAdjustedToUTC, and the vectorized reader takes the zone from the Iceberg type.

Could we drop this assertion so the test doesn't lock in that result? test_only_int96_timestamps_are_coerced already shows that non-INT96 columns keep their types.

@sam-1112

sam-1112 commented Nov 9, 2026

Copy link
Copy Markdown
Author

@mbutrovich Thanks for the detailed review! I've addressed all four points:

  • Changed coerce_int96_timestamps to return Result<Option<_>> and propagate errors through both callers. Added tests for ListView traversal failure and mismatched leaf counts.
  • Removed the projection-mask helper and assertions, keeping the full batch comparisons.
  • Reused write_int96_parquet_file for the timestamp and timestamptz cases, including name mapping and year-3333 values. The separate nanosecond fixture now uses parse_message_type, without the missing-field/default cases.
  • Removed the INT64 exclusion assertion. test_only_int96_timestamps_are_coerced still verifies that non-INT96 types remain unchanged.

@sam-1112
sam-1112 requested a review from mbutrovich November 11, 2026 09:46
…ection

# Conflicts:
#	crates/iceberg/src/arrow/reader/pipeline.rs

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sam-1112 for the quick turnaround. The Result change and the slimmer tests look good.

This PR also fixes #3616. Because only physical INT96 columns are coerced now, INT64 TIMESTAMP(NANOS) columns keep their nanosecond type and are no longer read 1000x too large. I ran test_read_int64_nanosecond_timestamps_as_microseconds from #3621 on this branch, and it passes. Could you add Closes #3616 to the description so the issue closes when this merges?

Comment on lines 38 to +39
/// specified by the Iceberg schema (`timestamp` → microsecond, `timestamp_ns` → nanosecond).
/// The Arrow schema must contain all physical columns in file order, before projection.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc comment still covers only the resolution. The timezone is the actual fix for #3611, and why it's needed isn't obvious from the code. Projection converts the file's Arrow type to an Iceberg type and drops the leaf when that type doesn't match the table's. Timestamp(Microsecond, None) converts to timestamp, not timestamptz. Could we say that here?

Suggested change
/// specified by the Iceberg schema (`timestamp` → microsecond, `timestamp_ns` → nanosecond).
/// The Arrow schema must contain all physical columns in file order, before projection.
/// specified by the Iceberg schema (`timestamp` → microsecond, `timestamp_ns` → nanosecond).
/// For `timestamptz` and `timestamptz_ns`, the hint also sets the UTC timezone. Without it,
/// projection sees a `timestamp` column where the table expects `timestamptz`, drops the
/// column, and the read returns NULLs.
/// The Arrow schema must contain all physical columns in file order, before projection.

Comment thread crates/iceberg/src/arrow/int96.rs Outdated
Comment on lines +442 to +459
for (message, expected_message) in [
(
"message schema { optional int96 ts; required int32 id; }",
"Arrow schema has fewer primitive leaves than the Parquet schema has columns",
),
(
"message schema { optional int96 ts; }",
"Missing Parquet column for Arrow primitive",
),
] {
let parquet = parquet_schema(message);
let fields = [
Field::new("ts", DataType::Timestamp(TimeUnit::Nanosecond, None), true)
.with_metadata(field_id_meta(1)),
Field::new("id", DataType::Int32, false).with_metadata(field_id_meta(2)),
];
let leaf_count = if parquet.num_columns() == 2 { 1 } else { 2 };
let arrow_schema = Arc::new(ArrowSchema::new(fields[..leaf_count].to_vec()));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The number of Arrow fields in each case comes from if parquet.num_columns() == 2 { 1 } else { 2 }. A reader has to work that out to see which case is which. Could the table list the Arrow leaf count next to the Parquet message instead?

Suggested change
for (message, expected_message) in [
(
"message schema { optional int96 ts; required int32 id; }",
"Arrow schema has fewer primitive leaves than the Parquet schema has columns",
),
(
"message schema { optional int96 ts; }",
"Missing Parquet column for Arrow primitive",
),
] {
let parquet = parquet_schema(message);
let fields = [
Field::new("ts", DataType::Timestamp(TimeUnit::Nanosecond, None), true)
.with_metadata(field_id_meta(1)),
Field::new("id", DataType::Int32, false).with_metadata(field_id_meta(2)),
];
let leaf_count = if parquet.num_columns() == 2 { 1 } else { 2 };
let arrow_schema = Arc::new(ArrowSchema::new(fields[..leaf_count].to_vec()));
let fields = [
Field::new("ts", DataType::Timestamp(TimeUnit::Nanosecond, None), true)
.with_metadata(field_id_meta(1)),
Field::new("id", DataType::Int32, false).with_metadata(field_id_meta(2)),
];
for (message, arrow_leaves, expected_message) in [
(
"message schema { optional int96 ts; required int32 id; }",
1,
"Arrow schema has fewer primitive leaves than the Parquet schema has columns",
),
(
"message schema { optional int96 ts; }",
2,
"Missing Parquet column for Arrow primitive",
),
] {
let parquet = parquet_schema(message);
let arrow_schema = Arc::new(ArrowSchema::new(fields[..arrow_leaves].to_vec()));


#[tokio::test]
async fn test_read_int96_timestamps_without_field_ids() {
async fn test_read_int96_with_mixed_timestamp_types_in_struct() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every nested INT96 test in this PR embeds field IDs, including this one and the list, map, and struct tests. Spark-migrated files have INT96 inside structs and no field IDs, so the reader resolves them through a name mapping. That's the layout Comet reads.

Could this test take with_field_ids like assert_int96_nanosecond_projections does? Without IDs, it would drop the = N IDs from the message and pass a nested name mapping (data with children micros and legacy) to read_int96_batches_with_name_mapping. I tried that on this branch and the expected batch matches for all three projections.

@sam-1112

sam-1112 commented Nov 17, 2026 •

Copy link
Copy Markdown
Author

Thanks @mbutrovich , I added Closes #3616 to my PR description and addressed all inline comments above. Could you take a look when you have a chance?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

INT64 nanosecond timestamps read 1000x too large since v0.9.1 INT96 timestamptz column silently returns NULL when projected alongside other columns

3 participants