Repository navigation
Conversation
rich7420
left a comment
There was a problem hiding this comment.
LGTM, thanks for the patch
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @sam-1112 for tracking this down from the Comet report and for the thorough tests. I also ran a filtered scan the way Comet's native Iceberg scan reads these files: no field IDs, a name mapping, a predicate on the INT96 timestamptz column, and row selection on. On main the filter keeps the right rows but ts comes back NULL. With this PR the values are correct.
| ) -> Option<Arc<ArrowSchema>> { | ||
| let mut visitor = Int96CoercionVisitor::new(iceberg_schema); | ||
| if !parquet_schema | ||
| .columns() | ||
| .iter() | ||
| .any(|column| column.physical_type() == PhysicalType::INT96) | ||
| { | ||
| return None; | ||
| } | ||
| let mut visitor = Int96CoercionVisitor::new(iceberg_schema, parquet_schema); | ||
| let coerced = visit_schema(arrow_schema, &mut visitor).ok()?; | ||
| if visitor.changed { | ||
| if visitor.changed && visitor.columns.len() == 0 { | ||
| Some(Arc::new(coerced)) | ||
| } else { | ||
| None | ||
| } | ||
| } |
There was a problem hiding this comment.
If the visitor fails, or doesn't consume every Parquet column, this returns None and the file is read without coercion. That brings back both bugs this function exists to fix. Dates outside ~1677-2262 overflow, and a timestamptz column fails the projection type check and reads as NULL (#3611).
One way to get here today is a file whose embedded ARROW:schema declares a ListView column. arrow-rs reads a Parquet list as ListView when the schema hint asks for it, and visit_schema rejects ListView. I called coerce_int96_timestamps with a ListView<Int32> column ahead of an INT96 column declared timestamptz, and it returned None.
Could this return a Result and fail when the Arrow leaves and Parquet columns don't line up? configure_arrow_reader_metadata already returns Result, so both callers only need a ?. A file like the one above would then fail to read instead of returning NULLs or overflowed values.
| ) -> Option<Arc<ArrowSchema>> { | |
| let mut visitor = Int96CoercionVisitor::new(iceberg_schema); | |
| if !parquet_schema | |
| .columns() | |
| .iter() | |
| .any(|column| column.physical_type() == PhysicalType::INT96) | |
| { | |
| return None; | |
| } | |
| let mut visitor = Int96CoercionVisitor::new(iceberg_schema, parquet_schema); | |
| let coerced = visit_schema(arrow_schema, &mut visitor).ok()?; | |
| if visitor.changed { | |
| if visitor.changed && visitor.columns.len() == 0 { | |
| Some(Arc::new(coerced)) | |
| } else { | |
| None | |
| } | |
| } | |
| ) -> Result<Option<Arc<ArrowSchema>>> { | |
| if !parquet_schema | |
| .columns() | |
| .iter() | |
| .any(|column| column.physical_type() == PhysicalType::INT96) | |
| { | |
| return Ok(None); | |
| } | |
| let mut visitor = Int96CoercionVisitor::new(iceberg_schema, parquet_schema); | |
| let coerced = visit_schema(arrow_schema, &mut visitor)?; | |
| if visitor.columns.len() != 0 { | |
| return Err(Error::new( | |
| ErrorKind::Unexpected, | |
| "Arrow schema has fewer primitive leaves than the Parquet schema has columns", | |
| )); | |
| } | |
| Ok(visitor.changed.then(|| Arc::new(coerced))) | |
| } |
| fn assert_int96_projection_mask(task: &FileScanTask, expected_indices: &[usize]) { | ||
| use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions}; | ||
| use parquet::file::reader::{FileReader, SerializedFileReader}; | ||
|
|
||
| use super::{ArrowReader, FileScanTaskReader}; | ||
|
|
||
| let reader = SerializedFileReader::new(File::open(task.data_file_path()).unwrap()).unwrap(); | ||
| let metadata = ArrowReaderMetadata::try_new( | ||
| Arc::new(reader.metadata().clone()), | ||
| ArrowReaderOptions::default(), | ||
| ) | ||
| .unwrap(); | ||
| let missing_field_ids = metadata | ||
| .schema() | ||
| .fields() | ||
| .iter() | ||
| .next() | ||
| .is_some_and(|field| !field.metadata().contains_key(PARQUET_FIELD_ID_META_KEY)); | ||
| let use_position_fallback = missing_field_ids && task.name_mapping().is_none(); | ||
| let metadata = FileScanTaskReader::configure_arrow_reader_metadata( | ||
| metadata, | ||
| task, | ||
| missing_field_ids, | ||
| false, | ||
| ) | ||
| .unwrap(); | ||
| let mask = ArrowReader::get_arrow_projection_mask( | ||
| task.project_field_ids(), | ||
| task.schema(), | ||
| metadata.parquet_schema(), | ||
| metadata.schema(), | ||
| use_position_fallback, | ||
| ) | ||
| .unwrap(); | ||
| let actual: Vec<bool> = (0..metadata.parquet_schema().num_columns()) | ||
| .map(|index| mask.leaf_included(index)) | ||
| .collect(); | ||
| let expected: Vec<bool> = (0..metadata.parquet_schema().num_columns()) | ||
| .map(|index| expected_indices.contains(&index)) | ||
| .collect(); | ||
| assert_eq!( | ||
| actual, | ||
| expected, | ||
| "projection {:?}", | ||
| task.project_field_ids() | ||
| ); | ||
| } |
There was a problem hiding this comment.
This helper copies the reader's field-ID detection and calls get_arrow_projection_mask directly. #3624 changes both: it detects field IDs from the Parquet schema, and it changes the signature of get_arrow_projection_mask. If the copy drifts from the reader, these assertions check a mask the reader no longer builds.
The batch comparisons already catch #3611, because the NULLs show up in the output. Could we drop the mask assertions and keep the batch comparisons?
| async fn assert_int96_timestamp_projections( | ||
| timestamp_type: PrimitiveType, | ||
| field_ids: Int96FieldIds, | ||
| with_nulls: bool, | ||
| ) { | ||
| use arrow_array::{TimestampMicrosecondArray, TimestampNanosecondArray, new_null_array}; | ||
| use parquet::basic::{LogicalType, Repetition, Type as PhysicalType}; | ||
| use parquet::data_type::{ByteArray, ByteArrayType, Int64Type, Int96, Int96Type}; | ||
| use parquet::file::writer::SerializedFileWriter; | ||
| use parquet::schema::types::Type as SchemaType; | ||
|
|
||
| use crate::arrow::schema::UTC_TIME_ZONE; | ||
| use crate::arrow::schema_to_arrow_schema; | ||
| use crate::spec::Literal; | ||
|
|
||
| let schema = Arc::new( | ||
| Schema::builder() | ||
| .with_fields(vec![ | ||
| NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(), | ||
| NestedField::optional(2, "ts", Type::Primitive(timestamp_type.clone())).into(), | ||
| NestedField::required(3, "s", Type::Primitive(PrimitiveType::String)).into(), | ||
| NestedField::optional(4, "missing_ts", Type::Primitive(timestamp_type.clone())) | ||
| .into(), | ||
| NestedField::optional(5, "default_s", Type::Primitive(PrimitiveType::String)) | ||
| .with_initial_default(Literal::string("default")) | ||
| .into(), | ||
| ]) | ||
| .build() | ||
| .unwrap(), | ||
| ); | ||
| let name_mapping = matches!(field_ids, Int96FieldIds::NameMapping).then(|| { | ||
| Arc::new(NameMapping::new(vec![ | ||
| MappedField::new(Some(1), vec!["id".to_string()], vec![]), | ||
| MappedField::new(Some(2), vec!["ts".to_string()], vec![]), | ||
| MappedField::new(Some(3), vec!["s".to_string()], vec![]), | ||
| ])) | ||
| }); | ||
| let with_field_ids = matches!(field_ids, Int96FieldIds::Embedded); | ||
| let parquet_schema = SchemaType::group_type_builder("schema") | ||
| .with_fields(vec![ | ||
| Arc::new( | ||
| SchemaType::primitive_type_builder("id", PhysicalType::INT64) | ||
| .with_repetition(Repetition::REQUIRED) | ||
| .with_id(with_field_ids.then_some(1)) | ||
| .build() | ||
| .unwrap(), | ||
| ), | ||
| Arc::new( | ||
| SchemaType::primitive_type_builder("ts", PhysicalType::INT96) | ||
| .with_repetition(Repetition::OPTIONAL) | ||
| .with_id(with_field_ids.then_some(2)) | ||
| .build() | ||
| .unwrap(), | ||
| ), | ||
| Arc::new( | ||
| SchemaType::primitive_type_builder("s", PhysicalType::BYTE_ARRAY) | ||
| .with_repetition(Repetition::REQUIRED) | ||
| .with_logical_type(Some(LogicalType::String)) | ||
| .with_id(with_field_ids.then_some(3)) | ||
| .build() | ||
| .unwrap(), | ||
| ), | ||
| ]) | ||
| .build() | ||
| .unwrap(); | ||
| let micros = vec![0_i64, 1_000_003, 2_000_006, 3_000_009, 4_000_012]; | ||
| let nanos: Vec<i64> = micros | ||
| .iter() | ||
| .map(|micros| { | ||
| micros * 1_000 | ||
| + i64::from(matches!( | ||
| timestamp_type, | ||
| PrimitiveType::TimestampNs | PrimitiveType::TimestamptzNs | ||
| )) | ||
| }) | ||
| .collect(); | ||
| let ids = vec![0_i64, 1, 2, 3, 4]; | ||
| let strings = vec!["0", "1", "2", "3", "4"]; | ||
| let definition_levels: Vec<i16> = (0..5) | ||
| .map(|index| i16::from(!with_nulls || index % 2 == 0)) | ||
| .collect(); | ||
| let int96_values: Vec<Int96> = nanos | ||
| .iter() | ||
| .zip(&definition_levels) | ||
| .filter(|(_, level)| **level == 1) | ||
| .map(|(nanos, _)| { | ||
| let mut value = Int96::new(); | ||
| value.set_data( | ||
| *nanos as u32, | ||
| (nanos >> 32) as u32, | ||
| UNIX_EPOCH_JULIAN as u32, | ||
| ); | ||
| value | ||
| }) | ||
| .collect(); | ||
| let string_values: Vec<ByteArray> = strings | ||
| .iter() | ||
| .map(|value| ByteArray::from(*value)) | ||
| .collect(); | ||
|
|
||
| let tmp_dir = TempDir::new().unwrap(); | ||
| let file_path = tmp_dir.path().join("timestamptz_name_mapping.parquet"); | ||
| let mut writer = SerializedFileWriter::new( | ||
| File::create(&file_path).unwrap(), | ||
| Arc::new(parquet_schema), | ||
| Default::default(), | ||
| ) | ||
| .unwrap(); | ||
| let mut row_group = writer.next_row_group().unwrap(); | ||
| let mut column = row_group.next_column().unwrap().unwrap(); | ||
| column | ||
| .typed::<Int64Type>() | ||
| .write_batch(&ids, None, None) | ||
| .unwrap(); | ||
| column.close().unwrap(); | ||
| let mut column = row_group.next_column().unwrap().unwrap(); | ||
| column | ||
| .typed::<Int96Type>() | ||
| .write_batch(&int96_values, Some(&definition_levels), None) | ||
| .unwrap(); | ||
| column.close().unwrap(); | ||
| let mut column = row_group.next_column().unwrap().unwrap(); | ||
| column | ||
| .typed::<ByteArrayType>() | ||
| .write_batch(&string_values, None, None) | ||
| .unwrap(); | ||
| column.close().unwrap(); | ||
| row_group.close().unwrap(); | ||
| writer.close().unwrap(); | ||
|
|
||
| let timezone = matches!( | ||
| timestamp_type, | ||
| PrimitiveType::Timestamptz | PrimitiveType::TimestamptzNs | ||
| ) | ||
| .then_some(UTC_TIME_ZONE); | ||
| let expected_timestamps: ArrayRef = match timestamp_type { | ||
| PrimitiveType::Timestamp | PrimitiveType::Timestamptz => { | ||
| let values: Vec<Option<i64>> = micros | ||
| .into_iter() | ||
| .zip(&definition_levels) | ||
| .map(|(value, level)| (*level == 1).then_some(value)) | ||
| .collect(); | ||
| Arc::new(TimestampMicrosecondArray::from(values).with_timezone_opt(timezone)) | ||
| } | ||
| PrimitiveType::TimestampNs | PrimitiveType::TimestamptzNs => { | ||
| let values: Vec<Option<i64>> = nanos | ||
| .into_iter() | ||
| .zip(&definition_levels) | ||
| .map(|(value, level)| (*level == 1).then_some(value)) | ||
| .collect(); | ||
| Arc::new(TimestampNanosecondArray::from(values).with_timezone_opt(timezone)) | ||
| } | ||
| other => panic!("Expected timestamp type, got {other}"), | ||
| }; | ||
| let missing_timestamps = new_null_array(expected_timestamps.data_type(), 5); | ||
| let expected_batch = | ||
| RecordBatch::try_new(Arc::new(schema_to_arrow_schema(&schema).unwrap()), vec![ | ||
| Arc::new(Int64Array::from(ids)), | ||
| expected_timestamps, | ||
| Arc::new(StringArray::from(strings)), | ||
| missing_timestamps, | ||
| Arc::new(StringArray::from(vec!["default"; 5])), | ||
| ]) | ||
| .unwrap(); | ||
| let mut actual = Vec::new(); | ||
| let mut expected = Vec::new(); | ||
| for field_ids in [ | ||
| vec![2], | ||
| vec![1, 2, 3], | ||
| vec![2, 1, 3], | ||
| vec![2, 4, 5, 1], | ||
| vec![4, 5], | ||
| ] { | ||
| let column_indices: Vec<usize> = | ||
| field_ids.iter().map(|id| (*id - 1) as usize).collect(); | ||
| expected.push(vec![expected_batch.project(&column_indices).unwrap()]); | ||
| if field_ids.iter().any(|id| *id <= 3) { | ||
| let task = int96_scan_task( | ||
| file_path.to_str().unwrap(), | ||
| schema.clone(), | ||
| field_ids.clone(), | ||
| name_mapping.clone(), | ||
| ); | ||
| let physical_indices: Vec<usize> = column_indices | ||
| .iter() | ||
| .copied() | ||
| .filter(|index| *index < 3) | ||
| .collect(); | ||
| assert_int96_projection_mask(&task, &physical_indices); | ||
| } | ||
| actual.push( | ||
| read_int96_batches_with_name_mapping( | ||
| file_path.to_str().unwrap(), | ||
| schema.clone(), | ||
| field_ids, | ||
| name_mapping.clone(), | ||
| ) | ||
| .await, | ||
| ); | ||
| } | ||
| assert_eq!(actual, expected); | ||
| } |
There was a problem hiding this comment.
This helper writes another INT96 file by hand next to write_int96_parquet_file. It also mixes in cases that aren't about INT96: a missing field, an initial default, column order, and a projection of only missing fields. The transformer tests already cover defaults and missing fields, for example test_all_four_spec_rules.
write_int96_parquet_file already writes an INT96 ts next to an id, with or without field IDs. Its year-3333 values also cover the overflow. I read that file with read_int96_batches_with_name_mapping, with ts declared timestamptz and projection [1, 2], once with embedded IDs and once with a name mapping. On main both read ts as NULL, and on this branch both match the expected batch.
Could the regression tests use that file? One way is to take the Iceberg timestamp type as a parameter in test_read_int96_timestamps_with_field_ids and test_read_int96_timestamps_without_field_ids, and add a name-mapping case. Then this helper only needs the nanosecond cases, which need values inside the nanosecond range. For that smaller file, parse_message_type (as in test_read_int96_with_mixed_timestamp_types_in_struct) is shorter than the schema builders.
| let incompatible_task = int96_scan_task( | ||
| file_path.to_str().unwrap(), | ||
| schema_with_micros_type(PrimitiveType::Timestamptz), | ||
| vec![1, 2, 6], | ||
| None, | ||
| ); | ||
| assert_int96_projection_mask(&incompatible_task, &[0, 2, 4]); |
There was a problem hiding this comment.
This assertion checks that an INT64 TIMESTAMP(MICROS,false) column is left out of the read when the table declares it timestamptz, so it reads as NULL. That's the same silent NULL as #3611, for a different encoding.
The spec only says what writers must produce. Iceberg Java returns the values in both Spark readers. The row reader dispatches on TIMESTAMP_MICROS without checking isAdjustedToUTC, and the vectorized reader takes the zone from the Iceberg type.
Could we drop this assertion so the test doesn't lock in that result? test_only_int96_timestamps_are_coerced already shows that non-INT96 columns keep their types.
|
@mbutrovich Thanks for the detailed review! I've addressed all four points:
|
…ection # Conflicts: # crates/iceberg/src/arrow/reader/pipeline.rs
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @sam-1112 for the quick turnaround. The Result change and the slimmer tests look good.
This PR also fixes #3616. Because only physical INT96 columns are coerced now, INT64 TIMESTAMP(NANOS) columns keep their nanosecond type and are no longer read 1000x too large. I ran test_read_int64_nanosecond_timestamps_as_microseconds from #3621 on this branch, and it passes. Could you add Closes #3616 to the description so the issue closes when this merges?
| /// specified by the Iceberg schema (`timestamp` → microsecond, `timestamp_ns` → nanosecond). | ||
| /// The Arrow schema must contain all physical columns in file order, before projection. |
There was a problem hiding this comment.
This doc comment still covers only the resolution. The timezone is the actual fix for #3611, and why it's needed isn't obvious from the code. Projection converts the file's Arrow type to an Iceberg type and drops the leaf when that type doesn't match the table's. Timestamp(Microsecond, None) converts to timestamp, not timestamptz. Could we say that here?
| /// specified by the Iceberg schema (`timestamp` → microsecond, `timestamp_ns` → nanosecond). | |
| /// The Arrow schema must contain all physical columns in file order, before projection. | |
| /// specified by the Iceberg schema (`timestamp` → microsecond, `timestamp_ns` → nanosecond). | |
| /// For `timestamptz` and `timestamptz_ns`, the hint also sets the UTC timezone. Without it, | |
| /// projection sees a `timestamp` column where the table expects `timestamptz`, drops the | |
| /// column, and the read returns NULLs. | |
| /// The Arrow schema must contain all physical columns in file order, before projection. |
| for (message, expected_message) in [ | ||
| ( | ||
| "message schema { optional int96 ts; required int32 id; }", | ||
| "Arrow schema has fewer primitive leaves than the Parquet schema has columns", | ||
| ), | ||
| ( | ||
| "message schema { optional int96 ts; }", | ||
| "Missing Parquet column for Arrow primitive", | ||
| ), | ||
| ] { | ||
| let parquet = parquet_schema(message); | ||
| let fields = [ | ||
| Field::new("ts", DataType::Timestamp(TimeUnit::Nanosecond, None), true) | ||
| .with_metadata(field_id_meta(1)), | ||
| Field::new("id", DataType::Int32, false).with_metadata(field_id_meta(2)), | ||
| ]; | ||
| let leaf_count = if parquet.num_columns() == 2 { 1 } else { 2 }; | ||
| let arrow_schema = Arc::new(ArrowSchema::new(fields[..leaf_count].to_vec())); |
There was a problem hiding this comment.
The number of Arrow fields in each case comes from if parquet.num_columns() == 2 { 1 } else { 2 }. A reader has to work that out to see which case is which. Could the table list the Arrow leaf count next to the Parquet message instead?
| for (message, expected_message) in [ | |
| ( | |
| "message schema { optional int96 ts; required int32 id; }", | |
| "Arrow schema has fewer primitive leaves than the Parquet schema has columns", | |
| ), | |
| ( | |
| "message schema { optional int96 ts; }", | |
| "Missing Parquet column for Arrow primitive", | |
| ), | |
| ] { | |
| let parquet = parquet_schema(message); | |
| let fields = [ | |
| Field::new("ts", DataType::Timestamp(TimeUnit::Nanosecond, None), true) | |
| .with_metadata(field_id_meta(1)), | |
| Field::new("id", DataType::Int32, false).with_metadata(field_id_meta(2)), | |
| ]; | |
| let leaf_count = if parquet.num_columns() == 2 { 1 } else { 2 }; | |
| let arrow_schema = Arc::new(ArrowSchema::new(fields[..leaf_count].to_vec())); | |
| let fields = [ | |
| Field::new("ts", DataType::Timestamp(TimeUnit::Nanosecond, None), true) | |
| .with_metadata(field_id_meta(1)), | |
| Field::new("id", DataType::Int32, false).with_metadata(field_id_meta(2)), | |
| ]; | |
| for (message, arrow_leaves, expected_message) in [ | |
| ( | |
| "message schema { optional int96 ts; required int32 id; }", | |
| 1, | |
| "Arrow schema has fewer primitive leaves than the Parquet schema has columns", | |
| ), | |
| ( | |
| "message schema { optional int96 ts; }", | |
| 2, | |
| "Missing Parquet column for Arrow primitive", | |
| ), | |
| ] { | |
| let parquet = parquet_schema(message); | |
| let arrow_schema = Arc::new(ArrowSchema::new(fields[..arrow_leaves].to_vec())); |
|
|
||
| #[tokio::test] | ||
| async fn test_read_int96_timestamps_without_field_ids() { | ||
| async fn test_read_int96_with_mixed_timestamp_types_in_struct() { |
There was a problem hiding this comment.
Every nested INT96 test in this PR embeds field IDs, including this one and the list, map, and struct tests. Spark-migrated files have INT96 inside structs and no field IDs, so the reader resolves them through a name mapping. That's the layout Comet reads.
Could this test take with_field_ids like assert_int96_nanosecond_projections does? Without IDs, it would drop the = N IDs from the message and pass a nested name mapping (data with children micros and legacy) to read_int96_batches_with_name_mapping. I tried that on this branch and the expected batch matches for all three projections.
|
Thanks @mbutrovich , I added |
Which issue does this PR close?
Closes #3611.
Closes #3616.
What changes are included in this PR?
INT96 timestamp coercion preserves a timezone-less Arrow type even when Iceberg requests timestamptz. Projection compatibility checks then exclude the column, causing multi-column reads to silently return NULL.
Coerce confirmed physical INT96 columns to the precision and timezone required by the Iceberg schema, including timezone-only adjustments for timestamptz_ns. Preserve field metadata and nested structures while leaving non-INT96 timestamp types unchanged.
Are these changes tested?
Regression tests compare complete batches and physical projection masks across projection orders, embedded field IDs, name mapping, and positional fallback. Coverage includes genuine NULLs, missing fields and defaults, mixed timestamp encodings, nested structs/lists/maps, and dates outside the nanosecond range.
Validation passed:
Also validated alongside #3624 at 30b9c20. Its additional Name Mapping parameter required a one-line adjustment to the mask-test helper in the isolated checkout.
AI Disclosure
OpenAI Codex assisted with implementation and tests. Review should focus on the assumption that full Arrow primitive traversal matches physical Parquet column order, and on UTC schema-hint handling for INT96 timestamptz types.