Skip to content

feat: read v3 row lineage metadata columns#822

Open
wgtmac wants to merge 1 commit into
apache:mainfrom
wgtmac:read_row_lineage
Open

feat: read v3 row lineage metadata columns#822
wgtmac wants to merge 1 commit into
apache:mainfrom
wgtmac:read_row_lineage

Conversation

@wgtmac

@wgtmac wgtmac commented Jul 6, 2026

Copy link
Copy Markdown
Member

Synthesize _row_id and _last_updated_sequence_number when reading data files that omit row lineage columns, and preserve physical non-null lineage values when present.

Carry inherited data sequence numbers on DataFile metadata so scan planning and readers follow the same model as Java.

Synthesize _row_id and _last_updated_sequence_number when reading
data files that omit row lineage columns, and preserve physical
non-null lineage values when present.

Carry inherited data sequence numbers on DataFile metadata so scan
planning and readers follow the same model as Java.
auto options =
MakeReaderOptions(*data_file, io_, projected_schema_, task.residual_filter(),
name_mapping_, properties_);
auto options = MakeReaderOptions(

@manuzhang manuzhang Jul 7, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REST scan-task JSON appears to round-trip only first-row-id for DataFile; FileScanTasksFromJson constructs the FileScanTask without any data sequence number. That means this call can pass data_file->first_row_id but an empty data_file->data_sequence_number, and MakeLastUpdatedSequenceNumberArray will return nulls for _last_updated_sequence_number.

The row-lineage spec says missing/null _last_updated_sequence_number should be replaced with the data file’s data sequence number.

Disclosure: I used an LLM to help review this PR and draft this comment.

const arrow::MetadataColumnContext& metadata_context,
::arrow::ArrayBuilder* array_builder);

Status AppendRowLineageValue(int32_t field_id,

@WZhuo WZhuo Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AppendRowLineageValue is defined verbatim in both avro_data_util.cc:57 and avro_direct_decoder.cc:147 with identical logic.


if (entry.data_file) {
entry.data_file->partition_spec_id = spec_id_;
entry.data_file->data_sequence_number = entry.sequence_number;

@WZhuo WZhuo Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BaseInheritableMetadata::Apply() only propagates data_sequence_number to the DataFile, but the Java equivalent (InheritableMetadataFactory.BaseInheritableMetadata.apply()) sets four fields:

file.setSpecId(specId);
file.setDataSequenceNumber(manifestEntry.dataSequenceNumber());
file.setFileSequenceNumber(manifestEntry.fileSequenceNumber());
file.setManifestLocation(manifestLocation);

Suggestion: either add the fields and propagate them now, or leave an explicit // TODO: file_sequence_number and manifest_location comment so the gap is visible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants