Skip to content

[spark] Fix nested struct values wrong order for insert columns - #9415

Open
ArnavBalyan wants to merge 1 commit into
apache:masterfrom
ArnavBalyan:arnavb/fix-spark-resplve
Open

[spark] Fix nested struct values wrong order for insert columns#9415
ArnavBalyan wants to merge 1 commit into
apache:masterfrom
ArnavBalyan:arnavb/fix-spark-resplve

Conversation

@ArnavBalyan

Copy link
Copy Markdown
Member

Purpose

  • Insert column lists should order both columns and nested struct fields by position.
  • Paimon maps nested struct fields by name, causing values to be reordered when the source and target field orders differ.
  • This causes the query results to differ, ensure we can fix by preserving the mapping recursively for structs inside arrays and maps.

Tests

  • UT

@ArnavBalyan

Copy link
Copy Markdown
Member Author

cc @JingsongLi thanks :)

}

private def renameFieldsInStruct(input: StructType, expected: StructType): StructType = {
if (input.length == expected.length) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve positional mapping when nested field counts differ

Returning input here falls back to the downstream by-name resolver whenever merge-schema fills a missing nested field. For target ARRAY<STRUCT<x: INT, y: INT, z: INT>>, a column-list write whose input struct fields are (y=20, x=10) stores [10, 20, null] instead of the positional [20, 10, null]. I reproduced this on Spark 3.5 with spark.paimon.write.merge-schema=true. Please rename the common ordinal prefix even when the lengths differ, preserve unmatched input fields, and let the existing strict/merge-schema handling process missing or extra fields. A regression test with unequal nested field counts would cover this path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants