[spark] Fix nested struct values wrong order for insert columns - #9415
[spark] Fix nested struct values wrong order for insert columns#9415ArnavBalyan wants to merge 1 commit into
Conversation
|
cc @JingsongLi thanks :) |
| } | ||
|
|
||
| private def renameFieldsInStruct(input: StructType, expected: StructType): StructType = { | ||
| if (input.length == expected.length) { |
There was a problem hiding this comment.
[P1] Preserve positional mapping when nested field counts differ
Returning input here falls back to the downstream by-name resolver whenever merge-schema fills a missing nested field. For target ARRAY<STRUCT<x: INT, y: INT, z: INT>>, a column-list write whose input struct fields are (y=20, x=10) stores [10, 20, null] instead of the positional [20, 10, null]. I reproduced this on Spark 3.5 with spark.paimon.write.merge-schema=true. Please rename the common ordinal prefix even when the lengths differ, preserve unmatched input fields, and let the existing strict/merge-schema handling process missing or extra fields. A regression test with unequal nested field counts would cover this path.
Purpose
Tests