What is the problem the feature request solves?
ShuffleWriter in operator.proto still has a top-level output_data_file, kept for native binaries that predate partition_writer. Nothing Comet ships needs it.
CometNativeShuffleWriter.buildUnifiedPlan always sets partition_writer, and for local output also copies the same path into the top-level field. In shuffle_writer_destination the branch that falls back to the top-level path when partition_writer is absent is unreachable from any Comet JVM, and otherwise the field only feeds a check that it matches partition_writer.local.output_data_file.
The jar and native library must come from the same release (docs/source/about/versioning_policy.md). After #5807 an older native library cannot serve a newer JVM anyway, since it has no getShufflePartitionOffsets.
Describe the potential solution
Reserve the tag and name of ShuffleWriter.output_data_file, stop setting it in CometNativeShuffleWriter, make shuffle_writer_destination require partition_writer, and drop the legacy-plan cases from the proto and planner tests.
Additional context
Raised in review of #5807.
What is the problem the feature request solves?
ShuffleWriterinoperator.protostill has a top-leveloutput_data_file, kept for native binaries that predatepartition_writer. Nothing Comet ships needs it.CometNativeShuffleWriter.buildUnifiedPlanalways setspartition_writer, and for local output also copies the same path into the top-level field. Inshuffle_writer_destinationthe branch that falls back to the top-level path whenpartition_writeris absent is unreachable from any Comet JVM, and otherwise the field only feeds a check that it matchespartition_writer.local.output_data_file.The jar and native library must come from the same release (
docs/source/about/versioning_policy.md). After #5807 an older native library cannot serve a newer JVM anyway, since it has nogetShufflePartitionOffsets.Describe the potential solution
Reserve the tag and name of
ShuffleWriter.output_data_file, stop setting it inCometNativeShuffleWriter, makeshuffle_writer_destinationrequirepartition_writer, and drop the legacy-plan cases from the proto and planner tests.Additional context
Raised in review of #5807.