[SQL / Core] Preserve tuple schema and validate types between SQL and Java operators - #803
[SQL / Core] Preserve tuple schema and validate types between SQL and Java operators#803AliRana30 wants to merge 1 commit into
Conversation
|
Just wanted to highlight the impact of this bug in production pipelines: In cross-platform workflows (e.g., PostgreSQL / BigQuery / Trino ➔ Java Streams / Spark), SQL joins and network transfers are typically the heaviest and most expensive steps. Because there was no plan-time validation, pipelines would spend significant compute time, cluster bandwidth, and cloud warehouse credits running complex joins across large datasets, only to fail at the very final Java step on the first record with a Beyond the wasted compute budget and delayed SLA cycles, if records were accessed by numerical field indices without explicit casts, the flattened tuple structure could have silently corrupted downstream analytics by mapping fields to the wrong offsets. Catching this early during channel conversion and properly preserving |
Closes #747
When executing a pipeline involving SQL TableSources, a SQL Join, and downstream Java operators (e.g.,
PostgresTableSource -> Join -> SqlToStream -> Java map -> collect), a runtimeClassCastExceptionoccurred.Root Cause
JoinOperatoris structured asTuple2<Record, Record>. However,SqlToStreamOperatorandGenericSqlToStreamOperatorhardcoded their output to flatRecordobjects, discarding the tuple structure. When downstream Java operators expectedTuple2<Record, Record>, they received rawRecordobjects and threw runtimeClassCastException.Channel.addConsumernorDefaultChannelConversion.convertperformed early schema/type validation between channels and consumers, so mismatches were not caught during plan compilation.Changes
SqlToStreamOperator,SqlToRddOperator, andGenericSqlToStreamOperatorgeneric (<Type>) with support for emittingTuple2<Record, Record>viaTuple2ResultSetIteratorandrowToTuple2.resolveLeftColumnCountandgetTableColumnCountusing database metadata and operator schemas to accurately partition joined rows into left and rightRecordinstances.ChannelConversionsacross all SQL platforms (postgres,sqlite3,trino,presto,bigquery,generic-jdbc) to pass the channel producer's data type.Channel.addConsumerandDefaultChannelConversion.convertto validateconsumerInput.getType().isSupertypeOf(producerSlot.getType())or adapt types viaadaptType(...), throwing an earlyIllegalArgumentExceptionon genuine mismatches.Tuple2evaluation with Java mapping inSqlToStreamOperatorTest, and channel type validation tests inChannelTypeValidationTest.Type of Change
How Has This Been Tested?
SqlToStreamOperatorTest(4 tests passed, includingtestJoinWithHsqldbYieldsTuple2andtestTypeAdaptationAndValidation).ChannelTypeValidationTest(3 tests passed, verifying type mismatch rejection and compatible consumer acceptance).JdbcJoinOperatorTestandGenericJdbcJoinOperatorTestpassed with 0 failures.