feat(isthmus): observe field reference types - #1080
Conversation
Report Calcite input-schema types for root, outer and lambda field references through TypeObserver while keeping RexNode construction unchanged. The conversion context now tracks input row types and lambda parameter types; a field reference converted with no input in scope is reported as an inference failure rather than failing the conversion. Aggregate, sort and fetch open a conversion scope so their expressions can resolve input field types. Calcite's Aggregate and Sort carry no variablesSet, so a correlation bound to one of those inputs is rejected instead of being silently dropped. Named update now applies its filter with the correlation ids resolved while converting the condition and the transformations.
2fc101f to
fc0c796
Compare
|
Hi @vbarua, when you have a chance, could you take a look at the next #379 slice? It extends |
| } | ||
| } | ||
|
|
||
| exitUncorrelatedScope(context, "aggregate"); |
There was a problem hiding this comment.
I wonder if we could use the Aggregate rel name for real rather than hardcoding a string here? (same thing for other spots)
| transform.getTransformation().accept(expressionRexConverter, context)); | ||
| } | ||
|
|
||
| java.util.Set<CorrelationId> correlationIds = context.exitScope(); |
There was a problem hiding this comment.
Question as I'm somewhat unfamiliar with what's going on here: what is this correlationIds thing, and why is it now being fed into the filter? I had thought the filter + build lines were moved because the type observer didn't want to see them or something?
2 participants
Summary
TypeObserverMotivation
#379 lists four conversion points where the Substrait-supplied type and the type Calcite would infer can diverge; scalar functions (#1015) and window functions (#1059) are already observed. Field references are the case the issue singles out as needing access to the underlying input schema, which is what the scope tracking here adds.
Changes
Aggregate, sort and fetch open a conversion scope so their expressions can resolve input field types. Calcite's
AggregateandSortcarry novariablesSet, so an outer reference bound to one of those inputs has nowhere to be declared and is rejected — previously it failed anyway, inrequireScope, because no scope existed at all.NamedUpdateapplies its filter with the correlation ids resolved while converting the condition and the transformations.Related to #379.