Skip to content

feat(core): update to substrait v0.100.0 - #1082

Merged
nielspardon merged 4 commits into
substrait-io:mainfrom
anasik:feat/update-substrait-100
Aug 12, 2026
Merged

feat(core): update to substrait v0.100.0#1082
nielspardon merged 4 commits into
substrait-io:mainfrom
anasik:feat/update-substrait-100

Conversation

@anasik

@anasik anasik commented Aug 11, 2026

Copy link
Copy Markdown
Contributor
  • Closes Update to Substrait v0.100.0 #1076
  • Bump substrait-packaging to 0.100.0
  • Load unsigned_integers.yaml into the default extension catalog
  • Add HasExtension/advanced_extension support to Expand, matching the new ExpandRel proto field, with round-trip test coverage

@anasik
anasik force-pushed the feat/update-substrait-100 branch from 4fbc30c to ecaaf51 Compare August 11, 2026 15:55
@anasik anasik changed the title Closes #1076: Update to Substrait v0.100.0 feat(core): update to substrait v0.100.0 Aug 11, 2026
@anasik anasik changed the title feat(core): update to substrait v0.100.0 feat(core): update to substrait v0.100.0 Aug 11, 2026
@nielspardon

Copy link
Copy Markdown
Member

I approved the workflow runs yesterday and it looks like the Spark dialect file also needs to be updated in this PR.

The following command should regenerate it:

./gradlew dialect

@anasik

anasik commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

@nielspardon done

@nielspardon

Copy link
Copy Markdown
Member

We fixed a bug in the Spark dialect generation code in #1086. If you rebase we don't need the Spark dialect changes anymore.

@nielspardon
nielspardon self-requested a review August 12, 2026 13:40
@anasik
anasik force-pushed the feat/update-substrait-100 branch from ccc4658 to fa9284c Compare August 12, 2026 13:47
@anasik

anasik commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

@nielspardon done

@nielspardon

Copy link
Copy Markdown
Member

I think you still would need to drop that second commit with the spark changes

@anasik
anasik force-pushed the feat/update-substrait-100 branch from fa9284c to e20656e Compare August 12, 2026 14:16
@anasik

anasik commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

@nielspardon try now

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that the generator drops variants whose signature has no Spark sample value, no unsigned function is emitted (Spark has no unsigned type mapping), so this alias has nothing referencing it: the rebased dialect has zero functions with source: "unsigned_integers". On main, declared and used aliases match exactly at 12/12, and functions_list/functions_set are precedent for a catalog extension that is simply not declared here. Suggestions below to drop both lines; they can come back together with the functions if Spark ever gains an unsigned type mapping.

Comment thread spark/src/main/scala/io/substrait/spark/utils/DialectGenerator.scala Outdated
Comment thread spark/spark_dialect.yaml Outdated

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I think we can safely drop the two lines in the spark module

anasik and others added 2 commits August 12, 2026 21:51
….scala


remove `unsigned_integers`

Co-authored-by: Niels Pardon <mail@niels-pardon.de>
remove unsigned_integers

Co-authored-by: Niels Pardon <mail@niels-pardon.de>
@nielspardon
nielspardon merged commit e65e9d4 into substrait-io:main Aug 12, 2026
21 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update to Substrait v0.100.0

2 participants