perf: avoid FFI import/export when passing batches between native plans - #3930

Closed

andygrove wants to merge 24 commits into

apache:mainfrom

andygrove:batch-stash-shuffle-optimization

Commits on Apr 12, 2026

docs: add design spec for batch stash shuffle optimization
andygrove
committed
86a4989
docs: add implementation plan for batch stash shuffle optimization
andygrove
committed
488522c
feat: add BatchStash registry for native batch handle passing
andygrove
committed
ca8ee6a
feat: add CometHandleBatchIterator Java class and JNI bridge
andygrove
committed
e68d777
feat: add executePlanBatchHandle JNI function for stash-mode output
andygrove
committed
07c86c7
feat: add stash mode to CometExecIterator for batch handle output
andygrove
committed
d2f87ab
feat: add handle-mode input path to ScanExec for batch stash retrieval
andygrove
committed
0a49ca6
feat: detect CometHandleBatchIterator in planner and enable handle mode
andygrove
committed
84862ef
feat: preserve CometExecIterator reference through shuffle dependency RDD
andygrove
committed
3588543
feat: integrate batch stash mode in CometNativeShuffleWriter
andygrove
committed
db137ad
style: apply formatting fixes
andygrove
committed
ed95e83
fix: use scan source name for handle mode detection instead of JNI is_instance_of
andygrove
committed
a827437
chore: remove JVM crash log files
andygrove
committed
3af0e9a
refactor: deduplicate executePlan logic and address code review findings
andygrove
committed
e6c3a42
refactor: use protobuf field for batch stash handle mode instead of scan source name
andygrove
committed
6cc0e78
chore: remove design docs from PR
andygrove
committed
c42be90
fix: add batch_stash_handle field to Scan struct literals in tests
andygrove
committed
f70119a
fix: use InputBatch::Complete for stashed batches to bypass schema reconciliation
andygrove
committed
c64d482
feat: add spark.comet.exec.shuffle.batchStash.enabled config (default true)
andygrove
committed
d37752a
fix: apply schema reconciliation for stashed batches with type mismatches
andygrove
committed
38af7d9

Commits on Apr 13, 2026

fix: address PR review feedback for batch stash
andygrove
committed
ba07bf7

Commits on Apr 16, 2026

refactor: move batch stash from global static into ExecutionContext
andygrove
committed
008d7cf

Commits on May 7, 2026

fix: address PR review feedback for batch stash
andygrove
committed
f4196d1
refactor: simplify batch stash review fixes
andygrove
committed
6f045ba