Repository navigation
Conversation
Eager planning selected columns from the unprojected schema, so a query like SELECT b FROM t returned every column. Also drop an unused import and move a use below the license header in config.rs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gabotechs
commented
Nov 17, 2026
| /// (source.file_scan_task_provider() as &dyn Any).downcast_ref::<EagerFileScanTaskProvider>() | ||
| /// } | ||
| /// ``` | ||
| pub trait FileScanTaskProvider: Any + Debug + Send + Sync { |
Contributor
Author
There was a problem hiding this comment.
@alexanderbianchi this could be hour hooking point in DFD for injecting our WorkUnitFeed, so the idea here is to kill two birds with one stone:
- providing polymorphism that supports both eager and lazy execution modes
- provide a public hooking point for injecting custom FileScanTask streams, like DFD's WorkUnitFeed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alternative to #67 for supporting both lazy and eager execution modes.
The idea here is to, rather than using the
datafusion::DataSourceas the trait that allows switching lazy and eager execution mode, to have a singleDataSourceimplementation and create a new extension point hosted in this project for customizingFileScanTaskstreaming.This way, enabling eager planning is just a matter of switching the
traitFileScanTaskProviderimplementation. This new trait should not only serve as the granular extension point for abstracting lazy VS eager behavior over the same singleIcebergDataSource, but also serve as a public extension point for people to inject their own customFileScanTask` stream providers.For example, in DFD, we have the IcebergWorkUnitFeed that can seamlessly stream
FileScanTasks either in-memory in singe-node or over the wire through a coordinator->worker connection. We could makeIcebergWorkUnitFeedimplementFileScanTaskProviderand finally stopping to host our customDataSourceimplementations in DFD.