Skip to content

feat: abstract Iceberg file discovery behind a FileScanTaskProvider trait - #109

Open
gabotechs wants to merge 3 commits into
apache:mainfrom
gabotechs:file-scan-task-provider
Open

gabotechs wants to merge 3 commits into
apache:mainfrom
gabotechs:file-scan-task-provider

Conversation

@gabotechs

@gabotechs gabotechs commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Alternative to #67 for supporting both lazy and eager execution modes.

The idea here is to, rather than using the datafusion::DataSource as the trait that allows switching lazy and eager execution mode, to have a single DataSource implementation and create a new extension point hosted in this project for customizing FileScanTask streaming.

                      IcebergDataSource
        reader, limit, metrics, EXPLAIN prefix, partitioning
                              │
                 Arc<dyn FileScanTaskProvider>
                    ┌─────────┴──────────┐
     LazyFileScanTaskProvider      EagerFileScanTaskProvider
     discovers files on first      file groups planned up front,
     poll, 1 partition             N partitions, memory reserved

This way, enabling eager planning is just a matter of switching the trait FileScanTaskProviderimplementation. This new trait should not only serve as the granular extension point for abstracting lazy VS eager behavior over the same singleIcebergDataSource, but also serve as a public extension point for people to inject their own custom FileScanTask` stream providers.

For example, in DFD, we have the IcebergWorkUnitFeed that can seamlessly stream FileScanTasks either in-memory in singe-node or over the wire through a coordinator->worker connection. We could make IcebergWorkUnitFeed implement FileScanTaskProvider and finally stopping to host our custom DataSource implementations in DFD.

gabotechs and others added 3 commits November 17, 2026 12:23
Eager planning selected columns from the unprojected schema, so a query
like SELECT b FROM t returned every column. Also drop an unused import
and move a use below the license header in config.rs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/// (source.file_scan_task_provider() as &dyn Any).downcast_ref::<EagerFileScanTaskProvider>()
/// }
/// ```
pub trait FileScanTaskProvider: Any + Debug + Send + Sync {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexanderbianchi this could be hour hooking point in DFD for injecting our WorkUnitFeed, so the idea here is to kill two birds with one stone:

  • providing polymorphism that supports both eager and lazy execution modes
  • provide a public hooking point for injecting custom FileScanTask streams, like DFD's WorkUnitFeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant