Repository navigation
feat(iceberg): re-pin a view's Iceberg tables each time a query names it - #2784
NoahKusaba wants to merge 1 commit into
Conversation
6492672 to
9900443
Compare
418965e to
9f0ea37
Compare
andygrove
left a comment
There was a problem hiding this comment.
Thanks @NoahKusaba, this is a really nice follow-up. Views reading the latest version brings us in line with what DataFusion does on its own. Keeping DESCRIBE, DROP VIEW and CREATE OR REPLACE VIEW working when a table is gone is a thoughtful touch. The re-pinning itself looks right to me. Each distinct table loads once, subqueries are covered, and each scan keeps its projected schema. One question about views built on top of a broken view, and a small test suggestion, both inline.
| } | ||
|
|
||
| /// The catalog-backed table a scan's pinned table was pinned from, if any. | ||
| fn origin(scan: &TableScan) -> Option<&IcebergCatalogTableProvider> { |
There was a problem hiding this comment.
One case I wanted to check with you. Say t's catalog is briefly unreachable and someone runs CREATE VIEW w AS SELECT * FROM v meanwhile. Then v resolves to a RepinnedView holding the error, and since it has no plan, w stores a plain TableScan of that RepinnedView. Later lookups of w find no PinnedTable here, so repin returns None and w keeps failing with the old error, even once t loads fine again and v itself works. Would it make sense for repin to also recognise a scan over a RepinnedView and re-pin its original's plan? Or is this rare enough to just note in the docs?
There was a problem hiding this comment.
Good catch, reproduced. Rather than special-casing RepinnedView scans in repin, a table that can't be loaded is now a PinnedTable holding the error, which still knows its catalog table. So a view is always inlined, w stores v's plan, and its next lookup reloads t. This also fixes #2882, since lookups no longer fail. Covered by should_read_a_view_created_while_its_table_was_gone.
| .await | ||
| .unwrap_err(); | ||
| assert_eq!( | ||
| without_location(&error.to_string()), |
There was a problem hiding this comment.
Small suggestion. Asserting the full string ties this test to the composed-codec layout and to DataFusion's "likely caused by a bug" boilerplate, so an unrelated change to either would break it. Would checking that it contains No such table be enough here? Separately, that boilerplate comes from distributed_query.rs wrapping every encode error as Internal, which isn't a great message for a missing table. I think that's worth its own small issue rather than something for this PR.
There was a problem hiding this comment.
Agreed. Errors that come through plan encoding now only check for No such table. Opened #2885 for the Internal wrapping in distributed_query.rs. The composed codec also flattens the cause into its message; that's a TODO on ComposedLogicalExtensionCodec and will be fixed as part of upstreaming it to DataFusion, rather than as a separate Ballista issue.
9f0ea37 to
eb27c7f
Compare
|
test failure is fake, apparently the CI can't download apache rat |
eb27c7f to
e5b7f95
Compare
Which issue does this PR close?
Closes #2726.
Closes #2882.
Follow-up to #2638, split out at andygrove's request so the plan rewriting gets its own review.
Rationale for this change
#2638 pins each catalog-backed Iceberg table to one version when a query looks it up. DataFusion stores a view as the plan it was created with, so a view over an Iceberg table keeps reading the version from when the view was created. In single-process DataFusion, the same view reads new commits on every run.
What changes are included in this PR?
When a query names a view, the pinning schema provider re-pins the view's stored plan: every scan of a pinned catalog-backed table, including scans inside subqueries, is replaced with the version its catalog serves now. Each distinct table is loaded once, and the loads run concurrently.
PinnedTablekeeps theIcebergCatalogTableProviderit was pinned from (never sent between nodes), so it can be pinned again.A view is any provider DataFusion inlines, one whose
get_logical_plan()returns a plan: aViewTablefromCREATE VIEW, aDataFrame'sinto_vieworinto_temporary_view, or a custom view. The lookup returns aRepinnedViewthat wraps the original and replaces only its plan, so the view keeps its table type, schema and definition.A lookup never fails because a table can't be loaded, for example because it was dropped from its catalog or the catalog is unreachable. The
PinnedTableholds the load error instead (Result<Table, Arc<DataFusionError>>) and still knows the catalog table it came from. A query fails with the error only when it reads the table: when a filter is pushed into it, when it's scanned, or when the client encodes the plan for the scheduler. So:DESCRIBE,DROP VIEW,CREATE OR REPLACE VIEWandDROP TABLEstill work, since they look the table or view up without reading it. Before,DROP TABLEon a registered table its catalog couldn't load failed (Iceberg: DROP TABLE fails on a registered table its catalog can't load #2882).This matches DataFusion, where
IcebergCatalogTableProviderloads its table inscan().It re-pins the stored plan rather than re-planning the view's SQL.
SchemaProvider::tablegets no session state to plan with, and a view created from aDataFramehas no SQL. Nested views need no recursion, since DataFusion inlines a view into any plan built over it.It also adds a TODO on
ComposedLogicalExtensionCodec's error handling inballista-core, to be fixed as part of upstreaming it to DataFusion: it joins its codecs' errors into one string, so the cause of an encoding failure can't be reached through the error chain.Limitations:
DataFrameor Flight SQL prepared statement still reads the version from when it was planned. Neither looks its tables up again.Are these changes tested?
Yes, with standalone Ballista over a memory catalog:
should_read_a_views_latest_versioncreates a view over a table and a second view over the first with a subquery on the table, commits a row, and checks that both views return it.should_read_a_dataframe_views_latest_versiondoes the same forinto_viewandinto_temporary_view, and checks that each keeps its table type.should_read_a_view_created_while_its_table_was_gonecreates a view over a view while the table is gone, checks that reading it fails, then brings the table back and checks that it reads the latest version.should_fail_to_read_a_view_whose_table_is_gonechecks the error from reading the view, with and without a filter.should_describe_drop_and_replace_while_a_table_is_gonechecks thatDESCRIBE,DROP VIEW,CREATE OR REPLACE VIEWandDROP TABLEstill work.Errors that reach the test through the client's plan encoding are checked only for their cause, since Ballista wraps them in text of its own (#2885).
Are there any user-facing changes?
Yes: views over Iceberg tables now read the latest version, and a table or view that can't be loaded fails only when read, so it can still be described, dropped or replaced. The docs' limitation for views is narrowed to reused
DataFrames and prepared statements.Follow-ups
RepinnedViewand the re-pinning intable()go away. It needs a hook in Ballista's core, since an integration can't wrapBallistaQueryPlanner.Internal.