Core: Detect metadata tables by identifier, not table name - #17847
Open
nimesh1601 wants to merge 2 commits into
Open
Core: Detect metadata tables by identifier, not table name#17847nimesh1601 wants to merge 2 commits into
nimesh1601 wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & Why
Metadata tables were detected from the last part of an identifier alone (
MetadataTableType.from(identifier.name())). This caused three related problems:files,history,manifests) #10550). A table likedb.fileswas interpreted as thefilesmetadata table of namespacedb, so it could not be created, loaded, or written — in Spark it surfaced asCannot write into v1 table/ appeared as a non-Iceberg table.ns1.ns2.snapshotswhen the base tablens1.ns2doesn't exist reportedTable does not exist: ns1.ns2instead of the identifier the caller actually requested.loadTable("ns1", "snapshots")fell back to loading a base table with an empty namespace, producingMalformed request: Ambiguous URI empty segmentinstead of a normalNoSuchTableException.Approach
Introduce
MetadataTableType.from(TableIdentifier)that treats an identifier as a metadata table only when its namespace has at least two levels — i.e. the base table (the identifier without its last part) still has a namespace. Route the catalog code through it:BaseMetastoreCatalog.loadMetadataTable/isValidMetadataIdentifierMetadataTableUtils.hasMetadataTableNameRESTSessionCatalog.loadTable(metadata-table fallback)and report the requested identifier when the base table is missing.
Because all metastore-backed catalogs (Hive, JDBC, Hadoop, Glue, DynamoDB, Nessie, BigQuery, Snowflake, ECS, InMemory) extend
BaseMetastoreCatalog, they inherit the fix;RESTSessionCatalogis the only catalog that reimplementsloadTable, so it gets the equivalent change. Spark/Flink use explicit$/#delimiters and are unaffected.db.files(real table, 1-level ns)dbdb.filesns1.ns2.snapshots, base missingTable does not exist: ns1.ns2Table does not exist: ns1.ns2.snapshotsns1.snapshotsvia RESTAmbiguous URI empty segmentNoSuchTableExceptiondb.tbl.snapshots(normal metadata table)Behavior change / limitation
Metadata tables of a root-namespace (empty-namespace) base table accessed via dotted syntax (e.g.
tbl.snapshotswheretbllives at the catalog root) are no longer recognized — a table sharing a metadata name in a single-level namespace now wins. This ambiguity is inherent (X.snapshotscan't mean both) and matches the direction agreed in the prior PRs; it is not reachable on REST, which rejects empty-namespace identifiers.Tests
TestMetadataTableTypeunit tests forfrom(TableIdentifier).CatalogTests.tableSharingMetadataTableName(all 16 metadata-table types ×{ns, ns1.ns2}) covering create/load of a table that shares a metadata name, resolution of its own metadata table, and correct "not found" messaging. Inherited by every catalog suite; verified on InMemory, JDBC, Hadoop, and REST.spotlessCheckand:iceberg-core:revapipass.Credits
Builds on prior work by @nastra in #11963 and @adutra in #13223, which stalled via the stale bot.
Closes #10550
Closes #13115