Skip to content

Core: Detect metadata tables by identifier, not table name - #17847

Open
nimesh1601 wants to merge 2 commits into
apache:mainfrom
nimesh1601:core-fix-metadata-table-name-handling
Open

Core: Detect metadata tables by identifier, not table name#17847
nimesh1601 wants to merge 2 commits into
apache:mainfrom
nimesh1601:core-fix-metadata-table-name-handling

Conversation

@nimesh1601

Copy link
Copy Markdown

What & Why

Metadata tables were detected from the last part of an identifier alone (MetadataTableType.from(identifier.name())). This caused three related problems:

  1. A regular table that shares a metadata table name was unusable ([Bug] Iceberg tables break when they're named any of the metadata table names (e.g. files, history, manifests)  #10550). A table like db.files was interpreted as the files metadata table of namespace db, so it could not be created, loaded, or written — in Spark it surfaced as Cannot write into v1 table / appeared as a non-Iceberg table.
  2. Wrong identifier in the "not found" error (REST Catalog: Attempting to load a table with a reserved name (e.g., snapshots) before creation causes an error #13115). Loading ns1.ns2.snapshots when the base table ns1.ns2 doesn't exist reported Table does not exist: ns1.ns2 instead of the identifier the caller actually requested.
  3. REST catalog failure on a shallow metadata identifier (REST Catalog: Attempting to load a table with a reserved name (e.g., snapshots) before creation causes an error #13115). loadTable("ns1", "snapshots") fell back to loading a base table with an empty namespace, producing Malformed request: Ambiguous URI empty segment instead of a normal NoSuchTableException.

Approach

Introduce MetadataTableType.from(TableIdentifier) that treats an identifier as a metadata table only when its namespace has at least two levels — i.e. the base table (the identifier without its last part) still has a namespace. Route the catalog code through it:

  • BaseMetastoreCatalog.loadMetadataTable / isValidMetadataIdentifier
  • MetadataTableUtils.hasMetadataTableName
  • RESTSessionCatalog.loadTable (metadata-table fallback)

and report the requested identifier when the base table is missing.

Because all metastore-backed catalogs (Hive, JDBC, Hadoop, Glue, DynamoDB, Nessie, BigQuery, Snowflake, ECS, InMemory) extend BaseMetastoreCatalog, they inherit the fix; RESTSessionCatalog is the only catalog that reimplements loadTable, so it gets the equivalent change. Spark/Flink use explicit $/# delimiters and are unaffected.

Identifier Before After
db.files (real table, 1-level ns) hijacked as metadata of db regular table db.files
ns1.ns2.snapshots, base missing Table does not exist: ns1.ns2 Table does not exist: ns1.ns2.snapshots
ns1.snapshots via REST Ambiguous URI empty segment clean NoSuchTableException
db.tbl.snapshots (normal metadata table) works works (unchanged)

Behavior change / limitation

Metadata tables of a root-namespace (empty-namespace) base table accessed via dotted syntax (e.g. tbl.snapshots where tbl lives at the catalog root) are no longer recognized — a table sharing a metadata name in a single-level namespace now wins. This ambiguity is inherent (X.snapshots can't mean both) and matches the direction agreed in the prior PRs; it is not reachable on REST, which rejects empty-namespace identifiers.

Tests

  • New TestMetadataTableType unit tests for from(TableIdentifier).
  • New parameterized CatalogTests.tableSharingMetadataTableName (all 16 metadata-table types × {ns, ns1.ns2}) covering create/load of a table that shares a metadata name, resolution of its own metadata table, and correct "not found" messaging. Inherited by every catalog suite; verified on InMemory, JDBC, Hadoop, and REST. spotlessCheck and :iceberg-core:revapi pass.

Credits

Builds on prior work by @nastra in #11963 and @adutra in #13223, which stalled via the stale bot.

Closes #10550
Closes #13115

@github-actions github-actions Bot added the core label Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

1 participant