[Bug] JDBC missing schema handling may cause NPE or repeated metadata reloads

[wenzhenghu]

Search before asking

• I searched the existing issues and found no similar issue.

Version

Verified against the following upstream heads on 2026-09-01:

• branch-4.0: 8a9961723ea4be00cdf923c60759607202c7e2e7
• branch-4.1: 6f4c6a4be42ab3f4e1811982703d4b71b5a8ea3c
• master: feb9e04f78490296c3393cbd594aef617af6b433

Related preload PRs:

• [improvement](fe) Preload external table metadata before internal table lock in mixed queries #64035 (master)
• branch-4.1:[improvement](fe) Preload external table metadata before internal table lock in mixed queries #64579 (branch-4.1 backport)

What is wrong?

JDBC external tables do not have a consistent, cache-safe failure contract when remote metadata returns no columns or the remote table handle can no longer be resolved.

branch-4.0 and branch-4.1

The failure chain is:

1. JdbcExternalTable.initSchema() returns Optional.empty() when listColumns() returns null or an empty list.
2. The external schema cache stores the negative result.
3. ExternalTable.getFullSchema() maps the empty value to null.
4. getBaseSchema() also returns null.
5. LogicalCatalogRelation.computeOutput() calls table.getBaseSchema().stream() and throws a null pointer exception.

Typical error:

Cannot invoke "java.util.List.stream()" because the return value of
"org.apache.doris.catalog.TableIf.getBaseSchema()" is null

branch-4.1 additionally supports JDBC metadata preload through #64579. When enable_preload_external_metadata=true and a mixed query contains both an internal table requiring a plan-time read lock and a JDBC table, PreloadExternalMetadata calls getBaseSchema() before locking but does not validate its result. Analysis later reaches the same stream() NPE. Disabling preload does not remove the underlying bug; it only changes when the schema is loaded.

branch-4.0 does not contain the preload feature, but regular JDBC query planning still has the same nullable-schema chain.

Relevant code:

• branch-4.0 JDBC loader:

  doris/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalTable.java

  Lines 129 to 136 in 8a99617

  	public Optional<SchemaCacheValue> initSchema() {
  	String remoteDbName = ((ExternalDatabase<?>) this.getDatabase()).getRemoteName();
  	// 1. Retrieve remote column information
  	List<Column> columns = ((JdbcExternalCatalog) catalog).listColumns(remoteDbName, remoteName);
  	if (columns == null || columns.isEmpty()) {
  	return Optional.empty();
  	}

• branch-4.0 nullable schema:

  doris/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java

  Lines 175 to 184 in 8a99617

  	@Override
  	public List<Column> getFullSchema() {
  	ExternalSchemaCache cache = Env.getCurrentEnv().getExtMetaCacheMgr().getSchemaCache(catalog);
  	Optional<SchemaCacheValue> schemaCacheValue = cache.getSchemaValue(new SchemaCacheKey(getOrBuildNameMapping()));
  	return schemaCacheValue.map(SchemaCacheValue::getSchema).orElse(null);
  	}
  	@Override
  	public List<Column> getBaseSchema() {
  	return getFullSchema();

• branch-4.1 JDBC loader:

  doris/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalTable.java

  Lines 130 to 157 in 6f4c6a4

  	public boolean supportsExternalMetadataPreload() {
  	return true;
  	}
  	@Override
  	public Optional<SchemaCacheValue> initSchema() {
  	String remoteDbName = ((ExternalDatabase<?>) this.getDatabase()).getRemoteName();
  	// A missing table pattern makes JDBC enumerate every table in the database, so honor the
  	// effective-name fallback before any metadata or identifier-mapping call.
  	String remoteTableName = getRemoteName();
  	if (DebugPointUtil.isEnable("JdbcExternalTable.initSchema.sleep")) {
  	long sleepMs = DebugPointUtil.getDebugParamOrDefault(
  	"JdbcExternalTable.initSchema.sleep", "sleepMs", 0L);
  	if (sleepMs > 0) {
  	LOG.info("debug point JdbcExternalTable.initSchema.sleep hit for {}.{}, sleep {}ms",
  	remoteDbName, remoteTableName, sleepMs);
  	try {
  	Thread.sleep(sleepMs);
  	} catch (InterruptedException ignore) {
  	Thread.currentThread().interrupt();
  	}
  	}
  	}
  	// 1. Retrieve remote column information
  	List<Column> columns = ((JdbcExternalCatalog) catalog).listColumns(remoteDbName, remoteTableName);
  	if (columns == null || columns.isEmpty()) {
  	return Optional.empty();

• branch-4.1 preload call:

  doris/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/PreloadExternalMetadata.java

  Lines 102 to 111 in 6f4c6a4

  	boolean preloadLatestSnapshot = latestOnlyRelation && supportsLatestSnapshot;
  	// Skip schema and partition warmup for snapshot-aware tables when only non-latest relations are referenced.
  	boolean preloadSchema = !supportsLatestSnapshot || latestOnlyRelation;
  	boolean preloadPartition = preloadSchema && table.supportInternalPartitionPruned();
  	if (preloadLatestSnapshot) {
  	statementContext.loadSnapshots(table, Optional.empty(), Optional.empty());
  	}
  	if (preloadSchema) {
  	table.getBaseSchema();
  	}

• nullable consumer:

  doris/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java

  Lines 134 to 141 in 6f4c6a4

  	@Override
  	public List<Slot> computeOutput() {
  	IdGenerator<ExprId> exprIdGenerator = StatementScopeIdGenerator.getExprIdGenerator();
  	Builder<Slot> slots = ImmutableList.builder();
  	table.getBaseSchema()
  	.stream()
  	.map(col -> SlotReference.fromColumn(exprIdGenerator.getNextId(), table, col, qualified()))
  	.forEach(slots::add);

master

master has migrated JDBC catalogs to PluginDrivenExternalTable and therefore needs a separate fix rather than a mechanical backport.

• PluginDrivenExternalTable.initSchema() returns Optional.empty() when the connector table handle is missing.
• With a valid handle but zero returned columns, it creates a present schema cache value containing an empty column list.
• ExternalTable.getFullSchema() still maps an empty optional to null, while consumers such as LogicalCatalogRelation assume a non-null list.

The latest metadata-cache refactor in #66633 changes the common default path: the cache loader now converts Optional.empty() into a generic CacheException before getFullSchema() returns null. This normally avoids the exact NPE, but it is not an equivalent JDBC fix:

• the error does not include actionable JDBC catalog and remote table context;
• failed loads are completed exceptionally and are not retained as negative cache entries;
• repeated queries can repeatedly execute remote table-handle or metadata resolution;
• zero-column schemas with a valid handle are still represented as a normal empty schema.

Relevant code:

• plugin-driven schema loader:

  doris/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java

  Lines 459 to 513 in feb9e04

  	public boolean supportsExternalMetadataPreload() {
  	if (!(catalog instanceof PluginDrivenExternalCatalog)) {
  	return false;
  	}
  	// F11: gate async metadata pre-load on the connector-declared SUPPORTS_METADATA_PRELOAD capability
  	// (replacing the legacy engine-name "jdbc" string, per the iron rule). jdbc and iceberg both declare
  	// it; connectors not yet validated for concurrent pre-warming (e.g. ES) do not, and fall back to
  	// synchronous load at binding time. Pure planning/lock-latency optimization, no correctness effect.
  	Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector();
  	return connector != null
  	&& connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_METADATA_PRELOAD);
  	}
  	@Override
  	public Optional<SchemaCacheValue> initSchema() {
  	PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog;
  	// Keep the JDBC schema delay debug point available for manual regression verification.
  	if ("jdbc".equalsIgnoreCase(pluginCatalog.getType())
  	&& DebugPointUtil.isEnable("PluginDrivenExternalTable.initSchema.sleep")) {
  	long sleepMs = DebugPointUtil.getDebugParamOrDefault(
  	"PluginDrivenExternalTable.initSchema.sleep", "sleepMs", 0L);
  	if (sleepMs > 0) {
  	LOG.info("debug point PluginDrivenExternalTable.initSchema.sleep hit for {}.{}, sleep {}ms",
  	db != null ? db.getRemoteName() : "", getRemoteName(), sleepMs);
  	try {
  	Thread.sleep(sleepMs);
  	} catch (InterruptedException ignore) {
  	Thread.currentThread().interrupt();
  	}
  	}
  	}
  	Connector connector = pluginCatalog.getConnector();
  	ConnectorSession session = pluginCatalog.buildCrossStatementSession();
  	try {
  	ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector);
  	String dbName = db != null ? db.getRemoteName() : "";
  	String tableName = getRemoteName();
  	if (isView()) {
  	// A connector view has no table handle (the SDK tableExists() is false for views); build the schema
  	// from the view definition's columns instead. Mirrors legacy IcebergUtils.loadViewSchemaCacheValue
  	// (icebergView.schema()). Gated on isView() => only view-supporting connectors (SUPPORTS_VIEW) reach
  	// here; view-less connectors (jdbc/paimon/maxcompute) keep isView()==false and skip this.
  	ConnectorViewDefinition viewDefinition = metadata.getViewDefinition(session, dbName, tableName);
  	ConnectorTableSchema viewSchema = new ConnectorTableSchema(
  	tableName, viewDefinition.getColumns(), null, Collections.emptyMap());
  	return Optional.of(toSchemaCacheValue(metadata, session, dbName, tableName, viewSchema));
  	}
  	Optional<ConnectorTableHandle> handleOpt = resolveConnectorTableHandle(session, metadata);
  	if (!handleOpt.isPresent()) {
  	LOG.warn("Table handle not found for plugin-driven table: {}.{}", dbName, tableName);
  	return Optional.empty();
  	}
  	ConnectorTableSchema tableSchema = metadata.getTableSchema(session, handleOpt.get());
  	return Optional.of(toSchemaCacheValue(metadata, session, dbName, tableName, tableSchema));

• nullable base implementation:

  doris/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java

  Lines 180 to 195 in feb9e04

  	@Override
  	public List<Column> getFullSchema() {
  	// NOT getFullSchema(Optional.empty()): an empty snapshot means "this reference has no pin" (=>
  	// latest), whereas the no-arg form means "I have no reference, resolve from the ambient context".
  	// Collapsing the two would strip the ambient resolution from every statement-global caller.
  	Optional<SchemaCacheValue> schemaCacheValue = getSchemaCacheValue();
  	return schemaCacheValue.map(SchemaCacheValue::getSchema).orElse(null);
  	}
  	/**
  	* The full schema AS OF {@code snapshot}. See {@link #getSchemaCacheValue(Optional)} for why the plan
  	* path must pass the reference's pin rather than relying on the ambient lookup.
  	*/
  	public List<Column> getFullSchema(Optional<MvccSnapshot> snapshot) {
  	Optional<SchemaCacheValue> schemaCacheValue = getSchemaCacheValue(snapshot);
  	return schemaCacheValue.map(SchemaCacheValue::getSchema).orElse(null);

• generic cache exception:

  doris/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java

  Lines 524 to 535 in feb9e04

  	private SchemaCacheValue loadSchemaCacheValue(SchemaCacheKey key) {
  	CatalogIf<?> catalog = Env.getCurrentEnv().getCatalogMgr().getCatalog(key.getNameMapping().getCtlId());
  	if (!(catalog instanceof ExternalCatalog)) {
  	throw new CacheException("catalog %s is not external when loading schema cache",
  	null, key.getNameMapping().getCtlId());
  	}
  	ExternalCatalog externalCatalog = (ExternalCatalog) catalog;
  	return externalCatalog.getSchema(key).orElseThrow(() -> new CacheException(
  	"failed to load schema cache value for: %s.%s.%s",
  	null, key.getNameMapping().getCtlId(),
  	key.getNameMapping().getLocalDbName(),
  	key.getNameMapping().getLocalTblName()));

• non-null consumer assumption:

  doris/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java

  Lines 152 to 159 in feb9e04

  	@Override
  	public List<Slot> computeOutput() {
  	IdGenerator<ExprId> exprIdGenerator = StatementScopeIdGenerator.getExprIdGenerator();
  	Builder<Slot> slots = ImmutableList.builder();
  	table.getBaseSchema()
  	.stream()
  	.map(col -> SlotReference.fromColumn(exprIdGenerator.getNextId(), table, col, qualified()))
  	.forEach(slots::add);

What did you expect?

When JDBC schema metadata cannot be resolved:

1. Query analysis should fail with a deterministic, actionable exception containing the catalog and remote database/table name.
2. No planner path should expose a null schema to callers that require a schema list.
3. A persistent metadata failure should not cause every repeated query to reconnect to or reload metadata from the remote JDBC source.
4. Preload-enabled and preload-disabled queries should have the same error semantics.
5. The generic preload rule should not invalidate the table schema cache on every failure.

How to reproduce

Possible reproduction conditions include a JDBC driver returning no rows from DatabaseMetaData.getColumns(), insufficient metadata permission, or a remote table being removed while Doris still has a table-name entry.

For branch-4.1:

1. Create a JDBC catalog and make one remote table visible through table listing.
2. Make the column metadata lookup return an empty result.
3. Enable enable_preload_external_metadata.
4. Run a Nereids mixed query joining an internal Doris table and the JDBC table.
5. Observe that preload reads the empty schema and later planning throws the getBaseSchema().stream() NPE.
6. Repeat with preload disabled; the same nullable schema can fail during normal relation output computation.

For branch-4.0, run the JDBC query without the preload-specific steps.

For master, make JdbcConnectorMetadata.getTableHandle() return empty, or return a valid handle with an empty ConnectorTableSchema, then repeat the query and inspect both the exception and the number of remote metadata calls.

Suggested fix and tests

For branch-4.0 and branch-4.1:

• preserve Optional.empty() in the existing schema cache as a negative entry;
• reject the missing schema at the JDBC table schema-consumption boundary with an actionable JdbcClientException;
• cover both getFullSchema() and inherited getBaseSchema() paths;
• keep PreloadExternalMetadata generic and unchanged.

For master, adapt the same contract to PluginDrivenExternalTable and the unified metadata cache, preferably with an explicit missing-schema result or bounded negative-cache representation rather than uncached loader exceptions.

Tests should cover:

• null and empty JDBC column metadata;
• missing connector table handle;
• preload enabled and disabled;
• JDBC-only and mixed internal/JDBC queries;
• repeated reads do not repeat remote metadata access while a negative entry is valid;
• normal non-empty schemas remain unchanged.

Anything else?

This report distinguishes the exact NPE on branch-4.x from the current master behavior after #66633. The underlying cross-version issue is the lack of a consistent JDBC missing-schema contract and bounded negative caching.

Are you willing to submit PR?

• Yes, I am willing to submit a PR.

Code of Conduct

• I agree to follow this projects Code of Conduct.

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_ba4943dd2476

Initial assessment

This is a credible FE/JDBC metadata error-contract issue, but the cross-version description needs one important correction before implementation. The issue is currently unlabelled and unassigned; it should be routed to the FE external-catalog/JDBC maintainers.

Code-verified facts

• branch-4.0 (8a99617): the reported NPE chain is valid. JdbcExternalTable.initSchema() returns Optional.empty() for null/empty columns; ExternalSchemaCache stores that Optional value; ExternalTable.getFullSchema() converts it to null; and LogicalCatalogRelation.computeOutput() calls .stream() without a null check. The negative result remains cached subject to the schema-cache expiry/refresh policy.
• branch-4.1 (6f4c6a4): the reported empty-schema NPE/negative-cache behavior is not the normal behavior at this SHA. This commit already contains the branch-4.1 version of [refactor](catalog) Unify external metadata cache framework #66633 (9a7174edb22). DefaultExternalMetaCache.loadSchemaCacheValue() converts ExternalCatalog.getSchema(...)=Optional.empty() into CacheException. Consequently, null/empty JDBC columns fail during the cache load, the failure is not retained as a negative entry, and repeated statements can reload remote metadata. With preload enabled the exception is raised by the preload getBaseSchema() call; without preload it is raised by the later schema access. For this condition, the value does not normally reach LogicalCatalogRelation.getBaseSchema().stream() as null.
• master (feb9e04): a missing connector table handle follows the same generic CacheException/uncached-failure path. A valid handle with an empty ConnectorTableSchema is different: ConnectorTableSchema normalizes null columns to an empty list, PluginDrivenExternalTable creates a present cache value, and SchemaCacheValue.validateSchema() does not reject an empty schema. computeOutput() then streams an empty list; the report does not yet demonstrate the resulting user-visible failure or incorrect plan.
• PreloadExternalMetadata only calls the table metadata APIs. It does not invalidate the schema cache or catch/suppress their exceptions. The repeated-load behavior comes from the failed cache load not producing a cacheable value, not from preload invalidating the entry.

Evidence still needed

1. Full FE exception/stack trace for each claimed SHA, with preload enabled and disabled. In particular, an NPE observed on 6f4c6a4 would indicate another cache route or race and needs a complete stack trace.
2. A deterministic reproducer or unit-test fixture identifying which condition is exercised: legacy listColumns() returns null/empty, connector getTableHandle() returns empty, or a valid handle returns zero columns.
3. JDBC database/driver name and version, metadata permissions, and whether the remote table was concurrently dropped/recreated.
4. Remote-call counters or debug logs for repeated statements, separating table-existence/handle resolution from column-metadata reads, plus the effective schema-cache TTL/capacity settings and any explicit REFRESH operations.
5. For the master zero-column case, the actual analysis result/stack trace and expected contract. The current code proves that an empty schema is accepted, but not by itself that it produces the same failure as a missing handle.

Recommended next steps

1. Correct the issue's version matrix: branch-4.0 has the nullable-schema NPE and cached absence; branch-4.1 and master have generic uncached loader failures for missing schema/handle after [refactor](catalog) Unify external metadata cache framework #66633.
2. Add failing tests first for the three distinct cases above. Assert exception type/message, preload on/off parity, and remote-call count across repeated statements. Keep a normal non-empty schema control case.
3. Fix the branches separately:
   • On branch-4.0, preserve the bounded cached absence but convert it at the JDBC table schema-consumption boundary into an actionable JdbcClientException containing catalog and remote database/table names; no required schema API should return null.
   • On branch-4.1 and master, an override that only checks getFullSchema() is insufficient because the unified loader throws before returning. Introduce a non-null, bounded negative result (or a dedicated failure cache/result type) with expiry and explicit-refresh invalidation, then translate it to the contextual JDBC error at the consumption boundary.
   • On master, explicitly define whether a normal JDBC table may ever have zero columns. If not, reject that state with JDBC table context before publishing a normal schema cache value; avoid applying a blanket rule to connector paths that intentionally use empty schemas unless their contracts are checked.
4. Keep PreloadExternalMetadata generic unless a separate preload-specific defect is reproduced; the schema provider/cache boundary should own consistent error semantics.

No code changes were made during this triage.