[DSIP-110][Datasource] Support Custom Datasource Types via External Plugins

[eye-gu]

Search before asking

• I had searched in the DSIP and found no similar DSIP.

Functional requirement origin: #18377 "[Improvement][datasource] Enable users to customize datasources".

Motivation

DolphinScheduler's datasource layer is pluginized in name but not in fact. Although datasource support is implemented as SPI plugins (DataSourceProcessor / DataSourceChannelFactory, 28 built-in plugins), the plugin contract is hard-wired to the compiled-in DbType enum (dolphinscheduler-spi/.../spi/enums/DbType.java):

• DataSourceProcessor#getDbType() returns DbType, a compile-time enum shipped inside dolphinscheduler-spi. A plugin compiled outside the DolphinScheduler tree can only return an existing constant — it can override a built-in type, but can never introduce a new one.
• BaseDataSourceParamDTO#getType() returns DbType as well, so the parameter DTOs have the same constraint.
• The type column of t_ds_datasource stores the enum's integer code (tinyint), which by definition cannot represent an unknown type.
• Both frontend surfaces — the "create datasource" dialog (dolphinscheduler-ui/src/views/datasource/list/use-form.ts) and the task-node datasource selector (.../node/fields/use-datasource.ts) — enumerate types from hardcoded arrays; a type installed on the backend is invisible to users.

As a result, anyone whose database is not among the built-in types (proprietary engines, internal database proxies, newly released databases) has no supported extension path short of forking the project, editing the enum, the DAO layer and the frontend, and rebuilding everything.

Goal (functional requirement) — a deployment administrator can install an externally compiled datasource plugin package into an existing DolphinScheduler installation, without modifying or recompiling any DolphinScheduler code, and the new datasource type becomes usable end-to-end:

• FR1 — Registerable: the plugin can declare a datasource type identity that is unique and resolvable across api-server, master and worker.
• FR2 — Visible: the new type shows up in the UI, both in the datasource creation entry and in the datasource selector of task nodes that consume datasources.
• FR3 — Full lifecycle: for the new type, users can create a datasource instance, test connectivity, update it, list/browse it, grant it to other users, and (where the driver supports it) browse metadata (databases / tables / columns).
• FR4 — Executable: tasks that reference datasource instances of the new type run correctly (minimum scope: SQL and Procedure tasks).
• FR5 — Install-only rollout: installation = drop the plugin package into the plugins directory and restart the servers; no source change, no rebuild of DolphinScheduler itself.

Non-goals (proposed for v1):

• No plugin hot-loading / installing plugins through the web UI. Plugin installation stays an administrator/deployment action (see Security alignment below).
• No plugin-provided custom UI forms. Custom types get a generic JDBC-style form (see open decision D3).
• No DataX / Sqoop support for custom types in v1 (their type→generator mappings are hardcoded; see Aspect 5).

Design Detail

The requirement decomposes into the following aspects. Each aspect states the current code fact that blocks the requirement, and what has to be true after the change.

Aspect 0 — Current state summary (why each aspect exists)

#	Aspect	Blocking fact today
1	Type identity in the SPI contract	getDbType() / getType() return compiled-in DbType; registry keyed by enum name
2	Type discovery	No API exposes registered types; frontend hardcodes arrays
3	Persistence	t_ds_datasource.type is tinyint storing the enum code; entity field is DbType
4	api-server handling	Routing and dialect behavior switch on DbType (DbType.ofName, valueOf, switch statements)
5	Task execution path	SqlTask/ProcedureTask/DataxTask do DbType.valueOf(...); context objects carry DbType
6	Frontend	Type lists, per-type forms and default ports are hardcoded in use-form.ts / use-datasource.ts / types.ts
7	Packaging & driver delivery	Works today (shaded jars on the classpath via plugins/datasource-plugins), needs conventions for external plugins
8	Security alignment	Plugin install must stay inside the existing trusted-deployment boundary

Aspect 1 — String-based type identity in the plugin contract

The SPI must stop using a compile-time enum as the identity of a datasource type.

• A datasource type is identified by a unique, case-normalized string name (e.g. "MYSQL", "DORIS", "MY_INTERNAL_DB"), declared by the plugin and used consistently by DataSourceProcessor, DataSourceChannelFactory and the parameter DTOs.
• DbType degrades from "the closed set of all possible types" to "named constants for the built-in types" (see open decision D1). All identity lookups (DataSourcePluginManager maps, DataSourceUtils routing) become string-keyed; today they already key maps by dbType.getName(), so the direction of change is natural.
• Conflict rules: two installed plugins declaring the same type name must fail fast at startup with a clear message. The existing PrioritySPI mechanism (same name resolved by priority, ambiguity raises) already provides the semantics for deliberately overriding a built-in type; new names simply must not collide.
• BaseDataSourceParamDTO carries the type as a string field, replacing getType(): DbType, so JSON round-tripping works for types unknown to the core.

Aspect 2 — Type registry and a discovery API

The backend must be able to enumerate what is installed, and the frontend must learn the type list from the backend instead of hardcoding it.

• DataSourcePluginManager (or a thin façade over it) exposes the registered types together with minimal metadata a UI needs: unique name, display label, and simple capability flags (e.g. supportsConnectivityTest, supportsMetadata, jdbcCompatible).
• A new REST endpoint, e.g. GET /datasources/types, returns that list. This is the single contract between backend plugins and both frontend surfaces (Aspect 6).
• Create/update APIs validate the submitted type against the registry and reject unknown types with an explicit error message ("datasource type X is not installed") instead of failing deep inside enum parsing (DbType.ofName currently throws NoSuchElementException for unknown names).
• Capability flags replace enum switches where behavior varies per type (see Aspect 4), so a custom type gets sane defaults without core code knowing it.

Aspect 3 — Persistence of the datasource type

t_ds_datasource.type must be able to store identities that do not exist at compile time.

• Change the column from tinyint (integer enum code) to varchar storing the type name string, in all three fresh-install schemas (MySQL, PostgreSQL, H2: dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_{mysql,postgresql,h2}.sql) and keep the UNIQUE (name, type) constraint semantics.
• The DataSource entity (dolphinscheduler-dao/.../entity/DataSource.java) changes its type field from DbType to String; mappers and repositories follow.
• Upgrade scripts under dolphinscheduler-dao/src/main/resources/sql/upgrade/<version>_schema/{mysql,postgresql}/ convert existing integer codes to name strings (a code→name mapping over the current 29 enum values; see Migration Plan). The stale column comment ("0:mysql,1:postgresql,...") is corrected at the same time.
• Graceful degradation: a datasource row whose type plugin is not installed must still be listable (the UI shows the raw type name and marks it unavailable); connect/run attempts return a clear "plugin not found" error rather than an NPE or enum parse failure. The message "datasource plugin '%s' is not found" already exists in DataSourceClientProvider and should become the uniform behavior.

Aspect 4 — api-server adaptations

• Routing: DataSourceUtils.buildDatasourceParam routes the submitted JSON to the right processor by string type name (today: DbType.ofName(type.toUpperCase())).
• API surface: DataSourceController#queryDataSourceList binds type as DbType (Spring enum-name conversion). Binding it as String keeps the wire format ("MYSQL") unchanged — the frontend already sends enum names, so this is not a breaking API change.
• Dialect behavior: type-specific switches such as DataSourceServiceImpl#getDbSchemaPattern (HIVE/ORACLE/SQLSERVER/CLICKHOUSE/DATABEND/PRESTO) and the AbstractDataSourceProcessor REDSHIFT special case must become capability/metadata-driven (Aspect 2) with safe defaults, so unknown types get working default behavior for metadata browsing instead of being silently wrong.
• Unchanged by design: password encryption (PasswordUtils), Kerberos handling, and datasource authorization are already type-agnostic (permissions bind to datasource ids).

Aspect 5 — Task execution path (master + worker)

Good news first: task definitions already store the datasource type as a string in task_params JSON (SqlParameters.type, DataxParameters.dsType/dtType are String), and datasource instances are referenced by integer id. The work is removing enum round-trips:

• master: TaskExecutionContextFactory#assembleDataSourceParameters puts a DbType into the execution context; context objects (DataxTaskExecutionContext, SqoopTaskExecutionContext, ...) carry DbType fields — all become strings.
• worker task plugins: SqlTask (DbType.valueOf(sqlParameters.getType())), ProcedureTask, and the DataX/Sqoop call sites resolve the processor by string type name via the plugin manager. Unknown type → clear task failure message naming the missing plugin.
• DataX / Sqoop scope: DataxUtils#getReaderPluginName/getWriterPluginName and the Sqoop source/target generators hardcode type→external-tool mappings that only make sense for built-in types. In v1 these tasks declare themselves limited to built-in types (the frontend already supports a per-task-type whitelist via supportedDatasourceType), so custom types are selectable only for SQL / Procedure tasks. Lifting this restriction is a follow-up, not part of this DSIP.

Aspect 6 — Frontend: dynamic type list + generic fallback form

• Type list source: both hardcoded arrays — datasourceTypeList in views/datasource/list/use-form.ts (28 entries with per-type labels/default ports) and datasourceTypes in node/fields/use-datasource.ts (an inconsistent subset, some entries disabled) — are replaced by data from GET /datasources/types (Aspect 2). Per-task-type whitelisting (supportedDatasourceType) continues to filter the dynamic list.
• Forms for custom types: built-in types keep their dedicated forms (changeType in use-form.ts). A type the frontend has no dedicated form for renders a generic JDBC form: host, port, user, password, database, extra JDBC parameters (other), optional principal — the fields the common BaseConnectionParam already supports. This is sufficient for the overwhelming majority of JDBC drivers; the display label and default port come from the type metadata.
• Type definitions: the IDataBase string-literal union in service/modules/data-source/types.ts becomes a plain string validated at runtime against the types API.
• i18n / labels: labels for built-in types keep current texts; custom types display the label from the API metadata (no per-type i18n keys needed for custom types).

Aspect 7 — Packaging, delivery and driver management

The deployment path already works mechanically (shaded plugin jars in plugins/datasource-plugins/ are appended to the classpath by the server start scripts, discovered via ServiceLoader); what this DSIP adds is convention and guarantees:

• A custom plugin is a single shaded jar bundling: the DataSourceProcessor + DataSourceChannelFactory implementations, META-INF/services registrations (e.g. via @AutoService), and the JDBC driver it needs (same pattern as built-in plugin poms, documented in docs/docs/en/contribute/backend/spi/datasource.md).
• The plugin must be installed on every api-server, master and worker node that will touch the type (api for UI/lifecycle, master+worker for execution); the developer/operations documentation states this explicitly.
• Driver version conflicts between plugins remain the known limitation of the flat-classpath model; document it as a constraint (one driver major version per classpath), same as today's behavior for built-in plugins. Classloader isolation is explicitly out of scope.
• Reference plugin: DbType contains H2(9) but no H2 plugin module exists. Implementing the H2 datasource plugin as the first "externally developed" plugin gives the proposal a living example that is testable in CI with zero external services (in-memory JDBC), and doubles as the test vehicle for the whole path (registry → UI → SQL task execution).

Aspect 8 — Security alignment

This feature must stay within the trust boundaries documented in the project's security model:

• Installing a plugin jar is an administrator/deployment action (filesystem access to the server), identical in trust level to installing task plugins or storage plugins today. Ordinary platform users cannot install plugins; they can only use an installed type within existing datasource authorization. The feature adds no new privilege escalation path.
• Parameters a user enters into a custom datasource (including JDBC other parameters) fall squarely under the model's existing "user-configured plugin parameters are trusted user behavior" clause.
• Registry conflict handling (Aspect 1) must fail fast and loudly, so a malicious/accidental same-name plugin cannot silently override a built-in type without priority rules catching it.

Open design decisions (for the mail-thread discussion)

• D1 — Fate of DbType: (a) recommended: keep the enum as a deprecated constants holder for migration/compat, remove its integer code from persistence, all runtime paths use strings — mirrors how task plugins use String taskType; (b) keep full enum semantics for built-ins and add a parallel string path — rejected as dual-maintenance.
• D2 — Storage: (a) recommended: in-place type column conversion tinyint → varchar(64) with data migration; (b) side table type_code → type_name mapping — rejected (extra join, worse ergonomics, still can't express custom types in the main column).
• D3 — UI for custom types: (a) recommended for v1: generic JDBC fallback form; (b) plugin-declared form schema returned by the types API (field list rendered dynamically) — more powerful, deferred to a follow-up DSIP.
• D4 — Task coverage in v1: SQL + Procedure only; DataX/Sqoop restricted to built-ins via existing whitelist mechanism.
• D5 — Types API shape: GET /datasources/types returning [{ type, label, defaultPort?, capabilities... }] — exact field list to be settled in review.

Suggested sub-task breakdown

Consistent with the DSIP process, this is large enough to be split:

1. SPI contract: string type identity in DataSourceProcessor / DTOs / DataSourcePluginManager (+ conflict rules).
2. DAO: DataSource entity, mappers, fresh-install schemas for MySQL/PG/H2.
3. Upgrade scripts (code→name migration, MySQL + PostgreSQL) and tools support.
4. api-server: string routing, GET /datasources/types, unknown-type validation, capability-driven dialect defaults.
5. master/worker: execution context and SQL/Procedure/DataX/Sqoop call-site adaptation.
6. Frontend: dynamic type lists, generic fallback form, whitelist filtering.
7. Reference H2 plugin + developer guide (contribute/backend/spi/datasource.md en/zh) + ops doc (install on all nodes, driver conflicts).
8. Tests & e2e (see Test Plan).

Compatibility, Deprecation, and Migration Plan

Data migration

• t_ds_datasource.type: tinyint/int → varchar, one upgrade script per dialect (MySQL, PostgreSQL) in the upgrade directory of the release carrying this change, following the existing t_ds_version-driven UpgradeDao flow. The DML maps each of the 29 live integer codes to its name constant (0→MYSQL, 1→POSTGRESQL, ..., 28→DOLPHINDB). Rows referencing codes with no enum value (corrupted data) are reported rather than silently converted.
• task_params in t_ds_task_definition / t_ds_task_instance already stores type names as strings and is not touched.
• connection_params JSON is plugin-owned and unchanged.

API compatibility

• Wire formats that already use enum names (e.g. GET /datasources/list?type=MYSQL) remain byte-compatible; only server-side binding types change.
• Java-level breakage is confined to internal modules (dolphinscheduler-spi, dolphinscheduler-datasource-api, dolphinscheduler-dao) — this is a dev-branch feature release, but downstream plugin authors are affected and must be informed via migration notes: existing third-party plugins compiled against getDbType() need a trivial recompile against the new contract.

Deployment / rolling upgrade

• All servers (api, master, worker) must be upgraded together in a cluster that uses custom types; a node missing a plugin degrades gracefully (Aspect 3) — datasource lists still render, connect/run fail with an explicit "plugin not installed" error.
• Built-in-only deployments keep working identically after migration; no user action required.

Deprecations

• DbType integer codes and @EnumValue persistence deprecated; enum retained (D1) as constants for one removal cycle.

Test Plan

Unit tests

• DataSourcePluginManager: string-keyed registration, duplicate-name fail-fast, priority override, missing-plugin lookup error.
• DataSourceUtils / processor routing by string type; unknown type rejected with explicit error.
• Migration converter: every one of the 29 codes maps to the expected name; unknown code surfaces an error.
• Controller binding: type accepts any registered name; unregistered names produce a 4xx with a clear message.

Integration / DAO tests

• Upgrade test from the previous schema with rows covering (at least) MYSQL, POSTGRESQL, HIVE, ORACLE, SSH, DOLPHINDB — assert post-migration names and uniqueness.
• api-server: create / update / connect-test / list / authorize a datasource of a custom test processor; datasource rows with an uninstalled type list correctly and fail connect with the documented error.

Worker / execution tests

• SqlTask and ProcedureTask against the reference H2 plugin (in-memory): full create → bind → execute path with a custom type.
• Failure-path test: task referencing an uninstalled type fails with the "plugin not found" message.

Frontend / e2e

• dolphinscheduler-e2e: create an H2 datasource through the generic fallback form, bind it to an SQL task node, run the workflow successfully.
• Regression: existing datasource e2e suites (mysql, postgresql, hive, clickhouse, sqlserver, dolphindb docker fixtures) pass unchanged.

Regression scope

• All 28 built-in plugins' existing processor/channel unit tests pass unmodified beyond the contract recompile.
• Built-in UI forms render exactly as before (no visual/behavioral change for known types).

Code of Conduct

• I agree to follow this project's Code of Conduct

[SbloodyS]

-1 from my side.

I have concerns about the current proposal because it seems to assume that a datasource is essentially JDBC-based, while this is not true for the existing datasource ecosystem in DolphinScheduler.

Besides JDBC-based datasource types, we already have (and may introduce more) datasource types whose connection model, parameters, metadata operations, and runtime behavior cannot be represented by a generic JDBC abstraction.

For example, the proposed generic JDBC-style fallback form (host, port, user, password, database, JDBC parameters, etc.) only works for JDBC-compatible datasources. The same assumption also appears in several other parts of the proposal, including connectivity testing, metadata browsing, driver packaging, and task execution.

This means that introducing a generic "custom datasource plugin" mechanism based primarily on the JDBC model may create significant compatibility and extensibility issues. We would likely need additional special cases later for every non-JDBC datasource type, which defeats the purpose of making the datasource framework truly pluggable.

I think before changing the core datasource SPI, persistence model, frontend, and execution path, we should first define a datasource abstraction that can cover both JDBC and non-JDBC datasource types.

[eye-gu]

@SbloodyS Thanks for the feedback. Let me keep it simple and state the scope clearly:

The datasource SPI contract stays as it is. The only change is the type identifier — DbType becomes a String — nothing else in the contract changes.

Most importantly, non-JDBC datasource plugins such as SSH and K8s remain unchanged — same implementation, same behavior, same registration. They already work under this contract today; the only mechanical touch is the identifier's return type. That is exactly the point: the contract does not need an abstraction change for this proposal.

A custom datasource plugin — JDBC or not — brings its own implementation; what this DSIP removes is only the compile-time enum that prevents an outside plugin from introducing a new type instead of just overriding a built-in one.

Does that address your concern?