diff --git a/CHANGELOG.md b/CHANGELOG.md index 4a0380734..d37b0b861 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,8 @@ ### Under the Hood +- Document serverless environment configuration for Python models (thanks @TangoEnSkai!) ([#1649](https://github.com/databricks/dbt-databricks/pull/1649) resolves [#1055](https://github.com/databricks/dbt-databricks/issues/1055)) + - Raise the `pytest-rerunfailures` lower bound to `>=16.2` and remove the `SchemaNameVarMixin` workaround so min-deps CI no longer pins 14.0, which leaked class-scoped dbt test fixtures across reruns (test-only, no runtime impact) ([#1618](https://github.com/databricks/dbt-databricks/pull/1618)) - Bump `databricks-sql-connector` ceiling to `<4.4.1` and pin to 4.4.0, which requires `thrift>=0.24.0`; resolves CVE-2026-48586 (data amplification DoS), CVE-2026-41603 (TLS cert hostname bypass), and CVE-2026-43868 (memory allocation) ([#1623](https://github.com/databricks/dbt-databricks/pull/1623) resolves [#1622](https://github.com/databricks/dbt-databricks/issues/1622)) - Reorganize `docs/` into `docs/flow/` and `docs/guides/`, and sync materialization flow diagrams with the current macros ([#1627](https://github.com/databricks/dbt-databricks/pull/1627)) diff --git a/README.md b/README.md index 7b1762fad..6d9ccef53 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,7 @@ These following quick starts will get you up and running with the `dbt-databrick - [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project) - Using dbt Cloud with Databricks ([Azure](https://docs.microsoft.com/en-us/azure/databricks/integrations/prep/dbt-cloud) | [AWS](https://docs.databricks.com/integrations/prep/dbt-cloud.html)) - [Submitting Python models as Databricks Workflows](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/workflow-job-submission.md) +- [Configuring serverless environments for Python models](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/workflow-job-submission.md#serverless-environments) - [Running a dbt project as a Databricks job](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/databricks-jobs.md) - [Using Unity Catalog with dbt-databricks](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/uc.md) - [Continuous integration in dbt](https://docs.getdbt.com/docs/deploy/continuous-integration) diff --git a/docs/README.md b/docs/README.md index 8dc9ed636..2fe3e8950 100644 --- a/docs/README.md +++ b/docs/README.md @@ -32,6 +32,6 @@ drift; treat them as starting points, not authoritative reference. - **[guides/databricks-jobs.md](guides/databricks-jobs.md)** — running a dbt project as a Databricks job. - **[guides/workflow-job-submission.md](guides/workflow-job-submission.md)** — submitting Python - models as long-lived Databricks Workflows. + models as long-lived Databricks Workflows, including serverless environment configuration. - **[guides/databricks-copy-into-macro-aws.md](guides/databricks-copy-into-macro-aws.md)** — loading S3 data into Delta with the `databricks_copy_into` macro. diff --git a/docs/guides/workflow-job-submission.md b/docs/guides/workflow-job-submission.md index 9d4a4c187..28d175a99 100644 --- a/docs/guides/workflow-job-submission.md +++ b/docs/guides/workflow-job-submission.md @@ -8,6 +8,9 @@ method, but allow for additional configuration. Some of that configuration can also be used for `job_cluster` models. +For one-time serverless runs instead of a long-lived workflow, see +[Serverless environments](#serverless-environments). + ```python # my_model.py import pyspark.sql.types as T @@ -110,6 +113,52 @@ dbt will generate a name based on the catalog, schema, and model identifier. - Similarly, you can define a reusable job cluster for the workflow and tell the task to use that - If none of those are in the configuration, the task cluster will be serverless +#### Serverless environments + +Python models submitted with `serverless_cluster` can install dependencies in a +[Databricks serverless environment](https://docs.databricks.com/aws/en/compute/serverless/dependencies). +Set both `environment_key` and `environment_dependencies` in the model's YAML configuration: + +```yaml +version: 2 + +models: + - name: my_python_model + config: + submission_method: serverless_cluster + environment_key: dbt_env + environment_dependencies: + - pandas==2.2.3 + - /Workspace/Shared/libraries/my_package-1.0.0-py3-none-any.whl +``` + +`environment_key` assigns the environment to the Python model task. When +`environment_dependencies` is also set, dbt-databricks creates a serverless environment using +environment version `4` and installs each listed PyPI package or workspace file. + +To control other environment settings, define the full Jobs API environment in +`python_job_config.environments`. The key used by the task must match the key in that list: + +```yaml +version: 2 + +models: + - name: my_python_model + config: + submission_method: serverless_cluster + environment_key: dbt_env + python_job_config: + environments: + - environment_key: dbt_env + spec: + environment_version: "4" + dependencies: + - pandas==2.2.3 +``` + +When `python_job_config.environments` contains an environment, it takes precedence over the +environment that dbt-databricks would generate from `environment_dependencies`. + ```yaml # Reusable job cluster config example