Allow configuring cascade deletion in DABs#5846
Conversation
Add a `cascade` field to the pipeline bundle resource that controls whether destroying a pipeline also deletes its datasets (materialized views, streaming tables, and views). When unset, the server default applies (cascade); set `cascade: false` to retain the datasets on destroy. The field is delete-time only: it is not part of the pipeline spec, so it is never sent on create/update. The direct engine persists it in state (via a PipelineState wrapper around CreatePipeline, mirroring the sql_warehouse lifecycle pattern) and reads it at delete time, force-sending cascade so an explicit false survives query-string omitempty (honored for query params as of databricks-sdk-go v0.152.0). A cascade-only change is a state-only update with no pipeline API call. cascade is classified input_only so it does not show remote drift. The terraform engine drops the field for now: the pinned provider has no such attribute yet (pending terraform-provider-databricks#5860), so terraform-engine support is a follow-up. Co-authored-by: Isaac
Approval status: pending
|
| pipelines: | ||
| my_pipeline: | ||
| name: test-pipeline-cascade | ||
| cascade: false |
There was a problem hiding this comment.
Could you make sure the default template also sets purge_on_destroy: false? With a very short comment to the side explaining it? Could be a separate PR.
There was a problem hiding this comment.
ack will put this change in a follow-up PR.
… message fix - Rename the pipeline config field cascade -> cascade_on_destroy, matching the Lakebase purge_on_delete precedent (per review discussion). - Add ValidateCascadeOnDestroy mutator: hard error when the field is used with the terraform engine, which does not support it yet. - Fix the destroy approval message so a pipeline with cascade_on_destroy: false is not described as deleting its STs/MVs, and mention the property otherwise. - Comment why ForceSendFields is needed in DoDelete. - Regenerate schema, refschema, and affected acceptance outputs. Co-authored-by: Isaac
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
| pipelines: | ||
| my_pipeline: | ||
| name: test-pipeline-cascade | ||
| cascade: false |
There was a problem hiding this comment.
ack will put this change in a follow-up PR.
2 participants
Changes
Add a
cascadefield to the pipeline bundle resource that controls whether destroying a pipeline also cascades to its datasets. When unset, we maintain the existing default of true.This change applies to the direct engine. We have a separate change on the Terraform side: databricks/terraform-provider-databricks#5860
Why
Previously, pipeline deletion would automatically cascade to its tables. After user feedback, we decided to make this configurable in the DeletePipeline API. This extends support to DABs as well, which was another common user ask.
Tests
Unit tests