Do not run post-materialization OPTIMIZE for auto_liquid_cluster

[sd-db]

Describe the issue

When a model uses auto_liquid_cluster: true, dbt-databricks emits CLUSTER BY AUTO and then runs a synchronous plain OPTIMIZE <relation> after materialization.

The current behavior is implemented in optimize.sql.

Databricks documents that:

Automatic liquid clustering requires predictive optimization for automatic key selection and clustering operations, and runs asynchronously.

Databricks also states:

Predictive optimization automatically runs OPTIMIZE commands for enabled tables. When using Predictive Optimization, Databricks recommends disabling any scheduled OPTIMIZE jobs.

Sources:

• Automatic liquid clustering
• How to trigger clustering
• dbt-databricks configurations

This raises a product ownership question: should auto_liquid_cluster delegate key selection and clustering operations to Predictive Optimization, or should dbt also trigger a synchronous OPTIMIZE?

Suggested behavior

auto_liquid_cluster should emit CLUSTER BY AUTO without triggering a post-materialization OPTIMIZE by itself.

The existing behavior for zorder and explicit liquid_clustered_by configurations should remain unchanged. Users who require immediate manual optimization can add an explicit post-hook.

Reproduction

{{ config(
    materialized='table',
    auto_liquid_cluster=true
) }}

select 1 as id

Running this model creates a table with CLUSTER BY AUTO and issues a plain OPTIMIZE during the same dbt run.