From f069128e9822819f21e293ff1faa965b29f78727 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 14 Sep 2026 09:31:11 -0600 Subject: [PATCH 1/2] ci: fold the Delta build gate and PyArrow UDF suite into the merge queue tiers Both workflows predate the merge queue and still carry their own push/pull_request triggers, so every merge runs them twice: once in the queue and again on push to main. Make them workflow_call reusables invoked from ci.yml, with routing in FILTERS and POLICY like every other heavy job. Both sit in the queue tier with an opt-in label (run-delta-build-gate, run-pyarrow-udf-tests), and feed Required Checks so a failure still blocks the merge. The Delta gate's path filter now covers only main sources and build inputs, since the script inspects build output and never runs tests. --- .github/workflows/README.md | 19 +++++--- .github/workflows/ci.yml | 21 ++++++++- .github/workflows/delta_build_gate.yml | 32 ++----------- .github/workflows/pyarrow_udf_test.yml | 38 ++-------------- dev/ci/check-ci-config.py | 44 ++++++++++++++++-- dev/ci/compute-changes.py | 63 ++++++++++++++++++++++++++ docs/source/contributor-guide/ci.md | 4 ++ 7 files changed, 146 insertions(+), 75 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 43b5be5a5c0..6a93a8e6a06 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -17,9 +17,10 @@ ruleset in `.asf.yaml`. That splits CI into two tiers: - **PR tier** (`pr`): fast feedback while a change is being iterated on. The Linux build, Spark 4.1 (catalyst and `sql_core` only) and Iceberg 1.11. - **Queue tier** (`queue`): the authoritative gate. Everything the PR tier - runs, plus the macOS build, the benchmark compile check, the Spark 4.1 - `sql_hive` shards, Spark 3.4/3.5/4.0 and Iceberg 1.8/1.9/1.10, evaluated - against the merge result rather than against the PR head. + runs, plus the macOS build, the benchmark compile check, the Delta contrib + build gate, the PyArrow UDF suite, the Spark 4.1 `sql_hive` shards, Spark + 3.4/3.5/4.0 and Iceberg 1.8/1.9/1.10, evaluated against the merge result + rather than against the PR head. Every queue-only job has a `run-*` label that opts a pull request into it early, listed in the diagram below. @@ -60,7 +61,9 @@ and TPC-H/TPC-DS caches on `main` stale until the next unrelated change. --------------- ----------------- --------------------------- pr_build_linux (+ push, for cache) docs pr_build_macos run-macos-tests spark_4_1 (catalyst + sql_core) pr_benchmark_check run-benchmark-check - iceberg_1_11 spark_4_1 sql_hive run-spark-4.1-hive-tests + iceberg_1_11 delta_build_gate run-delta-build-gate + pyarrow_udf_test run-pyarrow-udf-tests + spark_4_1 sql_hive run-spark-4.1-hive-tests spark_3_4 run-spark-3.4-tests spark_3_5 run-spark-3.5-tests spark_4_0 run-spark-4.0-tests @@ -80,8 +83,8 @@ and TPC-H/TPC-DS caches on `main` stale until the next unrelated change. reusable workflows invoked via `uses:`: pr_build_linux.yml spark_sql_test_reusable.yml pr_build_macos.yml iceberg_spark_test_reusable.yml - pr_benchmark_check.yml - docs.yaml + pr_benchmark_check.yml delta_build_gate.yml + docs.yaml pyarrow_udf_test.yml ``` ## What runs when @@ -93,6 +96,8 @@ and TPC-H/TPC-DS caches on `main` stale until the next unrelated change. | `pr_build_linux` | PR, merge group or push to main, paths matched | `dev/ci/compute-changes.py` | | `pr_build_macos` | merge group, **or** PR with `run-macos-tests` | `dev/ci/compute-changes.py` | | `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only | +| `delta_build_gate` | merge group, **or** PR with `run-delta-build-gate` | main sources, poms, `contrib/delta` | +| `pyarrow_udf_test` | merge group, **or** PR with `run-pyarrow-udf-tests` | map-in-batch and Python runner code | | `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` | | `spark_3_5` | merge group, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources | | `spark_4_1` | PR or merge group, paths matched; the `sql_hive` shards only in the merge group **or** with `run-spark-4.1-hive-tests` | Spark 4.1 sources | @@ -166,6 +171,8 @@ umbrella doesn't watch, or operate independently of the rest of CI: | `pr_build_linux.yml` | `pr_build_linux` | | `pr_build_macos.yml` | `pr_build_macos` | | `pr_benchmark_check.yml` | `pr_benchmark_check` | +| `delta_build_gate.yml` | `delta_build_gate` | +| `pyarrow_udf_test.yml` | `pyarrow_udf_test` | | `docs.yaml` | `docs` | | `spark_sql_test_reusable.yml` | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1` | | `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10`, `iceberg_1_11` | diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9904c2760b6..213e67351d7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -25,7 +25,8 @@ # pull_request fast feedback. Linux build + Spark 4.1 (catalyst and # sql_core only) + Iceberg 1.11. # merge_group the authoritative gate. Everything the PR tier runs, plus -# the macOS build, the benchmark compile check, the Spark 4.1 +# the macOS build, the benchmark compile check, the Delta +# contrib build gate, the PyArrow UDF suite, the Spark 4.1 # sql_hive shards, Spark 3.4/3.5/4.0 and Iceberg 1.8/1.9/1.10, # evaluated against the merge result rather than the PR head. # @@ -156,6 +157,8 @@ jobs: build_linux: ${{ steps.compute.outputs.build_linux }} build_macos: ${{ steps.compute.outputs.build_macos }} benchmark: ${{ steps.compute.outputs.benchmark }} + delta_gate: ${{ steps.compute.outputs.delta_gate }} + pyarrow_udf: ${{ steps.compute.outputs.pyarrow_udf }} docs: ${{ steps.compute.outputs.docs }} spark_3_4: ${{ steps.compute.outputs.spark_3_4 }} spark_3_5: ${{ steps.compute.outputs.spark_3_5 }} @@ -245,6 +248,20 @@ jobs: if: needs.changes.outputs.benchmark == 'true' uses: ./.github/workflows/pr_benchmark_check.yml + delta_build_gate: + name: Delta Contrib Build Gate + needs: changes + # Queue-only by default; PRs need the `run-delta-build-gate` label. + if: needs.changes.outputs.delta_gate == 'true' + uses: ./.github/workflows/delta_build_gate.yml + + pyarrow_udf_test: + name: PyArrow UDF Tests + needs: changes + # Queue-only by default; PRs need the `run-pyarrow-udf-tests` label. + if: needs.changes.outputs.pyarrow_udf == 'true' + uses: ./.github/workflows/pyarrow_udf_test.yml + docs: name: Deploy Comet site needs: changes @@ -401,6 +418,8 @@ jobs: - pr_build_linux - pr_build_macos - pr_benchmark_check + - delta_build_gate + - pyarrow_udf_test - spark_3_4 - spark_3_5 - spark_4_0 diff --git a/.github/workflows/delta_build_gate.yml b/.github/workflows/delta_build_gate.yml index 245c2d26738..cd7fd2ac19c 100644 --- a/.github/workflows/delta_build_gate.yml +++ b/.github/workflows/delta_build_gate.yml @@ -29,36 +29,10 @@ name: Delta Contrib Build Gate -concurrency: - group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }} - cancel-in-progress: true - +# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency +# live in the umbrella workflow. on: - push: - branches: - - main - paths-ignore: - - "benchmarks/**" - - "doc/**" - - "docs/**" - - "**.md" - - "dev/changelog/*.md" - - "native/core/benches/**" - - "native/spark-expr/benches/**" - - "spark/src/main/scala/org/apache/comet/GenerateDocs.scala" - - "spark-integration/**" - pull_request: - paths-ignore: - - "benchmarks/**" - - "doc/**" - - "docs/**" - - "**.md" - - "dev/changelog/*.md" - - "native/core/benches/**" - - "native/spark-expr/benches/**" - - "spark/src/main/scala/org/apache/comet/GenerateDocs.scala" - - "spark-integration/**" - workflow_dispatch: + workflow_call: permissions: contents: read diff --git a/.github/workflows/pyarrow_udf_test.yml b/.github/workflows/pyarrow_udf_test.yml index d75c5d27744..3bad71adc1b 100644 --- a/.github/workflows/pyarrow_udf_test.yml +++ b/.github/workflows/pyarrow_udf_test.yml @@ -17,41 +17,11 @@ name: PyArrow UDF Tests -concurrency: - group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }} - cancel-in-progress: true - +# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency +# live in the umbrella workflow. The list of files this suite covers is the +# `pyarrow_udf` entry in FILTERS in dev/ci/compute-changes.py. on: - push: - branches: - - main - paths: &feature-paths - - "pom.xml" - - "common/pom.xml" - - "common/src/main/scala/org/apache/comet/CometConf.scala" - - "spark/pom.xml" - - "spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala" - - "spark/src/main/scala/org/apache/spark/sql/comet/CometMapInBatchExec.scala" - - "spark/src/main/scala/org/apache/spark/sql/comet/shims/MapInBatchInfo.scala" - - "spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala" - - "spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala" - - "spark/src/main/spark-4.0/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala" - - "spark/src/main/spark-4.1/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala" - - "spark/src/main/spark-4.2/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala" - - "spark/src/main/spark-4.0/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala" - - "spark/src/main/spark-4.1/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala" - - "spark/src/main/spark-4.2/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala" - - "spark/src/main/spark-4.x/org/apache/spark/sql/comet/shims/Spark4xMapInBatchSupport.scala" - - "spark/src/main/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerBase.scala" - - "spark/src/test/resources/pyspark/conftest.py" - - "spark/src/test/resources/pyspark/test_pyarrow_udf.py" - - "spark/src/test/spark-3.5/org/apache/spark/sql/comet/CometMapInBatchSuite.scala" - - "spark/src/test/spark-4.x/org/apache/spark/sql/comet/CometMapInBatchSuite.scala" - - "spark/src/test/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerSuite.scala" - - ".github/workflows/pyarrow_udf_test.yml" - pull_request: - paths: *feature-paths - workflow_dispatch: + workflow_call: permissions: contents: read diff --git a/dev/ci/check-ci-config.py b/dev/ci/check-ci-config.py index 6ccd22979ae..355ceea66f4 100644 --- a/dev/ci/check-ci-config.py +++ b/dev/ci/check-ci-config.py @@ -93,12 +93,16 @@ "iceberg_1_11", } +# The two contrib/UDF gates also run ./mvnw, but consume no shared artifact. +MVN_JOBS = BUILD_JOBS | {"delta_gate", "pyarrow_udf"} + ROUTING_CASES = [ # The Maven wrapper and its config feed every job that runs ./mvnw: the - # Linux/macOS builds, setup-spark-builder, and the Iceberg `mvnw install`. - ([".mvn/maven.config"], BUILD_JOBS), - ([".mvn/wrapper/maven-wrapper.properties"], BUILD_JOBS), - (["mvnw"], BUILD_JOBS), + # Linux/macOS builds, setup-spark-builder, the Iceberg `mvnw install`, the + # Delta gate's effective-pom check and the PyArrow suite's `mvnw install`. + ([".mvn/maven.config"], MVN_JOBS), + ([".mvn/wrapper/maven-wrapper.properties"], MVN_JOBS), + (["mvnw"], MVN_JOBS), # The artifact wrappers are used by every producer and consumer of a # shared artifact. Without these, an edit confined to one of them routes # to nothing at all and merges having been exercised by no consumer. @@ -109,6 +113,16 @@ # Spot checks that the additions above did not widen unrelated routes. (["docs/source/user-guide/overview.md"], {"docs"}), (["native/core/benches/parquet_read.rs"], {"benchmark"}), + # The Delta gate script is read by nothing else; the contrib crate feeds + # only the gate. The PyArrow pytest lives under spark/, so the Linux and + # macOS builds see it too, but no Spark SQL or Iceberg suite does, and + # neither does the Delta gate, which only inspects build output. + (["dev/verify-contrib-delta-gate.sh"], {"delta_gate"}), + (["contrib/delta/native/src/lib.rs"], {"delta_gate"}), + ( + ["spark/src/test/resources/pyspark/test_pyarrow_udf.py"], + {"build_linux", "build_macos", "pyarrow_udf"}, + ), ] # Event policy. Each case is (event, expected set of jobs allowed to run), @@ -118,7 +132,7 @@ PR_TIER = {"build_linux", "spark_4_1", "iceberg_1_11"} SPARK_OPT_IN = {"spark_3_4", "spark_3_5", "spark_4_0", "spark_4_1_hive"} ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10"} -BUILD_OPT_IN = {"build_macos", "benchmark"} +BUILD_OPT_IN = {"build_macos", "benchmark", "delta_gate", "pyarrow_udf"} QUEUE_TIER = PR_TIER | SPARK_OPT_IN | ICEBERG_OPT_IN | BUILD_OPT_IN ALL_JOBS = QUEUE_TIER | {"docs"} @@ -151,6 +165,26 @@ {"name": "pull_request", "action": "synchronize", "labels": ["run-benchmark-check"]}, PR_TIER | {"benchmark"}, ), + # The Delta build gate and the PyArrow UDF suite were standalone workflows + # that ran on every pull request and again on push to main. Folded in as + # queue-only jobs, each with its own label, they follow the same rules. + ( + {"name": "pull_request", "action": "synchronize", "labels": ["run-delta-build-gate"]}, + PR_TIER | {"delta_gate"}, + ), + ( + {"name": "pull_request", "action": "synchronize", "labels": ["run-pyarrow-udf-tests"]}, + PR_TIER | {"pyarrow_udf"}, + ), + ( + { + "name": "pull_request", + "action": "labeled", + "label": "run-pyarrow-udf-tests", + "labels": ["run-pyarrow-udf-tests"], + }, + {"pyarrow_udf"}, + ), ( { "name": "pull_request", diff --git a/dev/ci/compute-changes.py b/dev/ci/compute-changes.py index 3380a1e4436..f187c3761fd 100644 --- a/dev/ci/compute-changes.py +++ b/dev/ci/compute-changes.py @@ -94,6 +94,62 @@ "native/spark-expr/benches/**", "spark/src/test/scala/org/apache/spark/sql/benchmark/**", ], + # dev/verify-contrib-delta-gate.sh proves the default cargo, Maven and + # libcomet builds carry no Delta surface and that the gated build does. + # It reads the cargo tree, the effective pom, the compiled classes and the + # dylib symbol table: main sources and build inputs, never tests. + "delta_gate": [ + "native/**", + "common/src/main/**", + "spark/src/main/**", + "contrib/delta/**", + "pom.xml", + "**/pom.xml", + ".mvn/**", + "mvnw", + "Makefile", + "rust-toolchain.toml", + "dev/verify-contrib-delta-gate.sh", + ".github/workflows/ci.yml", + ".github/workflows/delta_build_gate.yml", + ".github/actions/setup-builder/**", + "!**.md", + "!native/core/benches/**", + "!native/spark-expr/benches/**", + "!spark/src/main/scala/org/apache/comet/GenerateDocs.scala", + ], + # A real Python worker against each Spark 4.x Arrow runner. The list is + # deliberately narrow: the suite builds Comet three times, once per Spark + # version, and only the map-in-batch wiring can change its verdict. + "pyarrow_udf": [ + "pom.xml", + "common/pom.xml", + "common/src/main/scala/org/apache/comet/CometConf.scala", + "spark/pom.xml", + "spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala", + "spark/src/main/scala/org/apache/spark/sql/comet/CometMapInBatchExec.scala", + "spark/src/main/scala/org/apache/spark/sql/comet/shims/MapInBatchInfo.scala", + "spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala", + "spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala", + "spark/src/main/spark-4.0/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala", + "spark/src/main/spark-4.1/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala", + "spark/src/main/spark-4.2/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala", + "spark/src/main/spark-4.0/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala", + "spark/src/main/spark-4.1/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala", + "spark/src/main/spark-4.2/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala", + "spark/src/main/spark-4.x/org/apache/spark/sql/comet/shims/Spark4xMapInBatchSupport.scala", + "spark/src/main/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerBase.scala", + "spark/src/test/resources/pyspark/conftest.py", + "spark/src/test/resources/pyspark/test_pyarrow_udf.py", + "spark/src/test/spark-3.5/org/apache/spark/sql/comet/CometMapInBatchSuite.scala", + "spark/src/test/spark-4.x/org/apache/spark/sql/comet/CometMapInBatchSuite.scala", + "spark/src/test/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerSuite.scala", + ".mvn/**", + "mvnw", + ".github/workflows/ci.yml", + ".github/workflows/pyarrow_udf_test.yml", + ".github/actions/setup-builder/**", + ], "docs": [ ".asf.yaml", ".github/workflows/docs.yaml", @@ -353,6 +409,13 @@ # Benchmark sources are compiled and linted, never run, so a break there # cannot affect a PR's correctness verdict; the queue catches it. "benchmark": ["queue", "label:run-benchmark-check"], + # The Delta build gate only proves a build-system property, and the + # PyArrow suite builds Comet once per Spark 4.x version to drive a real + # Python worker. Neither changes often enough to earn a PR-tier slot; the + # queue catches a regression before it lands, and the label is the escape + # hatch for a change to the surface they cover. + "delta_gate": ["queue", "label:run-delta-build-gate"], + "pyarrow_udf": ["queue", "label:run-pyarrow-udf-tests"], # docs deploys to asf-site, so it must not run from a pull request or from # the queue's throwaway branch. "docs": ["push"], diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md index e6adb1c746d..77d0a21cf41 100644 --- a/docs/source/contributor-guide/ci.md +++ b/docs/source/contributor-guide/ci.md @@ -40,6 +40,8 @@ Which jobs run also depends on the event: | Iceberg Spark SQL tests, Iceberg 1.11 | yes | yes | | macOS build and Comet test suites | with label | yes | | Benchmark compile and lint check | with label | yes | +| Delta contrib build gate | with label | yes | +| PyArrow UDF tests, Spark 4.0 / 4.1 / 4.2 | with label | yes | | Spark SQL tests, Spark 3.4 / 3.5 / 4.0 | with label | yes | | Iceberg Spark SQL tests, Iceberg 1.8 / 1.9 / 1.10 | with label | yes | @@ -67,6 +69,8 @@ Each queue-only suite has a label that runs it on a pull request: | -------------------------- | ---------------------------------------------------- | | `run-macos-tests` | macOS build and Comet test suites | | `run-benchmark-check` | Benchmark compile and lint check | +| `run-delta-build-gate` | Delta contrib build gate | +| `run-pyarrow-udf-tests` | PyArrow UDF tests against Spark 4.0/4.1/4.2 | | `run-spark-4.1-hive-tests` | Spark SQL hive tests against Spark 4.1 | | `run-spark-3.4-tests` | Spark SQL tests against Spark 3.4 | | `run-spark-3.5-tests` | Spark SQL tests against Spark 3.5 | From f7271340eb32df691ca98969d3712c6720fe59df Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 14 Sep 2026 09:35:17 -0600 Subject: [PATCH 2/2] ci: limit the token to contents:read on the Delta gate and PyArrow caller jobs --- .github/workflows/ci.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 213e67351d7..787ece19508 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -253,6 +253,9 @@ jobs: needs: changes # Queue-only by default; PRs need the `run-delta-build-gate` label. if: needs.changes.outputs.delta_gate == 'true' + # The called workflow needs only a checkout. + permissions: + contents: read uses: ./.github/workflows/delta_build_gate.yml pyarrow_udf_test: @@ -260,6 +263,9 @@ jobs: needs: changes # Queue-only by default; PRs need the `run-pyarrow-udf-tests` label. if: needs.changes.outputs.pyarrow_udf == 'true' + # The called workflow needs only a checkout. + permissions: + contents: read uses: ./.github/workflows/pyarrow_udf_test.yml docs: