Skip to content

ci: shrink the pull request tier to the Linux build on the default Spark profile #5939

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andygrove merged 3 commits into from
Sep 15, 2026
Merged

ci: shrink the pull request tier to the Linux build on the default Spark profile #5939

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3173cd2
ci: move the Spark 4.1 and Iceberg 1.11 suites behind the merge queue
andygrove Sep 15, 2026
f17ddbe
ci: run the Linux test matrix against the default Spark profile only …
andygrove Sep 15, 2026
dfe52eb
ci: drop the history of the PR tier from the workflows README
andygrove Sep 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 65 additions & 41 deletions .github/workflows/README.md
View file
Open in desktop

Large diffs are not rendered by default.

65 changes: 42 additions & 23 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,14 @@
# Merging goes through GitHub's merge queue (see `rulesets` in `.asf.yaml`), so
# there are two tiers:
#
# pull_request fast feedback. Linux build + Spark 4.1 (catalyst and
# sql_core only) + Iceberg 1.11.
# merge_group the authoritative gate. Everything the PR tier runs, plus
# the macOS build, the benchmark compile check, the Delta
# contrib build gate, the PyArrow UDF suite, the Spark 4.1
# sql_hive shards, Spark 3.5/4.0 and Iceberg 1.8/1.9/1.10,
# evaluated against the merge result rather than the PR head.
# pull_request fast feedback. The Linux build only, with its test matrix
# run against the default Spark profile (4.1) alone.
# merge_group the authoritative gate. The Linux build with the test
# matrix against every Spark profile, plus the macOS build,
# the benchmark compile check, the Delta contrib build gate,
# the PyArrow UDF suite, Spark 3.5/4.0/4.1 and Iceberg
# 1.8/1.9/1.10/1.11, evaluated against the merge result
# rather than the PR head.
#
# Spark 3.4 is deprecated and sits outside both tiers: it runs only when a
# pull request carries `run-spark-3.4-tests`, or from a manual dispatch.
Expand Down Expand Up @@ -161,6 +162,7 @@ jobs:
outputs:
build_linux: ${{ steps.compute.outputs.build_linux }}
build_linux_full: ${{ steps.compute.outputs.build_linux_full }}
build_linux_all_profiles: ${{ steps.compute.outputs.build_linux_all_profiles }}
build_macos: ${{ steps.compute.outputs.build_macos }}
benchmark: ${{ steps.compute.outputs.benchmark }}
delta_gate: ${{ steps.compute.outputs.delta_gate }}
Expand Down Expand Up @@ -237,16 +239,31 @@ jobs:
pr_build_linux:
name: PR Build (Linux)
needs: changes
if: needs.changes.outputs.build_linux == 'true'
uses: ./.github/workflows/pr_build_linux.yml
# Two POLICY outputs feed one call, the same shape as spark_4_1 below.
# Three POLICY outputs feed one call, the same shape as spark_4_1 below.
# `build_linux` decides whether the workflow runs at all; `build_linux_full`
# decides whether it runs the lints and the 5x4 test matrix as well as the
# jobs that populate main's actions/cache entries. Only push to main sets
# the first without the second, which is the whole point: the queue has
# already tested that tree, so the push run is there for the caches alone.
# whether it runs the lints and the test matrix as well as the jobs that
# populate main's actions/cache entries; `build_linux_all_profiles` whether
# the test matrix covers every Spark profile or only the default one. The
# combinations that occur:
#
# pull request linux, full -> profiles: pr
# ... with the label linux, full, all -> profiles: all
# `labeled` run all -> profiles: queue-only
# merge queue linux, full, all -> profiles: all
# push to main linux -> cache-refresh-only
#
# Only push to main sets `build_linux` without `build_linux_full`, which is
# the whole point: the queue has already tested that tree, so the push run
# is there for the caches alone. A `labeled` run sets only the third, and
# then runs just the profiles the PR tier skipped.
if: needs.changes.outputs.build_linux == 'true' || needs.changes.outputs.build_linux_all_profiles == 'true'
uses: ./.github/workflows/pr_build_linux.yml
with:
cache-refresh-only: ${{ needs.changes.outputs.build_linux_full != 'true' }}
cache-refresh-only: ${{ needs.changes.outputs.build_linux_full != 'true' && needs.changes.outputs.build_linux_all_profiles != 'true' }}
profiles: >-
${{ needs.changes.outputs.build_linux_all_profiles != 'true' && 'pr'
|| needs.changes.outputs.build_linux_full != 'true' && 'queue-only'
|| 'all' }}

pr_build_macos:
name: PR Build (macOS)
Expand Down Expand Up @@ -339,11 +356,12 @@ jobs:
spark_4_1:
name: Spark SQL Tests (Spark 4.1)
needs: changes
# Two POLICY outputs feed one call, so the queue gets every module from a
# single 40-minute build instead of two. `spark_4_1` (PR tier) covers
# catalyst and the sql_core shards; `spark_4_1_hive` (queue-only, or the
# `run-spark-4.1-hive-tests` label) adds the sql_hive shards. A `labeled`
# run sets only the second, and then runs only the hive rows.
# Queue-only by default, like every other Spark SQL suite. Two POLICY
# outputs feed one call, so the queue gets every module from a single
# 40-minute build instead of two. `spark_4_1` covers catalyst and the
# sql_core shards; `spark_4_1_hive` adds the sql_hive shards. On a pull
# request `run-spark-4.1-tests` sets both, and `run-spark-4.1-hive-tests`
# sets only the second, which then runs only the hive rows.
if: needs.changes.outputs.spark_4_1 == 'true' || needs.changes.outputs.spark_4_1_hive == 'true'
uses: ./.github/workflows/spark_sql_test_reusable.yml
with:
Expand Down Expand Up @@ -384,8 +402,7 @@ jobs:
iceberg_1_10:
name: Iceberg Spark SQL Tests (Iceberg 1.10)
needs: changes
# Queue-only by default; PRs need the `run-iceberg-tests` label. Iceberg 1.11
# (Spark 4.1) is the PR-gated Iceberg job; 1.10 covers the Spark 3.5 path.
# Queue-only by default; PRs need the `run-iceberg-tests` label.
if: needs.changes.outputs.iceberg_1_10 == 'true'
uses: ./.github/workflows/iceberg_spark_test_reusable.yml
with:
Expand All @@ -398,7 +415,9 @@ jobs:
iceberg_1_11:
name: Iceberg Spark SQL Tests (Iceberg 1.11)
needs: changes
# Runs on every PR: Iceberg 1.11 is our only Spark 4.1 Iceberg coverage.
# Queue-only by default; PRs need the `run-iceberg-tests` label. Iceberg
# 1.11 is our only Spark 4.1 Iceberg coverage, and it was the PR-tier
# Iceberg job for that reason until every suite moved behind the queue.
if: needs.changes.outputs.iceberg_1_11 == 'true'
uses: ./.github/workflows/iceberg_spark_test_reusable.yml
with:
Expand Down
60 changes: 36 additions & 24 deletions .github/workflows/pr_build_linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
#
# Two modes, selected by `cache-refresh-only`:
#
# false the full pipeline: the lints, the 5x4 linux-test matrix, the Rust
# false the full pipeline: the lints, the linux-test matrix, the Rust
# tests and the TPC-H/TPC-DS correctness runs. Pull requests and the
# merge queue.
# true only the jobs that write an actions/cache entry, and each only far
Expand All @@ -32,6 +32,14 @@
# next pull request. See the `build_linux` / `build_linux_full`
# comments in dev/ci/compute-changes.py for why that tier exists.
#
# Within the full pipeline, `profiles` picks which Spark profiles the
# linux-test matrix runs the Comet test suites against: `pr` is the default
# build profile alone, `all` is every profile, and `queue-only` is the rest.
# The rows live in dev/ci/linux-test-profiles.py, which the `lint` job runs so
# that the matrix can be built from its output. The queue runs `all`; a pull
# request runs `pr` unless it carries `run-all-spark-profiles`. See the
# `build_linux_all_profiles` comment in dev/ci/compute-changes.py.
#
# Keeping both modes in this file is deliberate. The cache keys are the whole
# point of the push tier, and a separate cache-warming workflow would have to
# repeat every one of them; here they cannot drift apart. dev/ci/check-ci-config.py
Expand All @@ -46,6 +54,15 @@
test. Set by ci.yml on push to main.
type: boolean
default: false
profiles:
# Which Spark profiles the linux-test matrix runs. `pr` is the default
# build profile alone, `all` is every profile, `queue-only` is every
# profile except the `pr` one (a `labeled` run, where the PR-tier row
# already ran at this commit). Rows and tiers are in
# dev/ci/linux-test-profiles.py.
description: 'Which Spark profiles linux-test runs: all, pr (the default build profile) or queue-only (the rest)'
type: string
default: 'all'

env:
RUST_VERSION: stable
Expand All @@ -65,9 +82,18 @@
runs-on: ubuntu-24.04
container:
image: amd64/rust
outputs:
# The `profile` dimension of the linux-test matrix, selected by the
# `profiles` input. Computed here because this job already gates
# linux-test and a job-level `if:` cannot see `matrix`.
profile-matrix: ${{ steps.profiles.outputs.matrix }}
steps:
- uses: actions/checkout@v7

- name: Select linux-test profiles
id: profiles
run: python3 dev/ci/linux-test-profiles.py --profiles "${{ inputs.profiles }}" --github-output "$GITHUB_OUTPUT"

- name: Check Rust formatting
run: |
rustup component add rustfmt
Expand Down Expand Up @@ -379,224 +405,210 @@
key: ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}

linux-test:
needs: build-native
# `lint` is already upstream via build-native; it is listed here so this
# job can read its `profile-matrix` output.
needs: [lint, build-native]
if: ${{ !inputs.cache-refresh-only }}
strategy:
matrix:
# the goal with these profiles is to get coverage of all Java, Scala, and Spark
# versions without testing all possible combinations, which would be overkill
profile:
- name: "Spark 3.4, JDK 17, Scala 2.12"
java_version: "17"
maven_opts: "-Pspark-3.4 -Pscala-2.12"

- name: "Spark 3.5, JDK 17, Scala 2.13"
java_version: "17"
maven_opts: "-Pspark-3.5 -Pscala-2.13"

- name: "Spark 4.0, JDK 21"
java_version: "21"
maven_opts: "-Pspark-4.0"

- name: "Spark 4.1, JDK 17"
java_version: "17"
maven_opts: "-Pspark-4.1"

- name: "Spark 4.2, JDK 17"
java_version: "17"
maven_opts: "-Pspark-4.2"
# The Spark profiles live in dev/ci/linux-test-profiles.py and are
# selected by the `profiles` input: the default build profile on a
# pull request, all five in the merge queue. The goal of the full list
# is coverage of every Java, Scala and Spark version without testing
# every combination.
profile: ${{ fromJSON(needs.lint.outputs.profile-matrix) }}
# Suites are grouped by functional area into balanced buckets so that no test
# job runs much longer than ~23 min. See
# docs/superpowers/specs/2026-05-22-pr-build-consolidation-design.md for the
# per-suite timing analysis behind this grouping.
suite:
- name: "scans"
value: |
org.apache.comet.parquet.CometParquetWriterSuite
org.apache.comet.parquet.ParquetReadV1Suite
org.apache.comet.parquet.ParquetReadV2Suite
org.apache.comet.parquet.ParquetTimestampLtzAsNtzSuite
org.apache.spark.sql.comet.ParquetDatetimeRebaseV1Suite
org.apache.spark.sql.comet.ParquetDatetimeRebaseV2Suite
org.apache.spark.sql.comet.ParquetEncryptionITCase
org.apache.comet.exec.CometNativeReaderSuite
org.apache.comet.CometIcebergNativeSuite
org.apache.comet.CometIcebergEncryptionSuite
org.apache.comet.CometIcebergRewriteActionSuite
org.apache.comet.CometIcebergWriteActionSuite
org.apache.comet.CometIcebergWriteDetectionSuite
org.apache.comet.CometIcebergSystemFunctionSuite
org.apache.comet.iceberg.IcebergReflectionSuite
org.apache.comet.serde.operator.IcebergWriteProtoTranslationSuite
org.apache.comet.csv.CometCsvNativeReadSuite
org.apache.comet.CometFuzzTestSuite
org.apache.comet.CometFuzzIcebergSuite
org.apache.comet.DataGeneratorSuite
- name: "shuffle"
value: |
org.apache.comet.exec.CometShuffleSuite
org.apache.comet.exec.CometShuffle4_0Suite
org.apache.comet.exec.CometNativeColumnarToRowSuite
org.apache.comet.exec.CometNativeShuffleSuite
org.apache.comet.shuffle.CelebornShufflePartitionPusherSuite
org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManagerSuite
org.apache.spark.sql.comet.execution.shuffle.CometCelebornNativeShuffleWriterSuite
org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleFallbackSuite
org.apache.spark.sql.comet.execution.shuffle.CometCelebornConcurrentMaterializationSuite
org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleStatisticsSuite
org.apache.spark.sql.comet.execution.shuffle.CometCelebornLocalFetchFailureSuite
org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleReaderSuite
org.apache.spark.sql.comet.execution.shuffle.CometCelebornShufflePlanningSuite
org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleInputRDDSuite
org.apache.spark.sql.comet.execution.shuffle.CometDiskBlockWriterSuite
org.apache.comet.exec.CometShuffleEncryptionSuite
org.apache.comet.exec.CometShuffleManagerSuite
org.apache.comet.exec.CometAsyncShuffleSuite
org.apache.comet.exec.DisableAQECometShuffleSuite
org.apache.comet.exec.DisableAQECometAsyncShuffleSuite
org.apache.spark.shuffle.comet.CometBoundedShuffleMemoryAllocatorSuite
org.apache.spark.shuffle.sort.SpillSorterSuite
- name: "exec"
value: |
org.apache.comet.exec.CometAggregateSuite
org.apache.comet.exec.CometExec3_4PlusSuite
org.apache.comet.exec.CometExecSuite
org.apache.comet.exec.CometInMemoryCacheSuite
org.apache.comet.exec.CometInMemoryCacheKryoSuite
org.apache.comet.exec.CometGenerateExecSuite
org.apache.comet.exec.CometWindowExecSuite
org.apache.comet.exec.CometJoinSuite
org.apache.spark.sql.comet.CometMapInBatchSuite
org.apache.spark.sql.execution.python.CometArrowPythonRunnerSuite
org.apache.comet.CometNativeSuite
org.apache.comet.CometConfSuite
org.apache.comet.CometPublicApiSuite
org.apache.comet.QueryContextInternerSuite
org.apache.comet.CometSetOpWithGroupBySuite
org.apache.comet.CometSparkSessionExtensionsSuite
org.apache.spark.CometPluginsSuite
org.apache.spark.CometRuntimeShutdownSuite
org.apache.spark.CometTaskMemoryManagerSuite
org.apache.spark.CometExecIteratorLifecycleSuite
org.apache.spark.CometPluginsDefaultSuite
org.apache.spark.CometPluginsNonOverrideSuite
org.apache.spark.CometPluginsUnifiedModeOverrideSuite
org.apache.comet.rules.CometScanRuleSuite
org.apache.comet.rules.CometScanContribSuite
org.apache.comet.rules.CometScanSchemeFallbackSuite
org.apache.comet.rules.CometExecRuleSuite
org.apache.comet.rules.RevertNativeForTransitionHeavyStagesSuite
org.apache.spark.sql.CometTPCDSQuerySuite
org.apache.spark.sql.CometTPCDSQueryTestSuite
org.apache.spark.sql.CometTPCHQuerySuite
org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite
org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite
org.apache.spark.sql.comet.CometTaskMetricsSuite
org.apache.spark.sql.comet.CometDppFallbackRepro3949Suite
org.apache.spark.sql.comet.CometShuffleFallbackStickinessSuite
org.apache.spark.sql.comet.PlanDataInjectorSuite
org.apache.spark.sql.comet.CometDecimalArithmeticViewSuite
org.apache.spark.sql.comet.CometDecimalPromotionSuite
org.apache.spark.sql.comet.CometScanWithPlanDataSuite
org.apache.spark.sql.comet.util.UtilsSuite
org.apache.comet.vector.NativeUtilSuite
org.apache.comet.vector.CometVectorUtilsSuite
org.apache.comet.objectstore.NativeConfigSuite
org.apache.comet.serde.operator.CometIcebergNativeScanSuite
org.apache.comet.serde.operator.CometNativeScanSuite
org.apache.spark.sql.CometToPrettyStringSuite
org.apache.spark.sql.CometCollationSuite
org.apache.comet.CometFuzzAggregateSuite
org.apache.spark.sql.comet.execution.arrow.CometArrowStreamSuite
org.apache.spark.sql.CometSparkInternalFunctionsSuite
- name: "expressions"
value: |
org.apache.comet.CometExpressionSuite
org.apache.comet.CometSqlFileTestSuite
org.apache.comet.SqlFileTestParserSuite
org.apache.comet.CometExpressionCoverageSuite
org.apache.comet.CometVariantTypeSuite
org.apache.comet.CometHashExpressionSuite
org.apache.comet.CometTemporalExpressionSuite
org.apache.comet.CometArrayExpressionSuite
org.apache.comet.CometNativeCastSuite
org.apache.comet.CometDateTimeUtilsSuite
org.apache.comet.CometMathExpressionSuite
org.apache.comet.CometStringExpressionSuite
org.apache.comet.CometBitwiseExpressionSuite
org.apache.comet.CometMapExpressionSuite
org.apache.comet.CometCsvExpressionSuite
org.apache.comet.CometJsonExpressionSuite
org.apache.comet.CometJsonJvmSuite
org.apache.comet.SparkErrorConverterSuite
org.apache.comet.expressions.conditional.CometIfSuite
org.apache.comet.expressions.conditional.CometCoalesceSuite
org.apache.comet.expressions.conditional.CometCaseWhenSuite
org.apache.comet.CometRegExpJvmSuite
org.apache.comet.CometCodegenSuite
org.apache.comet.CometCoverageStatsSuite
org.apache.comet.CometCodegenSourceSuite
org.apache.comet.CometCodegenHOFSuite
org.apache.comet.CometFuzzMathSuite
org.apache.comet.CometCodegenFuzzSuite
org.apache.comet.CometScalaUDFClassLoaderSuite
org.apache.comet.codegen.CometSpecializedGettersDispatchSuite
org.apache.comet.CometStringDecodeSuite
org.apache.comet.CometWidthBucketSuite
org.apache.comet.CometUuidExpressionSuite
org.apache.comet.serde.CometScalarFunctionSuite
org.apache.comet.serde.CometLiteralSuite
org.apache.comet.CometFallbackInvarianceSuite
fail-fast: false
name: ${{ matrix.profile.name }} [${{ matrix.suite.name }}]
runs-on: ubuntu-24.04
container:
image: amd64/rust
env:
JAVA_TOOL_OPTIONS: '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED'

steps:
- uses: actions/checkout@v7

- name: Setup Rust & Java toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: ${{ env.RUST_VERSION }}
jdk-version: ${{ matrix.profile.java_version }}

- name: Download native library
uses: ./.github/actions/download-artifact-retry
with:
name: native-lib-linux
# Download to release/ since Maven's -Prelease expects libcomet.so there
path: native/target/release/

# Restore cargo registry cache (for any cargo commands that might run)
- name: Cache Cargo registry
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
key: ${{ runner.os }}-cargo-registry-${{ hashFiles('native/**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-registry-

- name: Java test steps
uses: ./.github/actions/java-test
with:
artifact_name: ${{ matrix.profile.name }}-${{ matrix.suite.name }}-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}
suites: ${{ matrix.suite.value }}
maven_opts: ${{ matrix.profile.maven_opts }}
upload-test-reports: true
skip-native-build: true

# TPC-H correctness test - verifies benchmark queries produce correct results.
# Runs in cache-refresh-only mode for its two cache entries: the SF=1 dataset,
# which is expensive to regenerate, and main's shared `java-maven` entry. Only
# the query run is skipped there; everything up to and including data
# generation still has to happen for the dataset cache to exist.
verify-benchmark-results-tpch:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}
needs: build-native
name: Verify TPC-H Results
runs-on: ubuntu-24.04
Expand Down
Loading
Loading