ci: shrink the pull request tier to the Linux build on the default Spark profile

[andygrove]

Which issue does this PR close?

Closes #5938.

Rationale for this change

Review is now largely agent-driven, and so is responding to review, so a pull request goes through several more push-review-push rounds before it is queued, and there are more contributors with PRs in flight. Every one of those pushes pays for the whole PR tier, and the merge queue runs all of it against the merge result before anything lands anyway. This PR shrinks the PR tier to the Linux build with its test matrix on the default Spark profile, and moves everything else behind the queue.

A sample of the last 100 pull_request runs of ci.yml (about 34 hours, 2026-09-13 to 2026-09-15; 42 of the runs ran the Linux build, the rest were docs-only changes or label runs):

Suite	Runner-min in sample	Runner-min/day	Share of PR total	Sole failing suite
Spark SQL (4.1, core)	5,679	4,050	16%	0 runs
Iceberg 1.11	5,736	4,090	17%	0 runs
Linux build, all jobs	20,197	14,400	58%	1 run
all jobs	34,597	24,650

Inside the Linux build, the linux-test matrix is three quarters of the cost and the five Spark profiles cost about the same each:

Linux build piece	Runner-min/day	Failed in sample
linux-test, Spark 4.1 profile (4 suites)	2,350	0
linux-test, Spark 3.4 / 3.5 / 4.0 / 4.2 profiles	8,700	3 (all [scans], one flaky run)
build-native + rust-test	1,090	1
TPC-H + TPC-DS verification	940	0
Lint Java (4 profiles) + Celeborn + Build Spark 4.1	1,020	0

Together the three moves take about 16,800 runner-minutes a day off pull requests, roughly two thirds of what they spend today, and in this window none of the moved jobs was the one that turned a PR run red. The PR tier's wall clock does not change: it is set by lint, then build-native at 28 minutes, then the [expressions] bucket at 33 minutes, and every profile runs that path in parallel.

The trade-off is that a Spark SQL, Iceberg, or version-specific Comet-suite regression now surfaces in the queue rather than on the PR, where a red run evicts the entry. The Lint Java matrix still compiles Spark 3.4/3.5/4.0 on every pull request, so a shim that fails to build is still caught there; only runtime behaviour waits. Reviewers should apply run-spark-4.1-tests, run-iceberg-tests or run-all-spark-profiles before queuing anything that touches the serde, the planner, a native operator, the version shims, or the Iceberg paths. Continues #5843, #5871 and #5930.

What changes are included in this PR?

Spark SQL and Iceberg suites (first commit):

• dev/ci/compute-changes.py: spark_4_1 and iceberg_1_11 move from ["pr", "queue"] to ["queue", "label:..."]. run-spark-4.1-tests gates both Spark 4.1 outputs so it runs every module in the one 4.1 build; run-spark-4.1-hive-tests still selects only the hive shards. run-iceberg-tests now opts a PR into all four Iceberg versions.
• The run-spark-4.1-tests label did not exist, despite the workflows README saying it "gates nothing". It is created now.

Linux test matrix (second commit):

• dev/ci/linux-test-profiles.py (new): the profile rows of the linux-test matrix, each with a tier. --profiles pr is Spark 4.1 alone, queue-only the other four, all every profile.
• .github/workflows/pr_build_linux.yml: a profiles input. The lint job runs the script and publishes the rows as a job output; linux-test reads them with fromJSON, the same way spark_sql_test_reusable.yml selects its module shards. The lint-java matrix keeps its literal profile list: it is the cross-version compile check and stays on every PR.
• dev/ci/compute-changes.py: a third Linux output, build_linux_all_profiles, queue-only with the new run-all-spark-profiles label (created). ci.yml folds it into the profiles input; a labeled run sets only that output and runs just the profiles the PR tier skipped.

Both:

• dev/ci/check-ci-config.py: PR_TIER shrinks to the Linux build; new synchronize and labeled cases pin the routing for both new labels, including the hive label applied on top of the suite label. A new check_linux_test_profiles asserts the profile tiers partition the list, that the pr tier is exactly the default profile, and that ci.yml passes the profiles input.
• ci.yml, .github/workflows/README.md, dev/ci/spark-sql-modules.py: comments, tier lists, diagram and "what runs when" table.
• docs/source/contributor-guide/ci.md, development.md, spark-sql-tests.md, iceberg-spark-tests.md: tier table, label table, and when to apply a label.

How are these changes tested?

python3 dev/ci/check-ci-config.py passes with the new POLICY_CASES and the profile-partition check, actionlint passes on ci.yml and pr_build_linux.yml, prettier --check passes on the edited Markdown, and apache-rat:check passes on the new script. Both new labels exist on the repository. The routing for each event shape was simulated against compute-changes.py: a plain pull request gets profiles: pr, one carrying the label gets all, the labeled event alone gets queue-only, the merge queue gets all, and push to main stays cache-refresh-only.

This PR's own runs are the first exercise of the new PR tier: the first commit's run skipped every Spark SQL and Iceberg job, and the second commit's run builds the linux-test matrix from the script's output with only the Spark 4.1 rows.

[sunchao]

Correctness

Reviewed 979dff07f0216301d071274d25133c98814ef6de against 9b63e7dacf70ac5428a7aa3ddb6f18f20f685778. I found no P1/P2 issue in the full 11-file diff.

The previous PR tier ran every Linux Spark profile plus the Spark 4.1 core SQL and Iceberg 1.11 suites. This change keeps the default Spark 4.1 Comet suites on ordinary PR updates and moves the other coverage to the merge queue or explicit labels. The existing path filters are unchanged. Comparing the old and new policy shows that every previously selected queue job remains selected. The five profile rows retain exactly the same Spark, Java and Scala arguments, and all four suite buckets are unchanged.

This changes when compatibility tests run. It does not change Spark expression or operator semantics, including nulls, ANSI errors, overflow or fallback. The deprecated Spark 3.4 SQL suite remains label/dispatch-only, as before. No local Spark/JVM compatibility execution was performed for this workflow-only change.

Validation and CI

The local CI configuration checker passed its 23 event-policy cases and eight path-routing cases. I also exercised 17 caller combinations using the expressions from ci.yml: ordinary PRs select four Linux matrix jobs, the queue selects 20, and adding run-all-spark-profiles selects the other 16. A later push with that label selects all 20. Whole-Spark-4.1 and hive-only labels remain distinct, unrelated labels select no heavy jobs, and push events to main retain cache-refresh-only behavior. Invalid profile input exits with an error.

At 07:43:01 UTC on September 15, GitHub showed 18 successful checks, 11 skipped checks and six running checks. Preflight passed the configuration checker and actionlint. The Linux lint log confirms --profiles pr. Rust tests passed 1,447 tests with five skipped. These jobs checked out merge commit 1c9b12b1, whose parents are the reviewed base and head and whose tree equals the head. The native builder has completed successfully. The four default-profile Comet test jobs and two TPC correctness jobs are still running. The skipped SQL/Iceberg jobs are not compatibility passes, and I did not trigger queue or label runs.

Performance

The ordinary PR Linux matrix drops from 20 jobs to four, removing 80% of that matrix's scheduled jobs while retaining the same default-profile suites. The separate Spark 4.1 SQL and Iceberg 1.11 pipelines also stop running on each unlabeled push. This reduces repeated runner work without changing the product's runtime hot paths.

The approximate two-thirds runner-minute saving in the description is the author's historical estimate, not a measured result of this review. It should not be read as a wall-clock speedup: the native build and default-profile critical path remain. Label-only Linux runs still execute the shared build, lint, Rust and TPC work in addition to the selected profiles. The documented cost is later detection of version-specific regressions and possible queue eviction.

Design

The existing queue remains the final gate for applicable coverage. Live branch metadata requires Required Checks, and the active Merge Queue ruleset uses ALLGREEN. The aggregator still waits for all test callers and rejects failures or cancellations. Label runs keep their separate concurrency subgroup and advisory check name, so they cannot replace the ordinary commit run's required verdict. Push and nightly workflows remain separate from this pre-merge coverage.

The PR retains one Linux reusable-workflow call. Its new profile input selects the matrix while the existing cache-refresh input selects push behavior. Native artifact production, run-scoped downloads, cache keys and permission boundaries are unchanged. There is no new secret forwarding or privileged PR trigger.

Abstraction & complexity

The small profile table earns its place because the matrix must be selected before expansion. Reusing the existing lint job for the output adds no scheduling stage, and explicitly listing lint in linux-test.needs makes that output available. Sharing the three Linux path-filter lists prevents routing drift.

The profile validation checks the default row, unique names, row shape and partition of all profiles. The extraction preserves the previous literal matrix and leaves the suite list in one place. I found no actionable simplification or additional blocking finding.

[manuzhang]

I'm not sure we need such historical explanation in the README.