feat: add native Delta Lake scan contrib module (page/row-group pruning)

.github/workflows/README.md

@@ -19,14 +19,15 @@ ruleset in `.asf.yaml`. That splits CI into three tiers:
   profile (4.1) only, and nothing else.
 - **Queue tier** (`queue`): the authoritative gate. The PR tier plus the
   macOS build, the benchmark compile check, the Delta contrib build gate, the
-  PyArrow UDF suite, Spark SQL on Spark 4.1 and Iceberg 1.11, evaluated
-  against the merge result rather than against the PR head. One Spark version
-  and one Iceberg version, both the default profile's.
+  PyArrow UDF suite, Spark SQL on Spark 4.1, Iceberg 1.11 and the Delta
+  contrib suites on Spark 3.5, evaluated against the merge result rather than
+  against the PR head. One Spark version and one Iceberg version, both the
+  default profile's, and one Delta profile.
 - **Nightly tier** (`nightly`): the regression sweep of everything else, once
   a day against `main` as it stands. The Comet test suites against the other
-  four Spark profiles, Spark SQL on Spark 3.5 and 4.0, and Iceberg
-  1.8/1.9/1.10. See [Nightly tier](#nightly-tier) below for how a failure
-  surfaces.
+  four Spark profiles, Spark SQL on Spark 3.5 and 4.0, Iceberg 1.8/1.9/1.10,
+  and the Delta contrib suites on Spark 4.0 and 4.1. See
+  [Nightly tier](#nightly-tier) below for how a failure surfaces.
 
 Every queue-only and nightly job has a `run-*` label that opts a pull request
 into it early, listed in the diagram below. The Lint Java matrix compiles
@@ -103,6 +104,7 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
     (Spark 4.1 profile only)                                    pr_benchmark_check  run-benchmark-check
                                                                 delta_build_gate    run-delta-build-gate
                                                                 pyarrow_udf_test    run-pyarrow-udf-tests
+                                                                delta_3_5           run-delta-tests
                                                                 spark_4_1           run-spark-4.1-tests
   label or dispatch only                                        spark_4_1 sql_hive  run-spark-4.1-hive-tests
   ----------------------                                        iceberg_1_11        run-iceberg-tests
@@ -116,6 +118,8 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
                                       iceberg_1_8         run-iceberg-tests
                                       iceberg_1_9         run-iceberg-tests
                                       iceberg_1_10        run-iceberg-tests
+                                      delta_4_0           run-delta-tests
+                                      delta_4_1           run-delta-tests
 
         |                                   |                                   |
         +-----------------------------------+-----------------------------------+
@@ -152,6 +156,9 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
 | `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check`                                                                                                                                                                                                      | benchmark sources only              |
 | `delta_build_gate`   | merge group, **or** PR with `run-delta-build-gate`                                                                                                                                                                                                     | main sources, poms, `contrib/delta` |
 | `pyarrow_udf_test`   | merge group, **or** PR with `run-pyarrow-udf-tests`                                                                                                                                                                                                    | map-in-batch and Python runner code |
+| `delta_3_5`          | merge group, **or** PR with `run-delta-tests`                                                                                                                                                                                                          | Delta contrib and native sources    |
+| `delta_4_0`          | nightly, **or** PR with `run-delta-tests`                                                                                                                                                                                                              | Delta contrib and native sources    |
+| `delta_4_1`          | nightly, **or** PR with `run-delta-tests`                                                                                                                                                                                                              | Delta contrib and native sources    |
 | `docs`               | push to main, paths matched                                                                                                                                                                                                                            | `.asf.yaml`, `docs/**`, `docs.yaml` |
 | `spark_3_5`          | nightly, **or** PR with `run-spark-3.5-tests`                                                                                                                                                                                                          | Spark 3.5 sources                   |
 | `spark_4_1`          | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests`                                                                                                                                         | Spark 4.1 sources                   |
@@ -267,6 +274,7 @@ umbrella doesn't watch, or operate independently of the rest of CI:
 | `pr_build_macos.yml`              | `pr_build_macos`                                             |
 | `pr_benchmark_check.yml`          | `pr_benchmark_check`                                         |
 | `delta_build_gate.yml`            | `delta_build_gate`                                           |
+| `delta_contrib_test.yml`          | `delta_3_5`, `delta_4_0`, `delta_4_1`                        |
 | `pyarrow_udf_test.yml`            | `pyarrow_udf_test`                                           |
 | `docs.yaml`                       | `docs`                                                       |
 | `spark_sql_test_reusable.yml`     | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1`           |

.github/workflows/ci.yml

@@ -208,6 +208,9 @@ jobs:
       iceberg_1_9:  ${{ steps.compute.outputs.iceberg_1_9 }}
       iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }}
       iceberg_1_11: ${{ steps.compute.outputs.iceberg_1_11 }}
+      delta_3_5:    ${{ steps.compute.outputs.delta_3_5 }}
+      delta_4_0:    ${{ steps.compute.outputs.delta_4_0 }}
+      delta_4_1:    ${{ steps.compute.outputs.delta_4_1 }}
     steps:
       - uses: actions/checkout@v7
         with:
@@ -396,6 +399,46 @@ jobs:
       spark-full: '3.5.9'
       java: 17
 
+  delta_3_5:
+    name: Delta Contrib Tests (Spark 3.5)
+    needs: changes
+    permissions:
+      contents: read
+    # Queue-only by default; PRs need the `run-delta-tests` label. Spark 3.5
+    # is the profile the contrib's MinIO and feature-off jobs already pin,
+    # which is why it is the one Delta profile the queue runs while 4.0 and
+    # 4.1 run nightly.
+    if: needs.changes.outputs.delta_3_5 == 'true'
+    uses: ./.github/workflows/delta_contrib_test.yml
+    with:
+      spark: '3.5'
+      java: '17'
+
+  delta_4_0:
+    name: Delta Contrib Tests (Spark 4.0)
+    needs: changes
+    permissions:
+      contents: read
+    # Nightly by default; PRs need the `run-delta-tests` label.
+    if: needs.changes.outputs.delta_4_0 == 'true'
+    uses: ./.github/workflows/delta_contrib_test.yml
+    with:
+      spark: '4.0'
+      java: '17'
+
+  delta_4_1:
+    name: Delta Contrib Tests (Spark 4.1)
+    needs: changes
+    permissions:
+      contents: read
+    # Nightly by default; PRs need the `run-delta-tests` label. No Spark 4.2
+    # caller: the contrib profile is dormant until a Delta release supports it.
+    if: needs.changes.outputs.delta_4_1 == 'true'
+    uses: ./.github/workflows/delta_contrib_test.yml
+    with:
+      spark: '4.1'
+      java: '17'
+
   spark_4_0:
     name: Spark SQL Tests (Spark 4.0)
     needs: changes
@@ -528,6 +571,9 @@ jobs:
       - pyarrow_udf_test
       - spark_3_4
       - spark_3_5
+      - delta_3_5
+      - delta_4_0
+      - delta_4_1
       - spark_4_0
       - spark_4_1
       - iceberg_1_8
@@ -595,7 +641,7 @@ jobs:
               failed.length ? 'Jobs that did not succeed:' : 'No individual job reported a failure; see the run for details.',
               ...failed,
               '',
-              'The nightly tier runs the Spark and Iceberg versions the merge queue does not',
+              'The nightly tier runs the Spark, Iceberg and Delta versions the merge queue does not',
               '(see `POLICY` in `dev/ci/compute-changes.py`). Close this issue once the failure',
               'is understood; the next red nightly opens a new one.',
             ].join('\n');

.github/workflows/delta_build_gate.yml

@@ -15,12 +15,15 @@
 # specific language governing permissions and limitations
 # under the License.
 
-# Build-gate verification for the optional contrib/delta integration.
+# Build-gate verification for the Delta split of the native library and the JVM build.
 #
 # A single cheap job that runs dev/verify-contrib-delta-gate.sh, which proves:
-#   - the DEFAULT cargo / mvn / dylib build carries ZERO Delta surface
-#     (no comet-contrib-delta / delta_kernel in the cargo tree, no io.delta in
-#     the effective pom, no Delta symbols in libcomet); and
+#   - the kernel-backed `comet-contrib-delta` crate stays out of every shipped
+#     build (no comet-contrib-delta / delta_kernel in the default or
+#     --no-default-features cargo tree, no io.delta in the effective pom, no
+#     contrib symbols in libcomet), while the small default-on `delta` feature
+#     (deletion-vector decoding) is deliberately in and its symbol footprint is
+#     pinned; and
 #   - the gated build (`--features contrib-delta` / `-Pcontrib-delta`) compiles
 #     and pulls the right delta-spark per Spark profile.
 #