What is the problem the feature request solves?
Anyone who wants to try a fix or feature before the next Comet release has to build from source, including the Rust native library, which needs a Rust toolchain, protoc and a JDK. Downstream projects that want to test against Comet's main branch have the same problem.
Other ASF projects such as Spark and Iceberg publish nightly SNAPSHOT artifacts to https://repository.apache.org/content/repositories/snapshots/ from a scheduled GitHub Actions workflow, using the Nexus credentials that ASF Infra provisions as repository secrets.
The repository previously had a tag-triggered workflow that published Docker images to GHCR. It was removed in #4241 after failing for a long time: first within a minute of starting, later as startup_failure because it used third-party actions that were not pinned to a commit SHA, and it also built arm64 under QEMU with caching disabled.
Describe the potential solution
Add a scheduled workflow that:
- builds
libcomet.so for linux/amd64 and linux/aarch64 on native runners, inside an Ubuntu 20.04 container so the glibc baseline matches the release builder
- assembles the shaded
comet-spark jars for Spark 3.4 (Scala 2.12), 3.5 (Scala 2.12), 4.0 and 4.1 (Scala 2.13) with both native libraries, the same way dev/release/build-release-comet.sh does
- deploys them with
./mvnw deploy using the NEXUS_USER / NEXUS_PW repository secrets; the org.apache:apache parent pom already routes SNAPSHOT deploys to apache.snapshots.https
- skips nights where
main has not changed
- supports a
dry_run dispatch that builds and verifies the jars without touching Nexus, so the workflow can be exercised on a fork
The installation guide should document where the snapshots live, which artifacts exist, and that they are unreleased builds for testing only.
macOS libraries are out of scope: the x86_64 build needs the Apple SDK cross-compile path, which CI cannot use.
Additional context
If the NEXUS_USER / NEXUS_PW secrets are not yet configured on this repository, an INFRA Jira ticket is needed before the first real publish succeeds.
What is the problem the feature request solves?
Anyone who wants to try a fix or feature before the next Comet release has to build from source, including the Rust native library, which needs a Rust toolchain, protoc and a JDK. Downstream projects that want to test against Comet's main branch have the same problem.
Other ASF projects such as Spark and Iceberg publish nightly SNAPSHOT artifacts to https://repository.apache.org/content/repositories/snapshots/ from a scheduled GitHub Actions workflow, using the Nexus credentials that ASF Infra provisions as repository secrets.
The repository previously had a tag-triggered workflow that published Docker images to GHCR. It was removed in #4241 after failing for a long time: first within a minute of starting, later as
startup_failurebecause it used third-party actions that were not pinned to a commit SHA, and it also built arm64 under QEMU with caching disabled.Describe the potential solution
Add a scheduled workflow that:
libcomet.sofor linux/amd64 and linux/aarch64 on native runners, inside an Ubuntu 20.04 container so the glibc baseline matches the release buildercomet-sparkjars for Spark 3.4 (Scala 2.12), 3.5 (Scala 2.12), 4.0 and 4.1 (Scala 2.13) with both native libraries, the same waydev/release/build-release-comet.shdoes./mvnw deployusing theNEXUS_USER/NEXUS_PWrepository secrets; theorg.apache:apacheparent pom already routes SNAPSHOT deploys toapache.snapshots.httpsmainhas not changeddry_rundispatch that builds and verifies the jars without touching Nexus, so the workflow can be exercised on a forkThe installation guide should document where the snapshots live, which artifacts exist, and that they are unreleased builds for testing only.
macOS libraries are out of scope: the x86_64 build needs the Apple SDK cross-compile path, which CI cannot use.
Additional context
If the
NEXUS_USER/NEXUS_PWsecrets are not yet configured on this repository, an INFRA Jira ticket is needed before the first real publish succeeds.