Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@ name: Nightly

run-name: 'Nightly ${{ github.ref_name }} · @${{ github.actor }}'

# Builds a nightly wheel of `coreai-opt` from `main` and publishes it to PyPI
# via Trusted Publishing (OIDC). The schedule, the skip rule, manual dry runs,
# and the one-time PyPI / `pypi-nightly` environment setup are described in
# RELEASE.md ("Nightly builds").
# Builds a nightly wheel from `main` and publishes it to PyPI as
# `coreai-opt-nightly` via Trusted Publishing (OIDC). The schedule, the skip
# rule, manual dry runs, and the one-time PyPI / `pypi-nightly` environment
# setup are described in RELEASE.md ("Nightly builds").
on:
schedule:
# 08:23 UTC = 00:23 PST / 01:23 PDT. Off the top of the hour, when GitHub
Expand Down Expand Up @@ -91,7 +91,7 @@ jobs:
timeout-minutes: 15
environment:
name: pypi-nightly
url: https://pypi.org/p/coreai-opt
url: https://pypi.org/p/coreai-opt-nightly
permissions:
id-token: write # mint the OIDC token PyPI validates for Trusted Publishing
contents: read
Expand Down
8 changes: 4 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -255,10 +255,10 @@ build:
build-dev:
@$(call use_env,VENV) && uv run --no-sync --active python $(SCRIPTS)/make/build.py --dev

# Build the nightly distribution published to PyPI with build.py: the release
# base with a unique, timestamped PEP 440 dev suffix (e.g. 0.2.2.dev202607231430),
# via `uv build --no-sources` like `build`. This is what the nightly workflow
# (.github/workflows/nightly.yml) runs.
# Build the nightly distribution published to PyPI as coreai-opt-nightly with
# build.py: the release base with a unique, timestamped PEP 440 dev suffix (e.g.
# 0.2.2.dev202607231430), via `uv build --no-sources` like `build`. This is what
# the nightly workflow (.github/workflows/nightly.yml) runs.
build-nightly:
@$(call use_env,VENV) && uv run --no-sync --active python $(SCRIPTS)/make/build.py --nightly

Expand Down
28 changes: 15 additions & 13 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ The following commands are available locally:
```bash
make build # build the artifacts for the current release to be published
make build-dev # build a local development wheel (e.g. 1.1.0.dev202607231430+abc1234)
make build-nightly # build the nightly wheel published to PyPI (e.g. 1.1.0.dev202607231430)
make build-nightly # build the nightly wheel published to PyPI as coreai-opt-nightly (e.g. 1.1.0.dev202607231430)
make version # show the version a release would publish (e.g. 1.1.0)
make version-dev # show the development version carried on the tree (e.g. 1.1.0.dev0)
make clean # remove build artifacts
Expand All @@ -19,12 +19,12 @@ make clean # remove build artifacts

There are four version formats:

| Format | Example | What it is |
| -------------------------------------- | ------------------------------- | -------------------------------------------------- |
| `X.Y.Z` | `1.0.0` | a published release |
| `X.Y.Z.dev0` | `1.1.0.dev0` | the version `main` carries in the repo |
| `X.Y.Z.dev<UTC-timestamp>` | `1.1.0.dev202607231430` | a nightly, published to PyPI |
| `X.Y.Z.dev<UTC-timestamp>+<short-sha>` | `1.1.0.dev202607231430+abc1234` | a local dev artifact, built from a specific commit |
| Format | Example | What it is |
| -------------------------------------- | ------------------------------- | ---------------------------------------------------- |
| `X.Y.Z` | `1.0.0` | a published release |
| `X.Y.Z.dev0` | `1.1.0.dev0` | the version `main` carries in the repo |
| `X.Y.Z.dev<UTC-timestamp>` | `1.1.0.dev202607231430` | a nightly, published to PyPI as `coreai-opt-nightly` |
| `X.Y.Z.dev<UTC-timestamp>+<short-sha>` | `1.1.0.dev202607231430+abc1234` | a local dev artifact, built from a specific commit |

Say `1.0.0` has just been released. `main` then carries `1.0.1.dev0`. That reads as "working toward a release after `1.0.0`, which has not shipped": the `.dev0` suffix marks the tree as unreleased and never appears in a built wheel. Nothing on `main` can be mistaken for a published version.

Expand Down Expand Up @@ -55,7 +55,7 @@ flowchart TB
`src/coreai_opt/_about.py` holds the last released version and `__version__`. Both are set together in one PR when `main` moves forward: the last released version becomes the release just branched, and `__version__` becomes the one after it. The `check-about-version` pre-commit hook enforces the relation between them: `__version__` must be the last released version with exactly one of its numbers raised by one, every number after that reset to zero, and `.dev0` on the end. From `1.0.0` it accepts `1.0.1.dev0`, `1.1.0.dev0`, or `2.0.0.dev0`, and nothing else. Any other value fails the commit, and the message prints the accepted ones, so there is nothing to work out by hand.

- `make build` builds the artifacts for the current release to be published.
- `make build-nightly` builds the nightly wheel published to PyPI, carrying a unique `.dev<UTC-timestamp>` suffix.
- `make build-nightly` builds the nightly wheel published to PyPI as `coreai-opt-nightly`, carrying a unique `.dev<UTC-timestamp>` suffix.
- `make build-dev` builds local wheels for development and testing, each carrying a unique `.dev<UTC-timestamp>+<short-sha>` suffix.

Therefore, we have the following order:
Expand All @@ -82,24 +82,26 @@ Cut the branch before moving `main` to the next dev release.

### Nightly builds

`.github/workflows/nightly.yml` publishes a nightly wheel of `coreai-opt` to PyPI from `main` every day at 08:23 UTC (00:23 PST, 01:23 PDT). It runs `make build-nightly` and publishes the wheel via Trusted Publishing. A nightly is a snapshot of `main`, published every night; the smoke tests run on releases (`release.yml`). A scheduled run builds only when `main` has moved since the last successful nightly. The commit a nightly was built from is the head SHA of its workflow run.
`.github/workflows/nightly.yml` publishes a nightly wheel to PyPI as its own project, [`coreai-opt-nightly`](https://pypi.org/project/coreai-opt-nightly/), from `main` every day at 08:23 UTC (00:23 PST, 01:23 PDT). The `coreai-opt` release history therefore lists only releases. It runs `make build-nightly` and publishes the wheel via Trusted Publishing. A nightly is a snapshot of `main`, published every night; the smoke tests run on releases (`release.yml`). A scheduled run builds only when `main` has moved since the last successful nightly. The commit a nightly was built from is the head SHA of its workflow run.

A manual run (Actions → Nightly → Run workflow, or `gh workflow run nightly.yml --ref <branch>`) always builds, and publishes only when started on `main`. On any other branch it is a dry run.

Publishing relies on two settings outside the repo, each set up once:

1. On GitHub, under Settings → Environments (repository admin access required), create the `pypi-nightly` environment. Leave Required reviewers and Wait timer unchecked so the nightly publishes unattended, and under Deployment branches and tags choose Selected branches and tags with a single Branch rule for `main`.
2. On PyPI, under `coreai-opt` → Manage → Publishing, add a GitHub Actions Trusted Publisher with owner `apple`, repository `coreai-optimization`, workflow `nightly.yml` (the filename, not the workflow's display name), and environment `pypi-nightly`.
2. On PyPI, under Your account → Publishing, add a GitHub Actions pending publisher with PyPI project name `coreai-opt-nightly`, owner `apple`, repository `coreai-optimization`, workflow `nightly.yml` (the filename, not the workflow's display name), and environment `pypi-nightly`. The first nightly upload creates the project; it then lists this publisher under `coreai-opt-nightly` → Manage → Publishing.

To verify the setup, start a manual run with `main` selected as the branch. The environment checks the branch the run was started on, so a run started from any other branch can't publish.

Nightlies are pre-releases, so installers pick them only when asked:
`coreai-opt-nightly` carries only dev versions, so installers pick them without a pre-release flag:

```bash
pip install --upgrade --pre coreai-opt # the newest nightly, or a newer release
pip install coreai-opt==1.1.0.dev202607231430 # one specific nightly
uv pip install --upgrade coreai-opt-nightly # the newest nightly
uv pip install coreai-opt-nightly==1.1.0.dev202607231430 # one specific nightly
```

`coreai-opt` and `coreai-opt-nightly` install the same `coreai_opt` package, so an environment uses one or the other.

### Extending the scheme downstream

A repo that uses this one as a submodule and includes this `Makefile` — building one combined wheel from both trees — can add its own 4th number. Set `COREAI_OPT_VERSION_EXTENSION` to the number it's about to release next, then call `make build`, `make build-dev`, `make build-nightly`, or `make version` unchanged.
Expand Down
38 changes: 35 additions & 3 deletions scripts/make/build.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@
artifact). Called by `make build`.
build.py --dev Build a dev wheel with a timestamped PEP 440 .dev
version. Called by `make build-dev`.
build.py --nightly Build the nightly wheel published to PyPI, with a
build.py --nightly Build the nightly wheel published to PyPI as
`<name>-nightly` (e.g. coreai-opt-nightly), with a
timestamped PEP 440 .dev version (e.g.
0.2.2.dev202607231430), via `uv build --no-sources`.
Called by `make build-nightly`.
Expand All @@ -31,12 +32,16 @@
import argparse
import subprocess
import sys
import tomllib
from contextlib import contextmanager, nullcontext
from pathlib import Path
from typing import TYPE_CHECKING

# build.py ships to OSS, where `external/` is the repo root, so it imports the
# helpers via the plain `scripts.*` namespace — that resolves to `external/scripts/*`
# internally (PEP 420 merging) and to the OSS-root `scripts/*` post-export. The
# internal-only scripts use the explicit `external.scripts.*` form instead.
from scripts._pyproject_utils import rename_package as _rename_package
from scripts._utils import find_repo_root as _find_repo_root
from scripts.release.release_utils import (
get_dev_version_override,
Expand All @@ -48,6 +53,9 @@
write_version,
)

if TYPE_CHECKING:
from collections.abc import Iterator


def run_build(*, no_sources: bool) -> None:
"""Run ``uv build`` to produce the wheel and sdist.
Expand All @@ -70,6 +78,26 @@ def run_build(*, no_sources: bool) -> None:
sys.stdout.write("Build complete! Check dist/ directory\n")


@contextmanager
def _nightly_package_name(repo_root: Path) -> Iterator[None]:
"""Build under ``<name>-nightly`` while the context is open.

Nightlies go to their own PyPI project, so the main project's release
history lists only releases. Renames ``[project] name`` in
``pyproject.toml`` and writes the original file back on exit.
"""
pyproject = repo_root / "pyproject.toml"
original = pyproject.read_text(encoding="utf-8")
name = tomllib.loads(original)["project"]["name"]
nightly_name = f"{name}-nightly"
_rename_package(repo_root, name, nightly_name)
sys.stdout.write(f"Package: {nightly_name}\n")
try:
yield
finally:
pyproject.write_text(original, encoding="utf-8", newline="\n")


def _build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Build the coreai-opt package.")
mode = parser.add_mutually_exclusive_group(required=True)
Expand All @@ -81,7 +109,10 @@ def _build_parser() -> argparse.ArgumentParser:
mode.add_argument(
"--nightly",
action="store_true",
help="Build the nightly wheel published to PyPI, with a timestamped PEP 440 .dev version",
help=(
"Build the nightly wheel published to PyPI as `<name>-nightly`, with a "
"timestamped PEP 440 .dev version"
),
)
mode.add_argument(
"--no-sources",
Expand Down Expand Up @@ -110,7 +141,8 @@ def main() -> None:
sys.stdout.write(f"Version: {build_version}\n")
# Only the local dev build resolves [tool.uv.sources]; the published
# builds (release and nightly) use `uv build --no-sources`.
run_build(no_sources=not args.dev)
with _nightly_package_name(repo_root) if args.nightly else nullcontext():
run_build(no_sources=not args.dev)
finally:
# Restore the exact bytes read before the build rewrote the version.
restore_about(about)
Expand Down
Loading