Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,21 @@ All notable changes to this project are documented here, following

## [Unreleased]

### Changed

- **`sdlc push` and `sdlc ci` now run `lint`**: `ruff check --no-fix .` and
`ruff format --check .`, read-only, first in push (#48). Ruff lived only in
the commit tier, where it runs with `--fix` and passes anything fixable, so
`sdlc ci` judged no lint at all and a `--no-verify` commit reached a green
required check unlinted. **This can turn a consumer's CI red** on its next
bump: a tree that ruff would change now fails. Measured against the 36
consuming repos' default branches with this image's ruff 0.17.1: 35 are
clean and data-collector has one fixable I001. Fix with `sdlc commit`, or
opt out with `[tool.sdlc.push] skip = ["lint"]`, which removes it from ci
as well; `[tool.sdlc.ci] skip` may not name it.
- README: `gatehouse` is no longer listed as exempt from python-sdlc; it runs
it as of gatehouse#134 (constitution#138).

### Fixed

- `sdlc-validate` now says "cannot read the constitution checkout" when the
Expand Down
17 changes: 11 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ Every CrunchTools Python check in one container image, run the same way on a
laptop and in CI.

Constitution XII (v1.30.0) requires it for every Python repo — any repo with a
`pyproject.toml` — except `gourmand`, `gatehouse` and this one, which build what
the image contains.
`pyproject.toml` — except `gourmand` and this one, which build what the image
contains. `gatehouse` runs it too, as of gatehouse#134 (constitution#138).

## Three commands, three config blocks

Expand Down Expand Up @@ -101,8 +101,13 @@ nested `pyproject.toml` still applies to the files under it.
| Command | Runs |
|---|---|
| `commit` | ruff-check, ruff-format, gourmand, hooks-installed |
| `push` | mypy, pytest, gourmand, gatehouse, semgrep, pip-audit, constitution, codeql |
| `ci` | mypy, pytest, semgrep, pip-audit, constitution, codeql |
| `push` | lint, mypy, pytest, gourmand, gatehouse, semgrep, pip-audit, constitution, codeql |
| `ci` | lint, mypy, pytest, semgrep, pip-audit, constitution, codeql |

`ruff-check` and `ruff-format` fix what they find, so they pass on anything
fixable. `lint` is the same two tools as a verdict — `ruff check --no-fix` and
`ruff format --check`, changing nothing — and it is the only ruff that push
and ci run.

`sdlc tiers` prints what each command runs in the repo you are in, and which
lane each check runs in.
Expand Down Expand Up @@ -197,7 +202,7 @@ One rule for all three blocks:

- `checks` **adds** to that command's defaults; `skip` removes from them.
What push skips, ci skips too.
- **`[tool.sdlc.ci] skip` may not name a push verdict** — mypy, pytest,
- **`[tool.sdlc.ci] skip` may not name a push verdict** — lint, mypy, pytest,
semgrep, pip-audit, constitution or codeql. Constitution XII says CI runs
push's checks except Gourmand and Gatehouse, and this tool decides which two
those are. Skipping a verdict in ci alone leaves it on pre-push, which is a
Expand Down Expand Up @@ -238,7 +243,7 @@ tests = ["tests/image/"]
| `semgrep` | `config` (default `.semgrep`), `tests` (default `.semgrep/run-tests.sh`), `paths` (default `["."]`) |
| `codeql` | `config` (default `.github/codeql/codeql-config.yml`), `baseline` (default `.sdlc/codeql-baseline.json`) |
| `constitution` | `manifest` (default `.specify/memory/constitution.md`) |
| `ruff-check`, `ruff-format`, `mypy`, `gourmand`, `gatehouse`, `pip-audit`, `hooks-installed` | none |
| `ruff-check`, `ruff-format`, `lint`, `mypy`, `gourmand`, `gatehouse`, `pip-audit`, `hooks-installed` | none |

### Kinds, for a repo's own checks

Expand Down
33 changes: 33 additions & 0 deletions sdlc
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,13 @@ DEFAULTS = {
# Everything that renders a verdict, once, over origin/<default>...HEAD.
# Never a container build: that is minutes, and a push you wait minutes for
# is a push people stop making.
#
# `lint` is ruff's VERDICT: the commit tier's two ruff steps fix what they
# can and pass, so ci -- which is push minus the reviewers -- had no ruff at
# all, and a `--no-verify` commit reached a green required check unlinted
# (python-sdlc#48). First, because it is the cheapest thing here to fail.
"push": [
"lint",
"mypy",
"pytest",
"gourmand",
Expand Down Expand Up @@ -867,6 +873,32 @@ def t_ruff_format(s: dict) -> int:
return run(["ruff", "format", "."])[0]


def t_lint(s: dict) -> int:
"""What ruff-check and ruff-format would change, as a finding rather than a fix.

The commit tier's ruff steps rewrite the tree, so a fixable violation passes
there -- which is right on a laptop and useless as a verdict. This changes
nothing, so it runs on a read-only tree in the main lane. Both always run,
so one push reports both kinds of finding.
`--no-cache` because the tree is read-only and `.ruff_cache` lives in it.

`--no-fix-only` because a repo's `[tool.ruff] fix-only = true` beats
`--no-fix`: ruff then reports nothing and exits 0, so lint would pass an
F821 without a word.
"""
need_project()
step("ruff check --no-fix .")
checked = run(["ruff", "check", "--no-fix", "--no-fix-only", "--no-cache", "."])[0]
step("ruff format --check .")
formatted = run(["ruff", "format", "--check", "--no-cache", "."])[0]
# A syntax error is a finding to `ruff check` (1) and unparsable to the
# formatter (2). It is the code that is wrong, not the gate, so a finding
# from either outranks the other's 2.
if FOUND in (checked, formatted):
return FOUND
return max(checked, formatted)


def t_mypy(s: dict) -> int:
"""Type check what `[tool.mypy] files` names. mypy's own exit 2 is ours too."""
need_project()
Expand Down Expand Up @@ -1581,6 +1613,7 @@ def t_codeql_baseline() -> int:
BUILTINS = {
"ruff-check": t_ruff_check,
"ruff-format": t_ruff_format,
"lint": t_lint,
"mypy": t_mypy,
"pytest": t_pytest,
"gourmand": t_gourmand,
Expand Down
55 changes: 54 additions & 1 deletion tests/contract.sh
Original file line number Diff line number Diff line change
Expand Up @@ -262,12 +262,23 @@ for target in ruff-check ruff-format semgrep; do
fi
done

# lint is the verdict half of those two, and it runs on a READ-ONLY tree: push
# and ci mount it that way, so a ruff that tried to write .ruff_cache would be a
# check that cannot run.
if out=$(podman run --rm -v "$FIXTURE:/repo:ro,z" -w /repo \
--entrypoint="" "$IMAGE" sdlc lint 2>&1); then
ok "lint runs clean on a read-only fixture project"
else
bad "lint failed on a clean, read-only fixture project"
printf '%s\n' "$out" | sed 's/^/ /' | tail -8
fi

# The plan the launcher reads must name every default check, in order, and
# resolve a repo's own setting. This is the seam between the TOML the image
# reads and the shell that acts on it, so it is checked on the real image.
plan=$(podman run --rm -v "$FIXTURE:/repo:ro,z" -w /repo --entrypoint="" \
"$IMAGE" sdlc plan push 2>&1 | cut -f1 | tr '\n' ' ')
want="mypy pytest gourmand gatehouse semgrep pip-audit constitution codeql "
want="lint mypy pytest gourmand gatehouse semgrep pip-audit constitution codeql "
if [ "$plan" = "$want" ]; then
ok "sdlc plan push lists the push defaults in order"
else
Expand Down Expand Up @@ -346,6 +357,48 @@ if podman run --rm -v "$FIXTURE:/repo:ro,z" -w /repo --entrypoint="" \
else
ok "ruff-check fails on broken code"
fi
# A finding ruff could fix is still a finding here -- that is the whole of #48.
# Both tools must report: a lint that stopped at the first failure would hide
# the formatter's findings behind the linter's.
out=$(podman run --rm -v "$FIXTURE:/repo:ro,z" -w /repo --entrypoint="" \
"$IMAGE" sdlc lint 2>&1) && rc=0 || rc=$?
if [ "$rc" -eq 1 ]; then
ok "lint fails on code ruff would fix, without fixing it"
else
bad "lint exited $rc on unused imports and bad formatting, want 1"
fi
case "$out" in
*F401*"1 file would be reformatted"*)
ok "lint reports the linter's and the formatter's findings in one run" ;;
*)
bad "lint did not report both an F401 and a file to reformat"
printf '%s\n' "$out" | sed 's/^/ /' | tail -12 ;;
esac
# One fault each, so a lint that judged only one tool, or turned a finding into
# a 2, cannot pass. lint_case <what> <file body> <want> [ruff config line]
lint_case() {
case_dir=$(mktemp -d)
printf '[project]\nname = "lint-case"\nversion = "0.0.0"\n\n[tool.ruff]\n%s\n' \
"${4:-}" > "$case_dir/pyproject.toml"
printf '%b' "$2" > "$case_dir/x.py"
before=$(cat "$case_dir/x.py")
podman run --rm -v "$case_dir:/repo:ro,z" -w /repo --entrypoint="" \
"$IMAGE" sdlc lint >/dev/null 2>&1 && rc=0 || rc=$?
after=$(cat "$case_dir/x.py")
rm -rf "$case_dir"
if [ "$rc" -eq "$3" ] && [ "$before" = "$after" ]; then
ok "lint exits $3 on $1"
else
bad "lint exited $rc on $1, want $3 and an untouched file"
fi
}
lint_case "a lint finding alone" 'import os\n' 1
lint_case "a format finding alone" 'x = ( 1 )\n' 1
lint_case "a syntax error (a finding, not a gate that cannot run)" 'def f(:\n' 1
# A repo's fix-only silenced `--no-fix`: exit 0, nothing reported (#49 review).
lint_case "an F821 under fix-only = true" 'def f():\n return undefined_name\n' 1 \
'fix-only = true'

if podman run --rm -v "$FIXTURE:/repo:ro,z" -w /repo --entrypoint="" \
"$IMAGE" sdlc semgrep >/dev/null 2>&1; then
bad "semgrep passed a file matching its own fixture rule"
Expand Down
20 changes: 20 additions & 0 deletions tests/test_plan.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,26 @@ def cannot_run(fn, *args) -> str | None:
ci == [c for c in S.DEFAULTS["push"] if c not in S.CI_EXCLUDES],
)

# python-sdlc#48: the commit tier's ruff steps fix and pass, so without a
# non-mutating verdict in push, ci ran no ruff at all.
check("ci renders the ruff verdict", "lint" in ci, str(ci))
check(
"lint reads the tree, never writes it",
S.plan_lines(empty, "lint") == [("lint", "image-ro", "lint")],
)
push_lanes = S.lane_of(empty, S.resolve(empty, "push"))
check(
"lint keeps push's lanes",
"lint" not in S.WRITES_TREE and set(push_lanes.values()) == {S.MAIN_LANE, S.REVIEW_LANE},
str(push_lanes),
)
msg = cannot_run(S.resolve, {"ci": {"skip": ["lint"]}}, "ci")
check("ci may not skip lint", msg is not None and "lint" in msg, str(msg))
check(
"push skip of lint takes it out of ci too",
"lint" not in S.resolve({"push": {"skip": ["lint"]}}, "ci"),
)

cfg = {"push": {"skip": ["codeql"]}, "ci": {"checks": ["image"], "image": {"kind": "image-build"}}}
check("push skip removes a default", "codeql" not in S.resolve(cfg, "push"))
check("a push skip carries into ci", "codeql" not in S.resolve(cfg, "ci"))
Expand Down
Loading