Skip to content

feat: v0.3.0 a command's checks run in lanes, at the same time - #5

Merged
fatherlinux merged 5 commits into
mainfrom
feature/parallel-lanes
Nov 4, 2026
Merged

fatherlinux merged 5 commits into
mainfrom
feature/parallel-lanes

Conversation

@fatherlinux

@fatherlinux fatherlinux commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Why

sdlc ci runs every check one after another. In trentina the image build depends on none of mypy, pytest, semgrep or pip-audit, but it waits for all four, so the job takes the sum of its checks rather than the length of its longest chain.

Check PR, gloval self-hosted main, GitHub-hosted 4 vCPU
uv sync + mypy 16s 33s
pytest 83s 148s
semgrep 14s 27s
pip-audit 5s 4s
image build 104s 8m20s
in-image tests ~50s 3m01s
whole job 5m09s 16m05s

What changed

A plan line gains a lane as its second field: check TAB lane TAB mode TAB payload. Steps of one lane run in sequence, in plan order; lanes run at the same time.

The image assigns the lanes, by two rules that name no repo:

  • an image-build check and every pytest-in-image check that runs the image it tags are one lane — the build has to finish before the tests start, and two builds writing one tag must not race;
  • everything else is the main lane, in plan order. That is where uv sync happens: mypy, pytest and pip-audit each sync the project venv in the shared cache volume, and one lane makes those sequential. uv does lock the target environment, but a gate should not bet its venv on a lock it did not take.

The launcher only carries it out — no check name appears in bin/sdlc.

For trentina's ci that is two lanes of roughly equal length: main at about 155s and the image chain at about 154s on gloval, against 309s sequential.

What is unchanged

  • A repo with no image-build check gets one lane, and one lane is the launcher's sequential path — same run, same output, same code path as before. sdlc push has no image chain either, so push is untouched.
  • The exit contract. 0 clean, 1 found, 2 could not run; a 2 anywhere beats a 1 whichever lane finished first; a failed step ends its own check and every other check still runs.
  • The host contract. POSIX sh, only git and podman, podman build|run only, a host step split on the unit separator and never evaled, and the plan still refuses a tab, a newline or that separator in any field — the lane included.

CPU and output

  • The budget is for the whole gate, not for each container: it is divided between the lanes, floor of one. Two lanes at half the budget take the share of the machine one lane at the whole budget took, so sdlc ci on a laptop is no busier than before.
  • Each lane's output is buffered per check and printed whole, in plan order, as each check finishes. Buffering collapses the timestamps people read off the == <step> markers, so each block now ends with == <check> passed in Ns.

Tests

tests/test_plan.py pins the lane assignment: a chain is one lane whichever order the repo wrote it in, two builds of one tag share a lane, two tags get two, tests on a prebuilt image get their own, every uv sync is in one lane, and a plan line has four fields with each check's steps contiguous in its lane.

tests/test_host_contract.sh pins the executor behaviourally, under a restricted PATH, with a podman shim whose steps print markers, sleep and exit on command:

  • both lanes really were running at the same time (so the ordering test is not a sequential run that could not interleave if it tried);
  • each check's block is printed whole, in plan order, never interleaved;
  • a 2 beats a 1 in both finishing orders;
  • a failed step ends its own check, and the next check in that lane plus everything in the other lane still runs;
  • a budget of 4 is split 2 and 2;
  • a single-lane plan does not go through the lane executor at all.

Each of those was checked against a deliberately broken launcher — forced serial, broken aggregation, unbuffered output — and fails on it.

tests/contract.sh adds the seam on the real image: a repo with no image build plans one lane; a repo with a chain plans two with only the chain off main, and mypy/pytest/pip-audit in one.

Second round on this branch

  • Nothing runs beside a check that rewrites the tree. A command containing ruff-check, ruff-format or gourmand gets no lanes at all — a build context or an in-image mount reading those files while ruff rewrites them is a check judging a tree that no longer exists. That makes sdlc commit sequential by definition. Reachable today only by a config nobody has written, which is the moment to close it.
  • The step index is resolved once, not per step. Gatehouse flagged an awk per step scanning the whole check-order file. The magnitude was trivial — N is about eight, and each step it sat next to starts a container — but the fix removes code: the single awk that already splits the plan by lane now numbers each check by first appearance and writes the index as the step's first field.

Drive-by

A __pycache__/*.pyc was committed and nothing ignored it, so running python3 sdlc or the unit tests left a modified tracked file behind. Untracked, and .gitignore grew the entry.

Still to do after this merges

Report sdlc ci on trentina for gloval and a GitHub-hosted runner against the table above. On a 4-vCPU runner the two chains compete for CPU, so that number gets measured rather than assumed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01R7Y1jesjfgxHeqtKJCvay6

`sdlc ci` ran every check one after another, so trentina's image build --
which depends on none of mypy, pytest, semgrep or pip-audit -- waited for
all four. The job took the sum of its checks rather than the length of its
longest chain: 5m09s on the self-hosted runner, 16m05s on a GitHub-hosted
one, of which the image chain was 2m34s and 11m21s.

Each plan line now carries a lane as its second field. Steps of one lane
run in sequence, in plan order; lanes run at the same time. The image
assigns them, by two rules that name no repo: an image-build check and
every pytest-in-image check that runs the image it tags are one lane, and
everything else is the main lane -- which is where uv sync happens, so two
syncs can never land in one venv at once.

A repo with no image-build check gets one lane, and one lane is the
launcher's sequential path: the same run, the same output, as before.

The CPU budget is for the whole gate and is divided between the lanes, so
two lanes take the share of the machine one lane used to. Each lane's
output is buffered and printed whole, in plan order, and each block ends
with how long its check took, because buffering is what the `== <step>`
timestamps used to tell you.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jesjfgxHeqtKJCvay6

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found 1 issues (1 medium)

Full review.

8 finding(s) dropped: their evidence is not in the diff.

Comment thread bin/sdlc Outdated
_mode=${_rest%%"$TAB"*}
_payload=${_rest#*"$TAB"}
case " $_failed " in *" $_check "*) continue ;; esac
_i=$(awk -v c="$_check" '$0==c {print NR; exit}' "$OUT/order") || _i=

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Performance Check): Each plan step starts an awk process that scans the full check-order file to find its index. With N checks and roughly one step per check, this adds O(N²) scanning and process launches; plans with many configured checks or lanes can make this overhead noticeable.

Suggestion: Precompute a check-to-index mapping once, or include each check's index when creating the lane step files, so each step can look up its output index without rescanning the full order.

Evidence:

_i=$(awk -v c="$_check" '$0==c {print NR; exit}' "$OUT/order") || _i=

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in eb7742e

The O(N²) is real but the magnitude was not: N is the number of checks in a command, about eight, and every step it sat next to starts a container. What made the suggestion worth taking is that it removes code. The single awk that already splits the plan by lane now numbers each check by first appearance and writes the index as the step's first field, so run_lane reads it instead of looking it up, and the per-step process and the die guarding a lookup that can no longer fail both go away.

… index is resolved once

Two findings from the first round on this branch.

A check in WRITES_TREE -- ruff-check, ruff-format, gourmand -- now takes the
whole command sequential. A build context or an in-image mount reading those
files while ruff rewrites them is a check judging a tree that no longer
exists. `sdlc commit` is ruff, so commit is sequential by definition, which
is what "seconds, offline, free" wanted anyway. Reachable today only by a
config nobody has written, which is the moment to close it.

And each lane step was starting an awk to scan the whole check-order file for
its own index -- O(N*N) and a process per step, next to steps that each start
a container. The index is now resolved by the single awk that already splits
the plan by lane and written as the step's first field, which is less shell,
not more.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jesjfgxHeqtKJCvay6

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found no issues.

Reviewed 1 commit since 1409cf7 (incremental).

2 finding(s) dropped: their evidence is not in the diff.

… done

A lane wrote its done marker after clearing its EXIT trap, so a failed write
in that window left no marker at all -- and `_lanes_running` would then return
"still running" forever while the printer slept a second at a time. The trap
now stays armed through that write: it writes the same marker, and the only
reader asks whether the file exists.

And the printer gives up ORDERING after an hour with no block finishing,
rather than polling for one. `wait` below it still blocks for each lane's real
status, so giving up on the order never guesses an answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jesjfgxHeqtKJCvay6

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found 1 issues (1 medium)

Reviewed 1 commit since eb7742e (incremental).

Comment thread bin/sdlc
# that finishes early is HELD BACK, never dropped and never cut into another
# lane's output; the lanes themselves keep running while this waits.
_i=1
_stall=0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Test Coverage): The new stall timeout adds a distinct ordering fallback, but no corresponding test is shown for reaching the threshold, printing available output, and still waiting for each lane’s real status. This leaves the timeout boundary and fallback behavior unverified.

Suggestion: Add a test with a controllable stall threshold or mocked sleep that exercises the timeout, verifies available output is printed without blocking on ordering, and confirms lane statuses are still collected.

Evidence:

_stall=0
...
[ "$_stall" -ge 3600 ] && break
_stall=$((_stall + 1))
sleep 1

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair, and answering it found the better shape.

The threshold could not have a test at 3600 seconds, so it is now SDLC_LANE_STALL, and tests/test_host_contract.sh reaches the boundary with a one-second threshold against a lane that takes six. It asserts all three things you named: the launcher says it has stopped waiting on the order, the block that did finish in time is still printed, and each lane's real status is still collected — the run exits 1 from the lane that was still sleeping when the printer gave up.

The part worth more than the test: the remaining blocks are now printed after every lane has been reaped rather than before. Two reasons. The status is the answer and is never inferred from what a file happened to contain; and once every lane has stopped, nothing is still writing, so a late block is printed whole instead of being read out from under the process writing it. The test pins that by construction — the late block's last line is written five seconds after the printer gives up, so it can only appear if the block was printed after the reap. Swapping the reap and the flush back makes that assertion fail, which I checked.

Reached only by a lane killed outright: SIGKILL, the OOM killer, a cancelled CI job. Every other ending runs the lane's EXIT trap, which is what writes the marker the printer waits on.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in f3931e0

(Re-posting in the form triage parses — my earlier reply on this thread said the same thing but did not open with fixed in <sha>, so it was never counted.)

The threshold could not have a test at 3600 seconds, so it is now SDLC_LANE_STALL, and tests/test_host_contract.sh reaches the boundary with a one-second threshold against a check that takes six. It asserts all three things you named: the launcher says it has stopped waiting on the order, the block that did finish in time is still printed, and each lane's real status is still collected — the run exits 1 from the lane still sleeping when the printer gave up.

Answering it also found the better shape, which was worth more than the test. The remaining blocks are now printed after every lane has been reaped rather than before, so the status is never inferred from what a file happened to contain, and a late block is printed whole instead of being read out from under the process writing it. The test pins that by construction: the late block's last line is written five seconds after the printer gives up, so it can only appear if the block was printed after the reap. Swapping the reap and the flush back makes the assertion fail, which I verified.

…t is left

Gatehouse: the stall threshold had no test. It could not have one at 3600
seconds, so it is now SDLC_LANE_STALL, and the host contract test reaches the
boundary with a one-second threshold against a lane that takes six.

Answering it found the better shape. The remaining blocks are now printed
AFTER every lane has been reaped, not before: the status is the answer and is
never guessed from what a file happened to contain, and once every lane has
stopped nothing is still writing, so a late block is printed whole instead of
being read out from under the process writing it. The test pins that -- the
late block's last line is written five seconds after the printer gave up, so
it can only be there if the block was printed after the reap -- and fails if
the two are swapped back.

The launcher also says when it stops waiting on the order, rather than
silently printing out of sequence.

Only a lane killed outright reaches any of this: SIGKILL, the OOM killer, a
cancelled CI job. Every other ending runs the lane's EXIT trap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jesjfgxHeqtKJCvay6

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found 2 issues (1 high, 1 low)

Reviewed 1 commit since c8c3aad (incremental).

3 finding(s) dropped: their evidence is not in the diff.

Comment thread CHANGELOG.md Outdated
### Added

- `SDLC_LANE_STALL` (default 3600 seconds): how long the printer waits for a
lane that has stopped answering. Only a lane killed outright -- SIGKILL, the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH (Documentation): The changelog says only an outright-killed lane reaches the stall threshold, but the timer is based on completed blocks, not detecting a kill. A slow or hung check that produces no completed block for the configured interval also reaches the threshold.

Suggestion: Describe the threshold as applying when no block finishes for the configured interval, and clarify that an outright-killed lane is one example.

Evidence:

Only a lane killed outright -- SIGKILL, the OOM killer, a cancelled CI job -- gets there

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 867a366

Correct, and it is my error rather than a wording nit. The timer counts seconds with no check finishing, so a check that honestly takes longer than the threshold trips it too. "Only an outright-killed lane reaches it" was false in four places — bin/sdlc's comment, the message it prints, the CHANGELOG and the README — and all four now say what is measured.

A killed lane is still the case worth guarding, because it runs no EXIT trap and so never reports at all, which leaves the printer polling for a marker that is never coming. A slow check is the other way in. That is why the default is an hour, why the message now names SDLC_LANE_STALL as the lever, and why tripping it is cosmetic: the order is abandoned, the verdict is not, because the reap is the authority on every lane's status.

Comment thread README.md Outdated
how long its check took, because buffering is what the `== <step>` timestamps
used to tell you.

A lane killed outright — SIGKILL, the OOM killer, a cancelled CI job — runs no

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW (General Review): The documentation says only a lane killed outright reaches this threshold, but the timeout also triggers when a lane is simply slow and does not finish a block before the configured interval. This overstates what the threshold indicates; the same claim appears in the changelog.

Suggestion: Describe the condition as no block completing within the configured interval, and present an outright-killed lane as one possible cause.

Evidence:

A lane killed outright — SIGKILL, the OOM killer, a cancelled CI job — runs no cleanup and never reports, so the printer stops waiting on the order after `SDLC_LANE_STALL` seconds (default 3600)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 867a366

Correct, and it is my error rather than a wording nit. The timer counts seconds with no check finishing, so a check that honestly takes longer than the threshold trips it too. "Only an outright-killed lane reaches it" was false in four places — bin/sdlc's comment, the message it prints, the CHANGELOG and the README — and all four now say what is measured.

A killed lane is still the case worth guarding, because it runs no EXIT trap and so never reports at all, which leaves the printer polling for a marker that is never coming. A slow check is the other way in. That is why the default is an hour, why the message now names SDLC_LANE_STALL as the lever, and why tripping it is cosmetic: the order is abandoned, the verdict is not, because the reap is the authority on every lane's status.

Gatehouse, and it is my error rather than a wording nit. The timer counts
seconds with no check FINISHING, so a check that honestly takes longer than
the threshold trips it too -- the claim that only an outright-killed lane
reaches it was false in bin/sdlc's comment, the message it prints, the
CHANGELOG and the README alike.

All four now say what is measured. A killed lane is the case worth guarding,
because it runs no EXIT trap and so never reports at all, leaving the printer
polling for a marker that is never coming. A slow check is the other way in,
which is why the default is an hour, why the message names SDLC_LANE_STALL,
and why tripping it is cosmetic: the order is abandoned, the verdict is not,
because the reap is the authority.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jesjfgxHeqtKJCvay6

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found no issues.

Reviewed 1 commit since f3931e0 (incremental).

1 finding(s) dropped: their evidence is not in the diff.

@fatherlinux
fatherlinux merged commit 3a9f864 into main Nov 4, 2026
28 of 34 checks passed
@fatherlinux
fatherlinux deleted the feature/parallel-lanes branch November 4, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant