Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# program.md is a frozen contract copied verbatim into the user's repository,
# and init.test.js parses it. Pin it to LF so a Windows checkout does not turn
# every paragraph boundary into CRLF under the test's feet.
templates/program.md text eol=lf
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
os: [ubuntu-latest, macos-latest, windows-latest]
node: ['20', '22']
runs-on: ${{ matrix.os }}
steps:
Expand Down
53 changes: 32 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,17 +173,21 @@ Three ways to end a run, in increasing order of force:
under way finishes, is measured and scored normally; the agent sees
`"stop_requested": true` on that verdict, applies it as usual (`KEEP`
stays, anything else resets), then exits the loop. Nothing is thrown away.
2. **`autor3search-javascript stop --force`** — writes the same request, then
sends `SIGTERM` to the running `eval`, which tears down its own child
process groups so no Vitest worker is left burning CPU. The agent sees
`"status": "ABORTED"`, exit code 2, and no `results.tsv` row — nothing was
measured, so nothing was recorded. `stop --force` reports what HEAD looks
like afterward; it does not touch the repository for you.
2. **`autor3search-javascript stop --force`** — writes a sticky forced-stop
marker. The running `eval` polls for it and cancels itself, and that
cancellation is what tears down the benchmark process tree, so no Vitest
worker is left burning CPU. The agent sees `"status": "ABORTED"`, exit
code 2, and no `results.tsv` row — nothing was measured, so nothing was
recorded. The marker is sticky on purpose: if nothing was running when you
forced the stop, it stands and aborts the *next* `eval` to start instead,
until you clear it with `stop --clear`. `stop --force` reports what HEAD
looks like afterward; it does not touch the repository for you.
3. **Ctrl+C** — the same abort path as `stop --force`, sent directly to a
foreground `eval`.

`autor3search-javascript stop --clear` cancels a pending stop request so the
loop continues — that is your decision, never the agent's to make on its own.
`autor3search-javascript stop --clear` cancels both a pending stop request and
a pending forced stop, so the loop continues — that is your decision, never
the agent's to make on its own.

## Commands

Expand Down Expand Up @@ -382,19 +386,26 @@ Two things to know if you are wiring this up on a fork or a new package:
|---|---|
| **Linux** | supported — CI runs the full suite on Node 20 and 22 |
| **macOS** | supported — CI runs the full suite on Node 20 and 22 |
| **Windows** | **not supported** |

On Windows nearly the whole suite passes, so measurement itself works. What does not
work is stopping: Node cannot deliver SIGINT to a child process group there
the way it does on POSIX, so an interrupted `eval` never reaches the `ABORTED`
path — it exits with a null code instead of 2 and can leave its claim behind.
An unattended harness that cannot be reliably stopped is not something to be
quiet about, so `doctor` says so on Windows rather than letting you find out
at 3am. Two further failures are the test suite's own POSIX assumptions
(a `chmod`-unreadable directory, and path separators) rather than product bugs.

WSL reports as Linux and is unaffected. Node 20 or newer is required
everywhere.
| **Windows** | supported — CI runs the full suite on Node 20 and 22 |

All three behave identically, by design. `stop` and `stop --force` write marker
files into the run's state directory rather than signalling a process: the
running `eval` polls for a forced stop and cancels itself, and that cancellation
is what tears the benchmark process tree down. A signal-based forced stop could
not behave the same way everywhere — Windows has no process-to-process SIGTERM,
so `eval` would be terminated outright and would never record what it abandoned.
Ctrl+C still works everywhere, because a console interrupt is real on Windows
too.

Three small things differ on Windows, none of them behavioural. `doctor` has no
load average to read there, so it makes one fewer check and says so rather than
reporting the fabricated zero the platform hands it. The state directory's
ownership and permission check is skipped, because Windows reports synthetic
mode bits and has no owner to compare against. And one test is skipped — the
one that makes a directory unlistable with `chmod`, which has no Windows
analogue.

WSL reports as Linux and is unaffected. Node 20 or newer is required everywhere.

## Limitations

Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@autor3search/javascript",
"version": "0.2.1",
"version": "0.2.2",
"description": "Autonomous AI-driven performance optimization for any JavaScript repository",
"type": "module",
"license": "MIT",
Expand Down
37 changes: 32 additions & 5 deletions src/cli/cmd-eval.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,13 @@ import { RUN_LOG_NAME, evalOnce } from '../pipeline.js'
import { RESULTS_PATH, appendRow, loadRows } from '../results.js'
import { BASELINE_FILE, WORKTREE_NAME, loadBaseline } from '../state/index.js'
import { claimEval } from '../state/lock.js'
import { stopRequested } from '../state/stop.js'
import { forceRequested, stopRequested } from '../state/stop.js'
import { REASON, STATUS, exitCode } from '../verdict.js'
import { expandSingleDashFlags, loadRepoConfig, resolveRun } from './context.js'

/** How often a running eval looks for a forced-stop marker. */
const FORCE_POLL_MS = 500

/**
* @param {string[]} args
* @param {{out: {write(s: string): void}, err: {write(s: string): void}}} io
Expand Down Expand Up @@ -72,9 +75,23 @@ export async function runEval(args, io) {
const onSignal = () => controller.abort()
process.on('SIGINT', onSignal)
process.on('SIGTERM', onSignal)
const offSignals = () => {
// The forced stop is a polled marker, not a signal — see state/stop.js for
// why. It is installed here, beside the handlers and BEFORE claimEval, for
// exactly the reason spelled out above: a marker already pending when this
// run starts must abort it at its first checkpoint, not after it has claimed
// the run and begun measuring.
const poll = setInterval(() => {
forceRequested(run.stateDir)
.then((forced) => {
if (forced) controller.abort()
})
.catch(() => {})
}, FORCE_POLL_MS)
poll.unref()
const detach = () => {
process.off('SIGINT', onSignal)
process.off('SIGTERM', onSignal)
clearInterval(poll)
}

let claim
Expand All @@ -83,7 +100,7 @@ export async function runEval(args, io) {
} catch (err) {
// dispatch() also runs in-process under test, so a leaked listener here
// would accumulate across runs rather than dying with the process.
offSignals()
detach()
io.err.write(`${err.message}\n`)
return 2
}
Expand Down Expand Up @@ -128,11 +145,21 @@ export async function runEval(args, io) {
if (controller.signal.aborted) {
// ABORTED is not a verdict: nothing was measured, so NO results.tsv row
// is written. The agent treats it as it would a FAIL.
//
// A sticky forced stop aborts every later eval on this tag too, so an
// abort that finds the marker still pending must say so and name the
// cure. It goes in `message` — which writeJson already emits and the
// human path already writes to io.err — rather than in a new line of
// output, because --json's contract is that stdout carries the verdict
// object and nothing else.
const forced = await forceRequested(run.stateDir).catch(() => false)
const aborted = {
status: STATUS.ABORTED,
reason: REASON.STOP_FORCED,
score: 0,
message: 'the experiment was interrupted before it could be measured',
message: forced
? 'the experiment was abandoned: a forced stop is pending for this tag — clear it with `stop --clear` before the next eval'
: 'the experiment was interrupted before it could be measured',
regressions: [],
warnings: [],
}
Expand All @@ -149,7 +176,7 @@ export async function runEval(args, io) {
}
throw err
} finally {
offSignals()
detach()
await closeLog(logStream)
await claim.release()
}
Expand Down
12 changes: 10 additions & 2 deletions src/cli/cmd-status.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import * as gitx from '../gitx.js'
import { RESULTS_PATH, loadRows } from '../results.js'
import { BASELINE_FILE, WORKTREE_NAME, loadBaseline } from '../state/index.js'
import { evalRunning } from '../state/lock.js'
import { stopRequested } from '../state/stop.js'
import { forceRequested, stopRequested } from '../state/stop.js'
import { expandSingleDashFlags, resolveRun } from './context.js'

/**
Expand Down Expand Up @@ -42,6 +42,7 @@ export async function runStatus(args, io) {
evalLine = `unknown (${err.message})`
}
const stopped = await stopRequested(run.stateDir)
const forced = await forceRequested(run.stateDir)

const field = (name, value) => io.out.write(`${name.padEnd(14)} ${value}\n`)
field('run tag', run.tag)
Expand All @@ -60,7 +61,14 @@ export async function runStatus(args, io) {
`${counts.crash} crash) — next is #${rows.length + 1}`,
)
field('eval', evalLine)
field('stop', stopped ? 'requested — the agent will end the run after the current experiment' : 'not requested')
field(
'stop',
forced
? 'FORCED — the next eval will abort at once; clear it with `autor3search-javascript stop --clear`'
: stopped
? 'requested — the agent will end the run after the current experiment'
: 'not requested',
)

io.out.write('\nto stop after the current experiment: autor3search-javascript stop\n')
io.out.write('to stop now, abandoning it: autor3search-javascript stop --force\n')
Expand Down
49 changes: 29 additions & 20 deletions src/cli/cmd-stop.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@
* and only then does the loop exit. Nothing is thrown away.
*
* `--force` is for when a long benchmark cannot be waited out. It writes the
* same request, then signals the running eval to abandon the experiment. It
* REPORTS what state that leaves the repository in; it does not drop anything
* for you, because deciding what to do with a half-finished experiment is the
* human's call.
* same request, then marks the run for the running eval to poll and abandon
* the experiment on its own. It REPORTS what state that leaves the repository
* in; it does not drop anything for you, because deciding what to do with a
* half-finished experiment is the human's call.
*/
import { parseArgs } from 'node:util'
import * as gitx from '../gitx.js'
import { evalRunning } from '../state/lock.js'
import { clearStop, requestStop } from '../state/stop.js'
import { clearStop, forceRequested, requestForceStop, requestStop } from '../state/stop.js'
import { expandSingleDashFlags, resolveRun } from './context.js'

/**
Expand Down Expand Up @@ -50,35 +50,44 @@ export async function runStop(args, io) {
await requestStop(run.stateDir)
io.out.write(`stop requested for ${run.tag}: the agent will end the run after the current experiment\n`)
if (!values.force) {
io.out.write('the experiment under way will still be measured, scored and applied\n')
// A plain `stop` only promises the current experiment survives when
// nothing is already asking to abandon it. A forced stop from EARLIER —
// `stop --force` issued when nothing was running, then never cleared —
// is still pending and sticky, and it will abort the next eval to start
// regardless of what this plain `stop` just requested.
if (await forceRequested(run.stateDir)) {
io.out.write('a forced stop from earlier is still pending, though, and will abort the next eval to start\n')
} else {
io.out.write('the experiment under way will still be measured, scored and applied\n')
}
io.out.write(`to cancel: autor3search-javascript stop -tag ${run.tag} --clear\n`)
return 0
}

// A corrupt pid file must not crash `stop --force` — it is refused by
// evalRunning rather than guessed at, but the request above already stands
// regardless, so fall back to "nothing known to signal" and keep going.
await requestForceStop(run.stateDir)

// Nothing is signalled between processes: the running eval polls this marker
// and cancels itself. That behaves identically on every platform — Windows
// has no process-to-process SIGTERM — and it cannot reach an unrelated
// process the way signalling a pid read from a file can, if the OS has
// recycled that pid since.
//
// evalRunning is therefore reporting, not mechanism. A corrupt pid file
// costs a line of the report and nothing else; the marker already stands.
let pid = null
let running = false
try {
;({ pid, running } = await evalRunning(run.stateDir))
} catch (err) {
io.err.write(`could not read the eval lock (${err.message}); assuming there is nothing to signal\n`)
io.err.write(`could not read the eval lock (${err.message}); cannot say whether an eval is running\n`)
}

if (running) {
// SIGTERM, not SIGKILL: eval installs a handler that tears down its child
// process groups. Killing it outright would leave Vitest workers running,
// burning CPU and corrupting every later measurement on this machine.
try {
process.kill(pid, 'SIGTERM')
io.out.write(`signalled eval (pid ${pid}) to abandon the current experiment\n`)
} catch (err) {
io.err.write(`could not signal eval (pid ${pid}): ${err.message}\n`)
}
io.out.write(`eval (pid ${pid}) will abandon the current experiment within a second\n`)
} else {
io.out.write('no eval is running, so there was nothing to interrupt — the request stands\n')
io.out.write('no eval is running, so there was nothing to interrupt — the forced stop stands and will abort the next eval to start\n')
}
io.out.write(`to cancel it: autor3search-javascript stop -tag ${run.tag} --clear\n`)

// Report, do not act — whether or not anything was actually interrupted,
// this is what the repository looks like right now. The commit an
Expand Down
37 changes: 8 additions & 29 deletions src/doctor.js
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,7 @@ export async function check(dir) {
const absDir = resolvePath(dir)

const platformCheck =
platform() === 'darwin'
? ['power', checkDarwin]
: platform() === 'linux'
? ['cpu governor', checkLinux]
: platform() === 'win32'
? ['platform', checkWindows]
: null
platform() === 'darwin' ? ['power', checkDarwin] : platform() === 'linux' ? ['cpu governor', checkLinux] : null

const checks = [
['node', checkNode],
Expand Down Expand Up @@ -106,6 +100,13 @@ function checkCpu() {
}

function checkLoad() {
// os.loadavg() is not implemented on Windows: it returns a hardcoded
// [0, 0, 0]. Reporting that would tell a user under heavy load that their
// machine is perfectly idle — worse than saying nothing, in a tool that
// exists to predict whether measurement will be reliable.
if (platform() === 'win32') {
return { name: 'load', detail: 'no load average on this platform — check the machine is quiet yourself', severity: SEVERITY.NA }
}
const [one] = loadavg()
const perCore = one / Math.max(1, cpus().length)
return {
Expand Down Expand Up @@ -144,28 +145,6 @@ async function checkVitest(dir) {
}
}

/**
* Windows is not a supported platform, and the reason is the stop path.
*
* 448 of 458 tests pass there, so most of the harness works — but Node cannot
* deliver SIGINT to a child process group on Windows the way it does on POSIX,
* so an interrupted `eval` does not reach the ABORTED path: it exits with a
* null code instead of 2, and the run may leave a claim behind. An unattended
* harness that cannot be reliably stopped is the wrong thing to be quiet about,
* so this says it up front rather than at 3am.
*
* WSL reports linux and is unaffected.
*/
function checkWindows() {
return {
name: 'platform',
detail:
'Windows is not a supported platform: an interrupted eval does not reach the ABORTED path, so a run ' +
'may not stop cleanly and can leave its claim behind. Measurement itself works. Use WSL for a supported setup.',
severity: SEVERITY.WARN,
}
}

function checkHeapHint() {
// Reported up front so a run does not discover mid-flight that the hint is
// unavailable, which would otherwise look like the benchmark allocating
Expand Down
11 changes: 11 additions & 0 deletions src/pipeline.js
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,17 @@ const GATE_VERDICT = {
* @returns {Promise<{result: object, measurements: {time: object[], bytes: object[]|null}|null}>}
*/
export async function evalOnce(o) {
// A sticky forced stop that is already pending when a run starts must not
// cost a full gate run before it is honoured: the scope diff, config hash,
// freeze restore and the gates themselves are all real work, and none of
// it should happen for an experiment that is about to be thrown away
// unmeasured. Checked here, before anything below touches git or the
// filesystem, rather than only after the gates (line 263's neighbour) —
// that later check exists for an abort that arrives WHILE gating is under
// way; this one is for an abort that was already true before evalOnce was
// ever called.
if (o.signal?.aborted) throw new Error('aborted before the experiment began')

const timeoutMs = parseDuration(o.cfg.timeout)

// ---- 1. Scope --------------------------------------------------------
Expand Down
15 changes: 15 additions & 0 deletions src/runner.js
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,21 @@ class Capped {
* Signals a child's whole process group. A negative pid addresses the group.
* Failures are swallowed: the group is already gone in the common case, and a
* teardown that throws would mask the real result being reported.
*
* On Windows there are no process groups, so `process.kill(-pid, …)` throws and
* the fallback terminates only the direct child — yet grandchildren still die,
* and runner.test.js proves it on every platform. The reason is libuv: it
* assigns each non-detached child to a global job object with
* JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, so when the direct child dies, its job
* handle closes and everything it spawned goes with it.
*
* That holds only because EVERY subprocess this harness starts is
* process.execPath — see adapters/bench/vitest.js and adapters/gates/*.js — so
* every intermediate is a Node process carrying such a job object. Two changes
* would silently break it and leave vitest's workers running on Windows,
* burning CPU and corrupting every later measurement: running a gate through
* `npm`/`npx` (a .cmd, so cmd.exe becomes the intermediate and there is no job
* object), or adding `detached: true` to an inner spawn.
*/
function killGroup(pid, signal) {
if (!pid) return
Expand Down
Loading