[bpamiri]

Closes #4502

Problem

heartbeat() didn't change when a dead worker's job was reclaimed. A run is capped in-process at the job's timeout, and the reaper waits at least twice that (claimTimeout + Max(60, claimTimeout)) whether the job heartbeats or not. So a long job whose worker died sat for twice its timeout no matter how often it had heartbeated.

Change

A job that has heartbeated is now reclaimed once its heartbeats stop for its heartbeat grace, even inside its timeout. For a row with a heartbeatAt, the reaper uses the earlier of the claim window (unchanged) and the grace.

Rows that never heartbeat behave exactly as before.

The grace.

• Setting. set(jobsHeartbeatGraceSeconds = n), default 300.
• Turning it off. 0 turns it off.
• Minimum. Values below 30 are raised to 30, so one slow heartbeat can't get a live job reclaimed.
• Per job. A job can set its own this.heartbeatGrace, including 0. The reaper resolves it once per job class.

What a reclaim does. It is a normal reap with the usual outcome:

• a retry (idempotent jobs);
• interrupted (non-idempotent jobs);
• failed, when retries are exhausted.

Claim fencing still stops the original attempt from completing over the new one. A final reclaim fires onFailure(isFinal = true) like any other reap (#4498).

Exclusive jobs (#4496). A heartbeat renews the lease to grace + Max(60, grace) from now, instead of the claim window. A job that stops heartbeating loses its lease on the same clock it is reclaimed on, so its retry isn't stuck behind a lease from the old claim window.

Scan floor. The reaper's candidate scan now starts at 30 seconds idle instead of 60, the smallest possible grace. The per-row windows are unchanged.

What heartbeat() doesn't do. It still never extends a job's timeout, which stays the hard cap on one attempt. Its docblock now says so, and says:

• heartbeat at least every grace / 3;
• a stopped heartbeat isn't proof that the worker died. A perform() stuck in a long call that can't heartbeat is reclaimed too, so either heartbeat around such calls or raise the grace.

Out of scope. Letting heartbeats extend the in-process deadline: an idle timeout with a separate hard maximum, as an opt-in this.timeoutMode = "idle" with this.maxRuntime. It's noted in #4502 as a possible later issue.

Tests

JobHeartbeatGraceSpec (7). Every row claims a one-hour timeout, so only the grace can reap it.

• Stopped heartbeats: a job whose heartbeats stopped 10 minutes ago is reclaimed.
• Recent heartbeat: one that heartbeated a minute ago isn't.
• Never heartbeated: a job idle 10 minutes is left to its claim window, as before.
• Turned off: with jobsHeartbeatGraceSeconds = 0 nothing is reclaimed by the grace.
• Per-class: this.heartbeatGrace = 1200 and this.heartbeatGrace = 0 both override the setting.
• Minimum: a grace below 30 is raised to 30.
• Lease: renewal uses grace + Max(60, grace) with a grace set, and the lease's own window without one.

On develop:

• 1 fails: the job whose heartbeats stopped isn't reclaimed.
• 2 error: the grace API doesn't exist.

Local results:

• Full core suite, Lucee 7 + SQLite: 7232 pass, 0 fail, 0 error.
• Jobs specs, Lucee 7: green on all seven databases; PostgreSQL and CockroachDB also green on a second run against the same database.
• Jobs specs, BoxLang: green on SQLite, PostgreSQL and CockroachDB.
• Jobs specs, Adobe 2023: green on SQLite and MySQL.
• Jobs specs, Adobe 2025: green on SQLite and SQL Server.
• Complexity gate: passes.

Docs

The guide's heartbeat section (#4485) covers the grace, the cadence (grace / 3), the "not proof of death" caveat and the timeout staying the hard cap. Changelog fragment included.

🤖 Generated with Claude Code

[github-actions]

Wheels Test Results

236 084 tests   230 841 ✅  1h 43m 42s ⏱️
 18 912 suites    5 243 💤
     32 files          0 ❌

Results for commit 0bc89af.

[bpamiri]

rev1-r3 review @ 0bc89af: APPROVE

The change follows the agreed #4502 design.

• Grace. jobsHeartbeatGraceSeconds (default 300, 0 = off, raised to a minimum of 30) or the class's own this.heartbeatGrace, resolved per class through $jobHeartbeatGrace(). That is memoised per worker, and a class that can't load falls back to the app setting.
• Reclaim. A row that has heartbeated (hasBeat, read as a CASE flag so no timestamp is diffed in CFML) is reapable at Min(claim window, grace). A row that never heartbeated keeps the claim window alone. this.timeout stays the hard cap, because the grace can only shorten the window.
• Lease. heartbeat() renews the feat(job): exclusive jobs and concurrency keys on a shared lease lock #4496 lease to now + grace + Max(60, grace), or the original window when the grace is off. A job that stops heartbeating therefore keeps its lease past the point it's reclaimed, and the retry defers on the lease until it lapses rather than overlapping.

Correctness checks:

• No live job is reaped on its claim time. Every stale guard uses $lastSeenSql() (COALESCE(heartbeatAt, updatedAt)): the candidate SELECT, $scheduleRetry, $markFailed and $markInterrupted. So a job claimed an hour ago that heartbeated 40 s ago doesn't match a 300 s cutoff.
• Only this attempt's heartbeats count. Both claim paths set heartbeatAt = NULL (JobWorker and Job.$processJob).
• The lower floor (60 → 30 s) only widens the candidate set. Each row's own cutoff still decides.
• Heartbeat-reaped final outcomes need no new wiring. They go through the reaper branches that feat(job): lifecycle hooks, an app-wide failure hook, and a stored result #4498 wired to onFailure(isFinal = true) ($markFailed, and $markInterrupted for non-idempotent jobs).
• Upgrade impact: none. heartbeat() doesn't exist in v4.1.2, so no released app relies on the old "heartbeat more often than the timeout" advice.
• Without a heartbeatAt column (an ALTER-blocked table), there's no hasBeat and the claim window applies, unchanged.

Matrix 37849369584 (full, on this head), read with my leg reader: 32 legs, all tests > 0 (7372–7389), 0 legs with fail or error. JobHeartbeatGraceSpec runs 7 specs on every leg, 0 bad. The existing JobHeartbeat* and JobLifecycleHooksSpec (15 each) are unchanged and green on every leg.

PR CI: 23 pass, 2 skipping, 0 fail. MERGEABLE. develop has moved by 5 commits since the base, all web/ blog and doc commits, with no overlap.

INFO, not blocking:

• The 30 s floor has no positive spec. The "below 30 is raised to 30" spec checks that a row that heartbeated 15 s ago survives, which would also pass under the old 60 s floor. A row with grace 30 and its last beat about 45 s ago being reclaimed would pin the floor change itself.
• The 4.2 guides don't document heartbeat() or the grace yet (no heartbeat hit under v4-2-0 outside channels). This belongs in the multi-server jobs guide fold (docs(web): add the 4.2 "Running Jobs on Multiple Servers" guide #4501), including the "heartbeat at least every grace / 3" guidance and the stuck-call caveat from the changelog.