[bpamiri]

Closes #4479

Problem

There was no way to say "only one of these running at a time". Two servers could run the same report job at once. The framework already had a cross-server lease lock with owner fencing, but only inside the migrator.

Change

wheels.LeaseLock

The migrator's lease, extracted. It is one table with one row per held lock: lockname (primary key), lockowner, lockhost, acquiredat and expiresat. Times are epoch milliseconds, so every database compares them as plain numbers.

• Statement builders. Owner, read, insert, expired-lease takeover, renew and release, plus the table DDL.
• Execution methods. tryAcquire, renew, release, ownerOf, read, run against a datasource.
• Owner fencing. Every change after the insert is fenced on the owner. A holder whose lease was taken over can neither renew nor release the new holder's lock.

The migrator is unchanged

The migrator keeps its own lock table and every one of its lock methods. Their bodies now build their SQL with LeaseLock and still run it through $migrationLockQuery(), which carries the migrator's datasource credentials and its test seams. The statements, parameters and their order are the same as before, and its clock (GetTickCount()) is unchanged.

Exclusive jobs

component extends="wheels.Job" {
    function config() {
        super.config();
        this.exclusive = true;          // one run of this class at a time, on any server
    }
}

component extends="wheels.Job" {
    // or a key, from the job's data: one run per account
    public string function concurrencyKeyFor(struct data = {}) {
        return "account-" & arguments.data.accountId;
    }
}

Lease names.

• this.exclusive = true leases job:<jobClass>.
• A concurrency key leases key:<key>, and a key implies exclusive.
• The key comes from this.concurrencyKey = "...", or from a concurrencyKeyFor(struct data) method. The method wins, and both can be set.
• A name longer than the table's 100 characters is hashed.

Lifetime.

• When it's taken. The lease is taken after the claim and before perform(), on both the worker path and processQueue().
• Owner. The owner is the attempt's claim token.
• Length. It lasts the claim timeout plus Max(60, claimTimeout): the same window after which the row itself can be reaped, so it never ends inside the claim timeout.
• Release. It is released afterwards (owner-fenced), on success or failure.
• After a timeout. It is not released then: perform() may still be running in its thread, so the lease is left to expire.

Busy lease. When another run holds the lease, perform() doesn't run:

• the job goes back to pending with runAt = now + Min(remaining lease, 30 s);
• the attempt its claim counted is given back, so waiting on a busy key never uses up retries;
• it counts as skipped, not failed: processNext() returns skipped and deferred, and processQueue() counts it in skipped.

Lost lease. If the lease was taken over by the time the job finishes, a warning is logged and the job is still recorded as completed, because it already ran.

• processNext() returns leaseLost.
• The worker counts lost leases in leasesLost.
• processQueue() returns a leasesLost count.

Table. wheels_job_locks is created on first use by an exclusive job. Whether it exists is asked of the database catalog, never by a failing probe query (which would abort an enclosing PostgreSQL transaction). That is never inside a Wheels transaction, where DDL would commit the open transaction on MySQL and Oracle; there the job fails with the reason and the CREATE TABLE, and is retried.

The guarantee and its limit (for the guide)

Exclusivity is only as strong as the lease.

• Without renewal. The lease covers the claim timeout plus a grace period. A run that outlives it can overlap a second run. Examples: a run stuck in a blocked JDBC call that a timeout can't interrupt, or one that doesn't cooperate with renewal. That overlap is logged as a lost lease.
• Renewal. heartbeat() (feat(job): heartbeat() + heartbeat-aware reaping; idempotent gating of retries after a reap #4477, now merged) renews the lease: each heartbeat extends it by the lease's window from now (claim timeout + Max(60, claimTimeout)). A run that keeps heartbeating keeps its lease. fix(job): let heartbeats shorten dead-worker detection for long jobs #4502 will move this to the heartbeat grace window.
• Clocks. Lease times come from each app server's clock (System.currentTimeMillis()), so servers sharing the table need synchronised clocks (NTP). Skew between two servers shortens or lengthens a lease by that much.

Out of scope

• Moving the uniqueKey upgrade's lock from the migrator lock to LeaseLock.
• A CLI to list or clear job leases.
• Exclusivity at enqueue time. That is uniqueKey.

Tests

JobLeaseSpec (12 specs).

LeaseLock:

• one owner takes a free lease, and no other until it is released;
• an expired lease is taken over, and the old owner can neither renew nor release it;
• renewal extends it, and an epoch-millisecond value round-trips;
• the migrator's statements target its own table.

Exclusive jobs:

• the lease is held during perform() and released after it;
• a lease held by another server stops the run, and the job waits as pending with its attempt given back;
• a permanently busy key never uses up retries (4 polls, maxRetries = 1);
• processQueue() counts a busy lease as skipped;
• an expired lease is taken over and the job runs;
• a lease taken over during perform() still completes the job and reports leaseLost;
• lease names follow the order concurrencyKeyFor() → this.concurrencyKey → exclusive (with both set, the method wins), and a long name is hashed.

On develop all 12 error, because none of this API exists yet. Behaviourally, develop has no lease, so an exclusive job runs even while another server holds one.

The migrator's own lock specs (MigrationLockSpec, the whole migrator directory) are the byte-for-byte guard. They run unchanged and stay green on every database.

Local results:

• Full core suite, Lucee 7 + SQLite: 7152 pass, 0 fail, 0 error (582 bundles).
• Jobs and migrator specs, Lucee 7: green on SQLite, H2, MySQL, PostgreSQL, CockroachDB, SQL Server and Oracle; PostgreSQL and CockroachDB also green on a second run against the same database.
• Jobs and migrator specs, BoxLang: green on SQLite, PostgreSQL, CockroachDB and SQL Server.
• Jobs and migrator specs, Adobe 2023: green on SQLite and MySQL.
• Jobs and migrator specs, Adobe 2025: green on SQLite and PostgreSQL.
• Complexity gate: passes.

Merge notes

• processNext() overlap. feat(job): per-host concurrency cap, drain/resume and JobRunner.tick() #4491 splits processNext() and fix(job): run processQueue() through the job worker #4495 pre-resolves the claim timeout there. This PR touches the same spot (the busy/lost-lease handling after $executeJob()), so whichever lands second carries the others.
• fix(job): run processQueue() through the job worker #4495's processQueue() loop. It must count a deferred outcome as skipped, not failed.
• Heartbeat. As above, whichever of feat(job): heartbeat() + heartbeat-aware reaping; idempotent gating of retries after a reap #4477's PR and this one merges second wires heartbeat() to renew the lease.

Docs

The guide section (exclusive jobs, concurrency keys, the lease-loss caveat and the clock requirement) goes into the 4.2 guide (#4485). Changelog fragment included.

🤖 Generated with Claude Code

Develop merge (4.2.0)

Merged develop (#4490, #4491, #4492, #4493, #4494, #4495), with a merge commit. As with the others, Job.cfc and JobWorker.cfc start from develop and this PR's own changes go on top. Six hunks needed placing by hand:

• Busy lease and lost lease. These now live in $runClaimedJob(), the half of feat(job): per-host concurrency cap, drain/resume and JobRunner.tick() #4491's processNext() split that runs a claimed job. A busy lease defers the job (skipped + deferred) and returns.
• this.exclusive. It sits next to feat(job): heartbeat() and idempotency-aware reaping #4490's this.idempotent.
• processQueue(). It now runs through the worker (fix(job): run processQueue() through the job worker #4495). It counts a lease-deferred job as skipped again and reports leasesLost. This reverses fix(job): run processQueue() through the job worker #4495's upgrade note that result.skipped is always 0, which the 4.2 guide should state.

A separate commit wires heartbeat() to renew the lease, with a spec showing the lease's expiry moving forward after a heartbeat.

Local results:

• Full core suite, Lucee 7 + SQLite: 7210 pass, 0 fail, 0 error (589 bundles).
• Jobs and migrator specs, Lucee 7: green on all seven databases. The jobs specs are also green on a second PostgreSQL/CockroachDB run against the same database.
• Jobs specs, BoxLang: green on SQLite, PostgreSQL and CockroachDB.
• Jobs specs, Adobe 2023: green on SQLite and MySQL.
• Jobs specs, Adobe 2025: green on SQLite and PostgreSQL.
• Complexity gate: passes.

[github-actions]

Wheels Test Results

220 680 tests   215 787 ✅  1h 39m 54s ⏱️
 17 670 suites    4 893 💤
     30 files          0 ❌

Results for commit 37fb206.

♻️ This comment has been updated with latest results.

[bpamiri]

Review: draft, head 8258542. REQUEST CHANGES: one MUST-FIX, in a spec file only. The lease design and the Migrator extraction are sound. Not for merge before the 4.2.0 cut anyway.

MUST-FIX

1. JobLeaseSpec doesn't parse on RustCFML, which zeroes the whole RustCFML core suite. Both RustCFML (Linux) in PR CI and the rustcfml matrix leg fail with:

Parse error in 'vendor/wheels/tests/specs/jobs/JobLeaseSpec.cfc' [line 34, col 9]: Expected LBrace, found Dot

• The spec names a variable lock (var lock = $lock();). Two statements then begin with it (lock.tryAcquire(...) at lines 34 and 46).
• RustCFML reads a statement-leading lock as the script lock { } keyword. The uses inside expect(...) parse because they're in expression position.
• Lucee, Adobe and BoxLang accept it, so only that lane shows it.
• Rename the variable (e.g. leaseLock) throughout the spec. The framework code itself only uses local.lock, which is fine.

LOW

• "The method wins" doesn't hold when a job sets both (Job.cfc $jobLeaseName).
  • Methods and this. properties share the this scope. A job that defines concurrencyKey(data) and sets this.concurrencyKey = "..." in config() has its method overwritten by the string.
  • So IsSimpleValue() is true and the static key is used. A BoxLang probe confirmed it (StructKeyExists true, IsSimpleValue true), and CFML scoping works the same way on Lucee and Adobe.
  • The spec covers each form alone, never both.
  • Either document "set one, not both", or give the method a name that can't collide (e.g. concurrencyKeyFor(data)).
• Table probe inside a transaction ($ensureJobLockTable). $jobLockTableExists() runs a SELECT that fails when the table is missing. Inside a PostgreSQL transaction that failure aborts the transaction before the helpful Wheels.JobLockTableMissing is thrown. Probing via $dbinfo or checking the transaction first would keep the transaction alive. Jobs rarely run inside one, so this is minor.

Verified

• Migrator unchanged. Every lock statement now comes from LeaseLock with identical SQL text:

  • release, renew, insert, takeover, owner, read, the DDL, and the expired-only clear (AND expiresat < :now appended only without force);
  • the same parameter names and cf_sql_varchar/cf_sql_bigint types;
  • still run through $migrationLockQuery(), so the migrator's credentials and the Adobe 2023 null-DML handling are kept.

  MigrationLockSpec is 24/24 on all 32 legs.

• The lease can't leak.

  • $runPerformWithTimeout() catches every perform() error and returns an outcome, so the release always runs.
  • It's skipped only after a timeout, on purpose: the thread may still be running, and the lease is left to expire.
  • An acquire failure never holds a lease.

• The attempt is given back: attempts = attempts - 1 … WHERE status='processing' AND attempts > 0 plus the claim-token guard. Specced for four busy polls with maxRetries = 1, ending with attempts at 0 and the job pending.

• Lease vs claim window. The lease runs claim + Max(60, claim) from acquire, matching the reaper's per-row grace. So a reaped re-run finds the lease busy and waits instead of overlapping the still-running attempt. A lost lease is logged and counted, and doesn't fail the job.

• tryAcquire. It inserts only when no row exists (so PostgreSQL rarely hits the duplicate-key abort) and rethrows a real INSERT failure. Takeover is owner-fenced with a read-back.

• DDL guarded. wheels_job_locks DDL is refused inside a Wheels transaction with a clear message.

• Cross-engine invariants.

  • The catch in $releaseJobLease writes through var state (invariant 11).
  • No loops in finally (invariant 12).
  • LeaseLock is standalone, not a mixin.
  • Row counts come from the result option, the same way the job fencing code already reads them.

• Clock. Job leases use System.currentTimeMillis().

Compat matrix 37724822970: 32/32 Lucee/Adobe/BoxLang legs green, 0 fail / 0 error, on every database (Oracle, CockroachDB and H2 included). JobLeaseSpec is 12/12 and MigrationLockSpec 24/24 on every leg. The rustcfml leg fails (MUST-FIX 1).

PR CI: 18 pass, 1 fail (RustCFML (Linux), MUST-FIX 1), 2 skipped.

[bpamiri]

Final review: head e2bf678. APPROVE (ready), with one pre-merge check below. Still a draft for v4.3.0; it stays held until after the 4.2.0 cut.

Earlier findings, all resolved

• MUST-FIX, RustCFML parse error. The variable is renamed leaseLock in JobLeaseSpec, and local.lock/arguments.lock in Job.cfc too, so no statement starts with lock. any more. The RustCFML core suite now runs: 7129 pass, 0 fail, 0 error.
• LOW, method vs property. The method is now concurrencyKeyFor(data), so a this.concurrencyKey can no longer overwrite it. $jobLeaseName checks the method first (!IsSimpleValue), then the simple this.concurrencyKey. KeyedProbeJob now sets both, so the precedence spec exercises the real collision case.
• LOW, table probe in a transaction. $jobLockTableExists() now asks each database's catalog, keyed on $detectDatabaseType(). The names match its return values (postgresql, mysql, sqlserver, oracle, h2, sqlite), and CockroachDB maps to postgresql, where current_schema() is valid. A catalog error falls back to the probe SELECT, which now runs only outside a Wheels transaction.

INFO

• H2's catalog query has no schema filter, so a wheels_job_locks in another schema would count as present.
• Unknown database inside a transaction. On a database outside the six, inside a Wheels transaction, the table is reported missing even when it exists. That fails closed with Wheels.JobLockTableMissing, which is safe.
• Lucee 7 + SQLite (LuCLI) and Tutorial E2E were skipped on this push (path-filtered). The LuCLI lane passed at 8258542, and the delta touches no CLI code.

Pre-merge check (narrowing gap)

The narrowed run covers sqlite, mysql, postgres and sqlserver. The new catalog branches for Oracle (user_tables), H2 (INFORMATION_SCHEMA.TABLES) and CockroachDB (the postgresql branch) are therefore read but not yet exercised. They look right, but a wrong-yet-valid query (0 rows) would make every exclusive job on that database try CREATE TABLE, fail "already exists", and rethrow. Before merge, run at least one engine on oracle, h2 and cockroachdb (e.g. lucee7) against this head. That's a test-only run, not a code change.

Evidence

• Narrowed matrix 37727087190 at e2bf678 (5 JVM engines × sqlite/mysql/postgres/sqlserver plus RustCFML):
  • 20/20 legs green, 0 fail / 0 error, with JobLeaseSpec 12/12 and MigrationLockSpec 24/24 on every leg;
  • RustCFML 7129/0/0 (the earlier head failed to parse).
• Full matrix 37724822970 at 8258542: 32/32 JVM legs green on every database. The delta since then is the rename, the method name, and the catalog check (above).
• PR CI at e2bf678: 17 pass, 0 fail, 4 skipped (path-filtered and reviewer jobs).

[bpamiri]

Pre-merge check, head e2bf678: green. Run 37728725254 (lucee7 + boxlang × oracle / h2 / cockroachdb; BoxLang has no H2 leg) exercises the catalog branches the narrowed run didn't reach:

leg	pass	fail	err	JobLeaseSpec	MigrationLockSpec
lucee7-oracle	7124	0	0	12/12	24/24
lucee7-h2	7117	0	0	12/12	24/24
lucee7-cockroachdb	7042	0	0	12/12	24/24
boxlang-oracle	7124	0	0	12/12	24/24
boxlang-cockroachdb	7042	0	0	12/12	24/24

The lease specs build wheels_job_locks and re-check it before every case. A catalog query that missed an existing table would have sent them into CREATE TABLE → "already exists" → rethrow, and errored them. They pass, so the Oracle (user_tables), H2 (INFORMATION_SCHEMA.TABLES) and CockroachDB (postgresql branch) queries find the table. The approval stands, with no further pre-merge conditions from review.

[bpamiri]

Review at 37fb206: develop merge (delta) + heartbeat lease renewal (new code). APPROVE (ready to merge).

(1) Develop merge f52ea78, delta against the approved e2bf678

• Files identical to the approved head. LeaseLock.cfc, the Migrator.cfc change and JobLeaseSpec.cfc produce byte-identical change sets against their bases. So the Migrator parity holds and MigrationLockSpec is unchanged.
• Logic moved into feat(job): per-host concurrency cap, drain/resume and JobRunner.tick() #4491's split, behaviour unchanged.
  • The busy-lease defer and the lost-lease counting now live in JobWorker.$runClaimedJob(): busy → $deferJobForLease() (attempt given back, claim-token fenced) → skipped + deferred. leaseLost → this.leasesLost++, counted whatever the outcome.
  • $executeJob() passes the worker cap as claimTimeout (so the lease window is still claim + Max(60, claim)), and windowSeconds is now carried on the lease struct.
  • The busy path clears the tenant context before returning.
• processQueue() (now worker-based, fix(job): run processQueue() through the job worker #4495) counts deferred as skipped and tallies leasesLost separately. Deferred jobs don't count toward limit, and the defer pushes runAt ≥ 1 s, so the loop is bounded (rev1-r3 checked this too).
  • LOW: a single processQueue(limit = n) call can claim and defer many exclusive jobs whose lease is held elsewhere, since deferred jobs don't count toward n. It's bounded by the number of due rows. Fine as is; worth a sentence in the guide fold.
• this.exclusive sits beside this.idempotent, and the two don't interact.

(2) 37fb206, heartbeat() renews the lease (reviewed as code)

• Fenced renewal. $renewJobLease() calls LeaseLock.renew(), which is UPDATE … SET expiresat WHERE lockname AND lockowner = :owner. A run can only extend its own lease, never a successor's.
• Ordering. A fenced heartbeat (claim reaped) throws Wheels.Job.Fenced before renewing, so a stale attempt never extends anything.
• Renewal failure.
  • If the lease was already taken over, renew() returns false. That's logged once (lostLogged) and left alone; the run's end then fails release() and reports leaseLost, as before.
  • A query error is caught and logged, and heartbeat() doesn't throw for it, so a lease-table hiccup can't fail the job.
• Expiry vs cadence. Each renewal sets now + windowSeconds (the original claim + Max(60, claim)). With perform() hard-capped at the attempt's timeout (≤ claim), a heartbeating run never sees its lease expire. Without heartbeats, behaviour is unchanged.
• Scope.
  • $setLeaseContext() runs only when the lease is held, on the per-run instance ($instantiateJobClass creates a fresh object each run), so no lease context leaks between runs or into non-exclusive jobs.
  • Outside a worker, heartbeat() still returns early.
• Cross-engine. The catch only logs and returns. leaseState is a var struct written in the try (invariant 11 is fine). No loops in finally.
• Spec. HeartbeatLeaseJob records expiresat before and after heartbeat() (with a 50 ms sleep) and asserts it moved forward and was released afterwards: 13/13 JobLeaseSpec on every leg.

Evidence

• Matrix 37808590705 at 37fb206 (5 JVM engines × sqlite/mysql/postgres/sqlserver/oracle/cockroachdb):
  • 30/30 legs green, 0 fail / 0 error, 589 bundles;
  • JobLeaseSpec 13/13 and MigrationLockSpec 24/24 on every leg;
  • RustCFML 7185/0/0.
• H2 isn't in this run. LeaseLock and the H2 catalog branch are byte-identical to e2bf678, which passed lucee7-h2 in pre-merge run 37728725254.
• PR CI at 37fb206: 17 pass, 0 fail, 4 skipped (reviewer jobs and the path-filtered LuCLI / Tutorial lanes). Merge state is clean.

Docs (INFO, not blocking): rev1-r3's guide points (the new meaning of skipped, leasesLost, the exclusive-jobs section) are queued for the #4501 fold with dev1-design.