Skip to content

docs(web): add the 4.2 "Running Jobs on Multiple Servers" guide - #4501

Merged
bpamiri merged 34 commits into
developfrom
peter/4485-multi-server-jobs-guide
Oct 9, 2026
Merged

bpamiri merged 34 commits into
developfrom
peter/4485-multi-server-jobs-guide

Conversation

@bpamiri

@bpamiri bpamiri commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

What

Adds a new 4.2 Digging Deeper guide, Running Jobs on Multiple Servers, as a companion to Background Jobs for when several app servers process one wheels_jobs table.

It documents the multi-server job features already on develop and clearly marks the parts still to come.

Covered (merged on develop)

  • The guarantees — one recorded outcome per attempt (per-claim fencing), at-least-once recovery, and why perform() must be idempotent.
  • heartbeat() for long-running jobs, and what Wheels.Job.Fenced means.
  • this.idempotent and the interrupted status (at-most-once recovery for work that must not run twice).
  • set(jobsMaxConcurrentPerHost = n) and the jobsHostName override.
  • Runner options — the CLI worker, a scheduled processQueue(), and JobRunner.tick().
  • Draining a server before a deploy — wheels jobs drain --wait / wheels jobs resume, the host registry, jobsCodeVersion.
  • The shared schema to apply by hand on a live multi-server database (rather than let a live server ALTER the table), with the column list and SQL.
  • Clocks and skew — keeping app-server clocks in sync.

Marked as not-yet-built (each links its tracking issue)

The token-gated /wheels/jobs/tick route, a single UTC database clock, recurring schedules, exclusive jobs and concurrency keys, lifecycle hooks, and wheels jobs install.

Also

  • Links both ways with the Background Jobs guide, and adds the sidebar entry and the Digging Deeper index card.
  • Records the heartbeat/idempotency, per-host cap, and wheels jobs drain/resume changes on the 4.1 → 4.2 upgrade guide (next to the existing claimTimeout, fencing, and uniqueKey entries).

Notes

🤖 Generated with Claude Code

A companion to Background Jobs for when several app servers process one
wheels_jobs table. It documents the multi-server features already on
develop and marks the parts still to come.

Covered (merged):
- the guarantees: one recorded outcome per attempt (per-claim fencing),
  at-least-once for idempotent jobs, and why perform() must be idempotent
- heartbeat() for long jobs, and Wheels.Job.Fenced
- this.idempotent and the interrupted status (at-most-once)
- set(jobsMaxConcurrentPerHost) and the host name override
- runner options: the CLI worker, a scheduled processQueue(), JobRunner.tick()
- draining a server before a deploy with wheels jobs drain --wait / resume
- the shared schema to apply by hand on a live multi-server database
- clock skew and NTP

Marked as not-yet-built, each with its tracking issue: the token-gated
/wheels/jobs/tick route, a single UTC database clock, recurring
schedules, exclusive jobs and concurrency keys, lifecycle hooks, and
wheels jobs install.

Links both ways with Background Jobs, adds the sidebar and index entries,
and records the heartbeat/idempotency, per-host cap, and jobs drain/resume
changes on the 4.1 to 4.2 upgrade guide.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Accuracy review: head b97fbfd, against develop at 631f1a8 (#4487–#4492 merged). CHANGES NEEDED: two HIGH, one MEDIUM, three LOW. The placeholders for #4482, #4486 and #4499 and the "Coming in 4.2" list are fine as interim text.

HIGH

1. heartbeat() does not let a job run past its timeout. The heartbeat section says "A job that legitimately runs longer than its timeout would be reaped and run a second time. To prevent that, call heartbeat()". The upgrade entry says "so a job that legitimately runs past its timeout isn't reaped and run again".

  • Hard cap. As merged, perform() is hard-capped. JobWorker.$executeJob() runs it through $runPerformWithTimeout() with Min(job timeout, worker timeout), which joins the thread with that timeout, terminates it and records Wheels.JobTimeout (retry). Heartbeats don't extend it.
  • The reaper can't catch a live attempt anyway. It waits claimTimeout + Max(60, claimTimeout) since the last heartbeat or claim, at least twice that hard cap, so it never reaches an attempt that's still running normally.
  • The example would fail. With this.timeout = 120 and a heartbeat per batch, a reindex longer than 120 s is killed at 120 s and retried, not kept alive.
  • What heartbeat() does do today:
    • renews the row (heartbeatAt, else updatedAt) for this claim only;
    • throws Wheels.Job.Fenced once the claim was reaped and reissued. That makes it a cheap "do I still own this job?" check between units of work.
  • Needs a decision before the text is fixed.
    • If feat(job): heartbeat() and idempotency-aware reaping #4490's intent was that heartbeats extend a running job's deadline, that's a code follow-up.
    • If not, the guide (and the heartbeat docblock in Job.cfc, which makes the same claim) should say:
      • set this.timeout to cover the whole job;
      • use heartbeat() to keep the row fresh and to stop early when fenced.

2. processQueue() honours neither the per-host cap nor drain, and doesn't reap.

  • The cap claim is false. "The cap is enforced for every runner on the host — the CLI worker, processQueue(), and JobRunner.tick()" doesn't hold. Job.processQueue() on develop selects pending rows and runs them. It has no cap or $hostDraining() check and no checkTimeouts() (the cap and drain live in JobWorker.processNext(), which tick() and the CLI worker use). That's what the page's own "Coming in 4.2" item (fix(job): run processQueue() through the job worker #4495) is for.
  • Side effects elsewhere:
    • the "Running the queue" bullet recommends a scheduled processQueue() without saying so;
    • "Drain a server" implies a drained server starts nothing, which is false if it runs processQueue().
  • Fix:

MEDIUM

3. The hand-written DDL uses heartbeatAt DATETIME, which fails on PostgreSQL and CockroachDB. The column table also lists DATETIME.

  • PostgreSQL and CockroachDB have no DATETIME type. The framework uses TIMESTAMP there and on H2 and Oracle ($heartbeatAlterSql()).
  • The adjustment note only mentions COLUMN and VARCHAR2/ADD (...). This section explicitly targets PostgreSQL and CockroachDB, so readers will run it as written.
  • Fix: say TIMESTAMP on PostgreSQL, CockroachDB, H2 and Oracle, and that Oracle's claimTimeout is NUMBER(10) (as in background-jobs).

LOW

  • 4. Interrupted rows can be purged. "clear interrupted ones with your own SQL" isn't needed: new wheels.JobWorker().purge(status = "interrupted", days = n) exists and works. $markInterrupted sets failedAt, which purge() ages on. Its docblock still lists only completed/failed; that's a code nit.
  • 5. Clocks. "Every guarantee here is timed against the database's clock where it can be" isn't true on develop. Every job timestamp (claim, heartbeat, reap cutoff, runAt, drain expiry) comes from each app server's Now() via $now(). Nothing reads the database clock yet. Suggest: "…is timed by each app server's own clock".
  • 6. Upgrading section. "the schema is added for you unless you turn it off" — there's no way to turn it off until feat(cli): wheels jobs install and jobsAutoCreateTables #4499 lands. Drop the clause, or mark it planned.

Verified accurate

  • Claims and fencing:
    • the claim's guarded UPDATE … AND status = 'pending';
    • the reap grace built from the claiming worker's recorded timeout (timeout + max(60, timeout));
    • fenced outcomes logged with the Wheels.Job.Fenced: prefix;
    • "one recorded outcome per attempt, not one execution";
    • fencing holding only once no 4.1 worker remains, and tokenless rows already processing being reaped as before.
  • heartbeat() mechanics: it renews heartbeatAt (else updatedAt) under the claim-token guard; the reaper reads COALESCE(heartbeatAt, updatedAt); it throws Wheels.Job.Fenced when the claim is gone and does nothing outside a worker. Also verified: model() in jobs, and findInBatches(batchSize, callback).
  • Idempotency and stats:
    • this.idempotent defaults to true; false → interrupted with no retry;
    • queueStats() returns {pending, processing, completed, failed, interrupted, total}, and getStats() counts interrupted;
    • purgeCompleted() deletes only completed rows.
  • Settings:
    • jobsMaxConcurrentPerHost (default 0 = no cap);
    • jobsHostName (≤128 characters, overriding InetAddress host name);
    • jobsCodeVersion (default: the Wheels version).
  • JobRunner:
    • tick(queues = "", maxConcurrent = -1, timeout = 300, maxJobs = 1) refreshes the registry, reaps, then runs jobs serially and stops on drain, cap or an empty queue;
    • drain(expiresInSeconds = 3600), resume() and status();
    • the wheels_job_hosts columns, and auto-create never inside a transaction.
  • CLI:
    • drain --wait (bare = 600 s, --wait=N, non-zero exit via Wheels.JobsDrainTimeout);
    • --expires (default 3600, 0 = until resumed);
    • resume;
    • status showing running count, cap and drain;
    • wheels jobs work exiting non-zero on a lost server connection.
  • Upgrade entries for fix(job): fence job completion, retry and failure with a per-claim token #4487/feat(job): de-duplicate enqueue with uniqueKey #4488: claimToken/claimedBy added on first use, the uniqueKey backfill and index built under the migration lock, and enqueueIn/enqueueAt accepting uniqueKey.
  • The text is neutrally worded.

bpamiri and others added 3 commits October 8, 2026 08:34
Fixes from rev1-r1's pass on the new guide and the 4.1 to 4.2 upgrade
entry:

- heartbeat() does not extend a job's timeout. The timeout is a hard cap
  per attempt (perform() runs in a bounded thread and is stopped when it
  elapses); heartbeat() only records liveness. Reframed the section and
  the upgrade entry, and noted the lease-renewal (#4496) and
  dead-worker-detection (#4502) work that builds on it.
- The heartbeatAt column is TIMESTAMP on PostgreSQL, CockroachDB, Oracle
  and H2 (they have no DATETIME); the manual DDL now gives per-database
  types.
- Documented purge(status="interrupted") for pruning interrupted rows,
  instead of "use your own SQL".
- Job timestamps are written with each server's application clock
  (Now()), not the database clock; corrected the clocks section.
- Dropped "unless you turn it off" from the upgrade note: turning the
  auto-schema off needs jobsAutoCreateTables, which isn't built yet.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
…e guide

processQueue() now runs through the job worker (#4495), so it reaps
stale jobs, honours the per-host cap, and stops starting new jobs on a
draining host. Document that as fact instead of a forward reference,
note that it still doesn't refresh the host registry's lastSeenAt the
way tick() does (its running count and drain state are read live), and
drop #4495 from the "Coming in 4.2" list now that it has landed.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Accuracy re-check: head ee1c853, against develop at 47ce1a8 (#4495 now merged). All six earlier findings are resolved. One small new inaccuracy, in text added by the fix.

Resolved

  • HIGH 1 (heartbeat()). The guide and the upgrade entry now say timeout is a hard cap per attempt and that heartbeat() records liveness without extending it. That matches $runPerformWithTimeout(). The heartbeat docblock in Job.cfc still makes the old claim; fine to leave it for fix(job): let heartbeats shorten dead-worker detection for long jobs #4502 or a follow-up, as you say.

  • HIGH 2 (processQueue()). Correct as fact now. On develop, processQueue() loops JobWorker.processNext():

    • it reaps first through checkTimeouts();
    • it honours the cap via $jobsMaxConcurrentPerHost(), because maxConcurrentPerHost defaults to -1 and $runningOnHost() gives the live count;
    • it stops on $hostDraining().

    The caveat that it doesn't refresh lastSeenAt/codeVersion (only tick() does) is accurate and worth keeping. fix(job): run processQueue() through the job worker #4495 is correctly removed from "Coming in 4.2".

  • MEDIUM. heartbeatAt is now shown as TIMESTAMP on PostgreSQL, CockroachDB, Oracle and H2, and DATETIME elsewhere, matching $heartbeatAlterSql().

  • LOWs.

    • purge(status = "interrupted", days = n) is documented. purge() takes completed, failed or interrupted, and ages interrupted rows by failedAt, which $markInterrupted sets.
    • The clock text now says each server's Now().
    • "unless you turn it off" is gone.

New LOW

Everything else I checked last round still holds on 47ce1a8. The PR diff against develop is docs-only (5 files). Neutral wording throughout.

#4496 as approved sets a job's lease expiry at acquire and does not
renew it from heartbeat() yet; the renewal is wired in #4496's develop
merge. Word the liveness aside as "once #4496 lands, heartbeat() also
renews an exclusive job's lease" so it's accurate until #4496 merges,
then it becomes plain fact.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Re-check at e12ffec: the liveness aside now says lease renewal arrives with #4496 ("once #4496 lands, heartbeat() also renews …"). That's accurate as forward-looking text, and I'll verify the wiring itself in #4496's develop-merge review. With that change I have no open accuracy findings on this PR.

bpamiri and others added 2 commits October 8, 2026 09:01
Fold the merged recurring-schedules feature into the multi-server jobs
guide as a full section, replacing the placeholder:

- The config/schedules.cfm DSL: schedule().job().cron() / .every(),
  .timezone(), .catchUp() / .catchUpWindow(), .queue()/.priority()/
  .data()/.enabled(). wheels new doesn't create the file.
- Running schedules with new wheels.JobScheduler().enqueueDue(): each
  slot carries a uniqueKey, so calling it on every server yields one job
  per slot.
- Catch-up ("latest" default within catchUpWindow, "none"), the
  wheels_job_schedules table, code-vs-db schedule sources, and the
  java.time/DST notes for non-UTC time zones.

Also correct the Background Jobs page, which said Wheels has no
recurring schedule, to point here, and add a 4.1 to 4.2 upgrade entry.
Remove recurring schedules from the "Coming in 4.2" list.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Re-check at e9eea3e ("Recurring schedules", against #4493 merged as d1b8df2): accurate, no findings.

Verified against JobSchedule.cfc, JobScheduler.cfc and JobCron.cfc on develop:

  • DSL:
    • .job/.cron/.timezone/.catchUp/.catchUpWindow/.queue/.priority/.data/.enabled;
    • every(count, "minutes" | "hours"), UTC/epoch-aligned, with anything else refused and pointing at cron();
    • cron with lists/ranges/steps/names and @hourly…@yearly (plus @midnight/@annually); seconds and Quartz refused.
  • Catch-up: catchUp is "latest" (default) or "none"; catchUpWindow defaults to 3600 s (the code also enforces a 60 s minimum, which the page doesn't need to mention).
  • Running and dedupe:
    • enqueueDue() must be called by the app (not wired into tick()) and returns {checked, enqueued, duplicates, skipped, errors};
    • the slot uniqueKey "<name>:<UTC ISO>" makes it safe on every server;
    • a slot at or before the last enqueued one never reruns, and a new schedule starts from now.
  • Storage and sync:
    • wheels_job_schedules is auto-created outside transactions;
    • the config/schedules.cfm sync at start-up treats the file as code schedules, db rows win on a clash, and removed code schedules are disabled rather than deleted.
  • Time zones: non-UTC on an engine without (or with only partial) java.time → Wheels.Job.InvalidSchedule. DST gap → next valid instant; overlap → first occurrence.
  • Other pages: the background-jobs correction and the upgrade entry (feat(job): recurring schedules (interval + cron, timezone/DST, catch-up) #4481) match. The schedules bullet is gone from "Coming in 4.2".

bpamiri and others added 3 commits October 8, 2026 09:05
enqueueDue() is called manually today. Note the planned schedule-runner
wiring: JobRunner.tick() and the CLI worker's poll will call it every
jobsScheduleCheckSeconds (default 30), so a running worker fires
schedules without a separate call. Clearly marked as not built yet.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
Fold in the improved SQL Server guidance: the filtered unique index
needs compatibility level 100 or higher. Below that, either raise the
level or use a plain unique index, but only once every server runs 4.2
(a plain index allows one NULL, so a 4.1 server writing NULL keys would
collide). The framework's warning and Wheels.Job.UniqueKeyUnavailable
now name the failed step and the per-database fix. Added to the guide's
schema section and the 4.1 to 4.2 upgrade entry; removed from
"Coming in 4.2".

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Re-check at 86790fe (#4494 folded in): accurate, no findings. Checked against $uniqueKeyUpgradeFailureText(), $uniqueKeyManualFix() and the Wheels.Job.UniqueKeyUnavailable text on develop (3a76104):

  • the SQL Server filtered index needs compatibility level ≥ 100;
  • below that the options are to raise the level (the framework then retries) or, once every server runs 4.2, use a plain unique index, because a plain index allows only one NULL and 4.1 servers write NULL keys;
  • the warning and the error name the failed step (column, backfill or index) and the fix for that database and level.

The 4.1→4.2 entry is consistent with this, and #4494 has been removed from "Coming in 4.2".

Since #4495 routed processQueue() through the worker, a row another
poller claims first is no longer counted as skipped — the atomic claim
just fails and the loser moves on. Fix the stale "shows up in the
skipped count" parenthetical, and add the fenced count (a current
field) to the result-shape comment, with a pointer to the fencing
guarantee. (leasesLost and the redefined skipped come with #4496.)

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
bpamiri and others added 2 commits October 8, 2026 11:02
Fold in the merged lease-lock feature:

- New "Exclusive jobs and concurrency keys" section: this.exclusive,
  this.concurrencyKey, and concurrencyKeyFor(data) (the method wins),
  the wheels_job_locks lease, a busy lease putting the job back to
  pending (counted as skipped), leasesLost, and the never-making-
  progress caution.
- heartbeat() now renews an exclusive job's lease: flipped the
  forward-looking aside to plain fact.
- Background Jobs: processQueue()'s result gains leasesLost, and its
  skipped/fenced/leasesLost counts are explained with links.
- Added a 4.1 to 4.2 upgrade entry; noted wheels_job_locks and
  wheels_job_schedules as cheap auto-create tables; removed exclusive
  jobs from "Coming in 4.2".

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Re-check at da38a4d (#4496 folded in, against develop e63973b): accurate, one LOW nit.

Verified

  • Exclusive options and keys: this.exclusive defaults to false. this.concurrencyKey / concurrencyKeyFor(struct data) work as described, and the method wins ($jobLeaseName checks it first).
  • Busy lease: the job goes back to pending with the attempt given back, and counts as skipped.
  • Lease ownership: the lease is owned by the claim token. wheels_job_locks is auto-created and refused inside a transaction (Wheels.JobLockTableMissing).
  • heartbeat() renewal: it renews an exclusive run's lease (owner-fenced). A lost lease is logged, counted in leasesLost, and doesn't fail the run.
  • processQueue() result: {processed, failed, skipped, fenced, leasesLost, errors}.
  • Other pages: the upgrade entry and the auxiliary-tables note match, and feat(job): exclusive jobs and concurrency keys on a shared lease lock #4496 is gone from "Coming in 4.2".

LOW

  • Lease window wording. "lasts as long as the claim's window (the job's timeout plus the reap grace)" isn't quite right. The window is claimTimeout + Max(60, claimTimeout), where claimTimeout is the claiming worker's timeout (JobWorker.$executeJob passes workerCap, e.g. wheels jobs work --timeout), not the job's own this.timeout. Under processQueue() (per-job timeouts since fix(job): run processQueue() through the job worker #4495) the two coincide. Suggest: "…the claiming worker's timeout plus the reap grace, timeout + max(60s, timeout)".

INFO

  • "so a long exclusive run keeps its slot instead of having the lease expire out from under it": today perform() is hard-capped at the attempt's timeout (≤ the claim timeout) and the lease runs for at least twice that, so a lease can't actually expire mid-run. The renewal is a safety margin rather than something a run currently depends on. The text isn't wrong, just stronger than needed; fine to leave.

…ding

Per rev1-r1: the exclusive-job lease window is sized from the timeout
the job was CLAIMED with (the claimTimeout passed to $acquireJobLease),
not this.timeout — they coincide only under processQueue. Reword to
"the timeout it was claimed with plus the reap grace", matching the
reaper language. And because that window outlasts the hard per-attempt
timeout cap, the lease can't actually expire mid-run today, so drop the
"expire out from under it" framing: heartbeat lease renewal is
groundwork, like its effect on the reaper.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Re-check at 29a4f9a: the lease-window wording ('the timeout it was claimed with plus the reap grace') and the reworded heartbeat aside match the code. No open accuracy findings.

bpamiri and others added 2 commits October 8, 2026 11:26
Fold in the merged lifecycle-hooks feature:

- New "Lifecycle hooks and failure handling" section: beforePerform/
  afterPerform (run under the timeout; a throw fails the attempt),
  onSuccess(result) / onFailure(error, attempt, isFinal) (best-effort,
  after the outcome is recorded), isFinal covering reaped and
  interrupted jobs, the app-wide set(jobsOnFailure=...) hook with its
  data-free struct, and the stored wheels_jobs.result column.
- Added the result column to the schema table and DDL.
- Added a 4.1 to 4.2 upgrade entry.
- Removed the now-empty "Coming in 4.2" section (remaining placeholders
  live in their own inline asides).

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Re-check at 7174b59 (#4498 folded in, against develop 87bbd01): accurate, with three LOW fixes.

Verified

  • beforePerform / afterPerform: beforePerform(data) and afterPerform(data, result) run inside the timed thread with perform(), called with named data / result arguments, and a throw fails the attempt.
  • onSuccess / onFailure: both run after the outcome is recorded, are best-effort (logged, never change the outcome), and are called positionally. onSuccess(result) is called with no argument when perform() returned nothing.
  • isFinal: true on the last failed retry, on a reaped last attempt (currentAttempts > maxRetries), and on an interrupted non-idempotent job.
  • jobsOnFailure:
    • "component.path.method" (split at the last dot);
    • the struct carries {jobId, jobClass, queue, attempt, maxRetries, isFinal, error} with no job data;
    • a hook that can't be called is logged once per app.
  • Stored result:
    • a simple value is stored as-is, anything else as JSON, an empty value as NULL;
    • over the limit, it's cut on a character boundary and ends with ...[truncated];
    • the column is NVARCHAR(4000) on SQL Server, VARCHAR2(4000) on Oracle and VARCHAR(4000) elsewhere, and nothing is stored without it.
  • Other pages: the upgrade entry is consistent, and the now-empty "Coming in 4.2" section is removed.

LOW

  1. Name the jobsOnFailure method's argument. $callJobsOnFailure() uses Invoke(hook, method, {event = …}), a named argument, so the method must declare a parameter called event (e.g. public void function report(required struct event)). A natural report(struct failure) receives nothing. Say so, and show the signature next to the set(jobsOnFailure = …) example.
  2. The example leaks a variable. order = model("Order").findByKey(…) in ChargeCardJob.perform() is unscoped, so it lands in the component's variables scope. Use var order = … (house style for examples).
  3. Cap wording. "capped at 4000 characters (bound as NVARCHAR on SQL Server, UTF-8 bytes elsewhere)" mixes units. The code caps at 4000 characters on SQL Server and 4000 UTF-8 bytes elsewhere, so non-ASCII text holds fewer characters there. Suggest: "capped at 4000 characters on SQL Server (NVARCHAR) and 4000 UTF-8 bytes on other databases".

bpamiri and others added 2 commits October 8, 2026 11:31
Per rev1-r1 on the #4498 fold:
- jobsOnFailure is invoked with a named `event` argument, so the hook
  method must declare `function report(required struct event)`. Show
  the signature in the guide and note it in the upgrade entry.
- Use `var order` in the ChargeCardJob example.
- The stored result cap is 4000 characters on SQL Server but 4000 UTF-8
  bytes on VARCHAR / Oracle VARCHAR2; reword to say so.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
…emantics

Per rev1-r1: only Oracle VARCHAR2 counts bytes; PG/MySQL/SQLite VARCHAR count characters, and the code caps at 4000 UTF-8 bytes there as a safe bound.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
bpamiri and others added 2 commits October 8, 2026 15:13
The groundwork the heartbeat section pointed at has landed: a job that
heartbeats is now reclaimed once its beats stop for a heartbeat grace,
even inside its timeout, instead of waiting out twice the timeout.

- Reframe the heartbeat section from "groundwork" to the real
  behaviour: the grace is jobsHeartbeatGraceSeconds (default 300, floor
  30, 0 = off) or this.heartbeatGrace; a job that never heartbeats
  keeps the full claim window; timeout stays the hard cap. Give the
  example a long timeout so the grace matters.
- The exclusive-job lease is renewed by the grace on each beat, so the
  lease and the reclaim run on one clock; retitle and update the aside.
- Drop the #4502 forward reference (it's this change) and add a 4.1 to
  4.2 upgrade entry.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Re-check at 307364c (#4507 folded in, against develop a6c2d00): mostly accurate; one MEDIUM and one LOW.

Verified

  • Grace setting: $heartbeatGraceSeconds() uses this.heartbeatGrace, else set(jobsHeartbeatGraceSeconds) (default 300). ≤ 0 turns it off, and the floor is Max(30, n).
  • Reclaim timing: a row that has heartbeated (hasBeat) is reaped at Min(claim window, heartbeat grace), so it can be reclaimed inside its timeout. A row that never heartbeated keeps the full timeout + Max(60, timeout) window.
  • Hard cap: timeout is still the cap on a single attempt.
  • Upgrade entry: consistent with the above.

MEDIUM: the 3600-second example doesn't hold under wheels jobs work

The example sets this.timeout = 3600; // a full rebuild may legitimately run up to an hour. Under the CLI worker, that job is stopped at 300 s:

  • wheels jobs work polls /wheels/cli?command=jobsProcessNext with no timeout;
  • CliBridge.jobsProcessNext() defaults it to 300;
  • JobWorker.$executeJob() runs perform() for Min(job timeout, workerCap) = Min(3600, 300).

Only processQueue() (per-job timeouts since #4495) and JobRunner.tick(timeout = …) honour a job timeout above 300. Either:

  • say so next to the example ("under wheels jobs work a job's timeout is capped at 300 s; run long jobs through tick(timeout = …) or processQueue()"), or
  • treat it as a code gap: the CLI worker has no --timeout option and silently caps every job at 300 s. That one is worth an issue regardless of the docs. Flagging it to orch1.

LOW: the claim and the lease don't lapse together

  • The aside says "each heartbeat() also renews the run's lease by the grace window … stop beating, and both the claim and the lease lapse together after the grace". The exclusive section says it renews "by the heartbeat grace".
  • In the code, $leaseRenewalSeconds() renews by grace + Max(60, grace) (600 s at the default), while the reaper reclaims after the grace alone (300 s).
  • So when a job stops beating, its claim is reclaimed first, and the lease lapses about one grace later. Until then, a reclaimed exclusive job is deferred (no attempt spent).
  • Suggest: "renews the lease by grace + max(60s, grace), so the lease outlasts the reclaim by about one grace, and a reclaimed exclusive job waits that long before its retry can take the lease."

bpamiri and others added 2 commits October 8, 2026 15:18
Per rev1-r1 on the #4507 fold:
- MEDIUM: this.timeout is only the requested cap; the wheels jobs work
  worker caps every job at 300 s (Min(job, workerCap) with workerCap
  defaulting to 300 and this.perJobTimeout false), so a 3600 s job is
  still stopped at 300 s under the stock worker. Added a caution: run a
  long job through processQueue(timeout=) or JobRunner.tick(timeout=).
- LOW: the exclusive lease is renewed to grace + max(60, grace) (600 s
  at the default), which outlasts the reclaim grace (300 s), so the
  claim and the lease don't "lapse together" — the reclaim happens
  first and the old lease frees shortly after. Reworded the aside and
  the exclusive-jobs line.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
Per rev1-r1:
- processQueue() already honours a job's own timeout (perJobTimeout
  mode), so the timeout= arg is optional and only lowers the cap; only
  wheels jobs work caps at 300. Reword and link #4517, which tracks the
  CLI cap.
- The exclusive lease renews to grace + max(60s, grace) — ~2x the grace
  (600s at the default), not "a little longer". After a reclaim the old
  lease frees about one grace later, and the retry waits deferred
  without spending an attempt. Reword the aside and the exclusive-jobs
  line.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Re-check at 2d2a264: the #4507 fold is accurate. The runner timeout caution (the CLI worker caps at 300 s, #4517; processQueue() honours per-job timeouts) and the lease timing (grace + max(60 s, grace), freeing about one grace after the reclaim, with the retry deferred without spending an attempt) match JobWorker.$executeJob and Job.$leaseRenewalSeconds on develop. No open accuracy findings.

bpamiri and others added 3 commits October 8, 2026 15:26
Per rev1-r1 reading the #4509 branch: jobsScheduleCheckSeconds defaults to 15, not 30; processQueue() also runs schedules (through the worker); and a draining host still enqueues due slots to run elsewhere. Still marked not-built-yet until #4509 merges.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
Two merged changes:

- #4499: wheels jobs install writes a migration that creates (or tops
  up) the job tables from the running app's own schema, and
  set(jobsAutoCreateTables = false) turns off runtime CREATE/ALTER
  (then a missing table throws Wheels.Job.SchemaMissing). Restructure
  "The shared schema" to lead with install + the setting as the
  recommended multi-server path, keeping the hand-written DDL as the
  fallback; replace the "planned" aside; add an upgrade entry.
- #4518 (fixes #4517): wheels jobs work now runs each job for its own
  timeout, with --job-timeout to cap, instead of stopping every job at
  300 s. Rewrite the long-timeout caution accordingly and add an
  upgrade entry.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Re-check at 64c2a6f (#4499 and #4518 folded in, against develop c284bc0): accurate, with three LOW nits.

Verified

  • wheels jobs install: writes app/migrator/migrations/<ts>_CreateWheelsJobTables.cfc from the running app's JobSchema. It covers all four tables, is idempotent (adopts or tops up), has down(), and needs a running server. Nothing touches the database until migrate latest.
  • jobsAutoCreateTables = false: stops runtime CREATE/ALTER, and a missing wheels_jobs makes enqueue() and a worker poll throw Wheels.Job.SchemaMissing.
  • fix(cli): let wheels jobs work run each job with its own timeout #4518: wheels jobs work runs each job for its own timeout, --job-timeout caps it, and processQueue(timeout = n) only lowers it. tick() still defaults to 300.
  • Other pages: both upgrade entries are consistent, and the schedule-runner aside is still marked not built (feat(job): run job schedules from the worker poll and JobRunner.tick() #4509 isn't merged).

LOW

  1. "With it set and a table missing, enqueue() and a worker poll throw SchemaMissing" is true for wheels_jobs, and for wheels_job_locks when an exclusive job runs. A missing wheels_job_hosts or wheels_job_schedules is logged once and that feature is off (drain/host record; schedules), not thrown. Say "a missing wheels_jobs table…", or add the auxiliary-table behaviour.
  2. down() should carry its warning here too. The generated migration's down() drops all four job tables, including ones it adopted rather than created, and every job in them. The generated header says so. The guide's "ships a down()" should too, since this is the page that recommends the command for production.
  3. "--timeout … stops the whole worker after that many seconds" is runtime-dependent. That's what happens on LuCLI builds where the root parser takes it (no module-timeout fix). On fixed builds it reaches the module, where jobs has no such option. Suggest: "don't use --timeout: on some CLI runtimes it's taken as the runtime's own whole-command limit and stops the worker".

bpamiri and others added 4 commits October 8, 2026 16:26
Per rev1-r1:
- Wheels.Job.SchemaMissing is thrown only for a missing wheels_jobs
  (and wheels_job_locks for exclusive jobs); a missing host-registry or
  schedules table just turns that feature off with a logged note.
- Warn that the install migration's down() is destructive: it drops all
  four job tables, including adopted ones, and every job in them.
- The --job-timeout note: --timeout stops the whole worker only on some
  CLI runtimes (where LuCLI's root option takes it), not universally.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
The #4518 line in background-jobs.mdx said --timeout stops the whole worker outright; it only does so on CLI runtimes where LuCLI's root option takes it. Match the wording used in the multi-server guide and upgrade entry.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
Fold in the merged token-gated tick route, replacing the placeholder
with a real "Drive it over HTTP: the tick route" subsection:

- POST /wheels/jobs/tick runs one JobRunner.tick() and returns JSON.
- The token is the boundary: 404 until set(jobsRunnerToken = "..."),
  403 without the right X-Wheels-Jobs-Token (constant-time), ?token=
  only with jobsRunnerTokenInQuery; any forwarding header is refused, so
  it's called on the server itself, not through a proxy. Answered before
  routing; overlapping ticks skipped.
- wheels jobs tick runs one tick from the CLI.
- Add a 4.1 to 4.2 upgrade entry.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Re-check at 4d76f39 (#4516 tick route folded in, against develop 85f0d8c): accurate on the security rules; one MEDIUM on throughput.

Verified against JobTickEndpoint.cfc

  • 404 until a token is set. It returns 404 unless jobsRunnerToken is set (non-blank).
  • The token. The token goes in X-Wheels-Jobs-Token. ?token= is accepted only with jobsRunnerTokenInQuery = true. The comparison is constant-time: both sides are SHA-256 hashed, then OR-of-XOR over the digests. A missing or wrong token gets 403.
  • Forwarding headers. Any of X-Forwarded-For/Host/Proto/Port/Prefix, Forwarded, X-Real-IP, X-Client-IP, CF-Connecting-IP, True-Client-IP or X-Cluster-Client-IP, even empty, gets 403 whatever the token.
  • Request handling.
    • It's answered before routing and the public-component gate.
    • An overlapping tick on the same host is skipped through a try-once per-host lock (200 with skipped).
    • A failed tick returns {ok: false, requestId}, with the error kept in the server log.
  • CLI. wheels jobs tick runs one tick through the CLI bridge on the running server, so no token is needed there.
  • Other text. The --timeout softening and the jobsAutoCreateTables / down() wording also match.

MEDIUM: each HTTP call runs at most one job, capped at 300 s

The route calls JobRunner.tick() with no arguments, so it uses tick's defaults:

  • maxJobs = 1: one job per call;
  • timeout = 300: each job is capped at 300 s;
  • all queues, and the configured per-host cap.

A cron curl "once a minute, from each server", as the page suggests, therefore processes at most one job per server per minute. A job with this.timeout above 300 is cut off at 300, and there's no request parameter to change either value. Readers coming from the tick(maxJobs = 5, timeout = …) example above will expect more.

  • Fix: say it ("each call runs one job (tick's defaults: maxJobs = 1, timeout = 300), so call it as often as your throughput needs, or use wheels jobs work / a CFML scheduled tick(maxJobs = n) for volume").
  • If more than one job per call is intended, that's a code change (tick parameters or settings on the route) for feat(job): token-gated /wheels/jobs/tick route and wheels jobs tick #4516's follow-up. Flagging it to orch1.

INFO

  • The route accepts GET as well as POST (other methods get 405). The page shows POST, which is fine; mention GET for schedulers that can only GET.
  • A tick runs inside the HTTP request, so the app server's request timeout also bounds it. That's another reason one short tick per call is the right shape.

- #4509: a running worker checks schedules itself — wheels jobs work,
  processQueue() and JobRunner.tick() each enqueue due slots as they
  poll, at most once every jobsScheduleCheckSeconds (default 15, 0 =
  off) per application. Rewrite "Running the schedules" so enqueueDue()
  is only for when nothing polls the queue; note the draining-server
  behaviour and tick()'s scheduled count; update the upgrade entry.
- #4516 review (rev1-r1): the tick route calls tick() with no args, so
  each call reaps, enqueues due schedules and runs at most ONE job
  (capped at 300 s), with no request parameter for either — say so and
  point volume at wheels jobs work or a CFML tick(maxJobs=n). Note GET
  and POST are both accepted.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Re-check at d9de7de: accurate, no findings.

Optional: link #4520 next to the one-job-per-call note, since it tracks making those limits configurable.

bpamiri and others added 3 commits October 8, 2026 17:20
The route runs one job per call today; #4520 will make its batch size and timeout configurable. Both orch1 and rev1-r1 asked for the forward link.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
The last multi-server piece. Rewrite "Clocks and skew" from "each
server's own clock, keep NTP tight" to the real behaviour: wheels.JobClock
keeps job timestamps on the database's UTC clock (read ~once a minute,
skew-corrected, compared in SQL), so skewed or differently-zoned servers
no longer disagree about stale/due. Document that H2 and SQLite use the
server's own UTC (H2 needs a UTC JVM); the CFML API stays local
(enqueueAt converts, status/stats convert back); and the install
migration converts pre-4.2 rows once via this.convertLocalOffsetMinutes
in one transaction with a marker (idempotent; SQLite skipped). Add the
upgrade entry.

With every placeholder now folded, this closes the guide.

Closes #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Final re-check at 9a23903 (#4486/#4521 UTC clock folded in, against develop aa1ea2e): accurate on the clock; one MEDIUM on the upgrade path.

Verified against JobClock.cfc / JobSchema.cfc

  • One database clock. It reads the database's UTC time (SYSUTCDATETIME, UTC_TIMESTAMP, clock_timestamp(), SYS_EXTRACT_UTC(SYSTIMESTAMP)) at most every 60 s per application and datasource, and keeps the skew. Timestamps are built from integers in SQL (timestampSql() / epochSql()), so comparisons still run in SQL. A failed read keeps the last good skew and logs once.
  • H2 and SQLite use the server's own UTC. The H2 UTC-JVM requirement matches fix(job): store job timestamps in UTC on the database's clock #4521's changelog and its skip-guarded H2 spec (H2 1.4 does timestamp arithmetic in the JVM zone).
  • The CFML API stays local: fromLocal() for enqueueAt(), toLocal() for status and stats.
  • The conversion. convertLocalOffsetMinutes runs once, in one transaction with a wheels_job_locks marker (conversionMarker()), so it's idempotent. SQLite is skipped (epoch ms).
  • The rest is consistent: the upgrade entry, the removed placeholder, and Closes #4485.

MEDIUM: the upgrade path is missing its other option and the cost of doing neither

Both the guide aside and the upgrade entry present convertLocalOffsetMinutes as the way to handle rows written before 4.2. The generated migration's own comment and #4521's changelog say more:

  1. The other option. Either drain every server (no pending or processing jobs) before upgrading and leave the value at 0, or set the offset.
  2. The cost of doing neither on servers that don't run in UTC: a delayed job runs early or late by the offset, and on a server behind UTC an in-flight job looks stale and is reclaimed and run again. That's the double-run this whole guide is about avoiding.
  3. Servers already on UTC need neither.
  4. The sign convention: minutes from UTC, -300 for UTC−5, 60 for UTC+1.

Add those four points (one or two sentences) to the aside and the upgrade entry. "Can shift them once" reads as optional housekeeping when it's the step that prevents early, late and double runs during the upgrade.

Per rev1-r1, enrich the pre-4.2 timestamp conversion guidance (the
aside and the upgrade entry), from the generated migration comment and
#4521's changelog:

- The drain alternative: upgrade with the queue empty (no pending/
  processing jobs) and leave convertLocalOffsetMinutes at 0.
- The cost of doing neither on a non-UTC server: a delayed job runs
  early/late by the offset, and on a server behind UTC an in-flight job
  looks stale and is reclaimed and run again.
- Servers already on UTC (and SQLite) need neither.
- The offset sign: -300 for UTC-5, 60 for UTC+1.

Refs #4485

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri

bpamiri commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Final accuracy pass at bdd935f: no open findings. The upgrade path for rows written before 4.2 now gives all four points:

  • drain first (empty queue, offset 0), or convert in place (convertLocalOffsetMinutes, with -300 for UTC−5 and 60 for UTC+1, one transaction plus a marker);
  • the cost of doing neither on a non-UTC server: early or late delayed jobs, and reclaim plus re-run behind UTC;
  • UTC servers and SQLite need nothing.

This matches the generated migration's comment and #4521's changelog. Across every fold (#4487–#4496, #4498, #4499, #4507, #4509, #4516, #4518, #4521), each API name, setting, default, return shape and guarantee I checked matches develop at aa1ea2e. I'm leaving the build/CI result to the PR checks (still running at the time of this comment).

@bpamiri
bpamiri merged commit 3a810db into develop Oct 9, 2026
24 checks passed
@bpamiri
bpamiri deleted the peter/4485-multi-server-jobs-guide branch October 9, 2026 05:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: "Running jobs on multiple servers" guide

1 participant