Repository navigation
docs(web): add the 4.2 "Running Jobs on Multiple Servers" guide - #4501
Conversation
A companion to Background Jobs for when several app servers process one wheels_jobs table. It documents the multi-server features already on develop and marks the parts still to come. Covered (merged): - the guarantees: one recorded outcome per attempt (per-claim fencing), at-least-once for idempotent jobs, and why perform() must be idempotent - heartbeat() for long jobs, and Wheels.Job.Fenced - this.idempotent and the interrupted status (at-most-once) - set(jobsMaxConcurrentPerHost) and the host name override - runner options: the CLI worker, a scheduled processQueue(), JobRunner.tick() - draining a server before a deploy with wheels jobs drain --wait / resume - the shared schema to apply by hand on a live multi-server database - clock skew and NTP Marked as not-yet-built, each with its tracking issue: the token-gated /wheels/jobs/tick route, a single UTC database clock, recurring schedules, exclusive jobs and concurrency keys, lifecycle hooks, and wheels jobs install. Links both ways with Background Jobs, adds the sidebar and index entries, and records the heartbeat/idempotency, per-host cap, and jobs drain/resume changes on the 4.1 to 4.2 upgrade guide. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Accuracy review: head b97fbfd, against develop at 631f1a8 (#4487–#4492 merged). CHANGES NEEDED: two HIGH, one MEDIUM, three LOW. The placeholders for #4482, #4486 and #4499 and the "Coming in 4.2" list are fine as interim text. HIGH1.
2.
MEDIUM3. The hand-written DDL uses
LOW
Verified accurate
|
Fixes from rev1-r1's pass on the new guide and the 4.1 to 4.2 upgrade entry: - heartbeat() does not extend a job's timeout. The timeout is a hard cap per attempt (perform() runs in a bounded thread and is stopped when it elapses); heartbeat() only records liveness. Reframed the section and the upgrade entry, and noted the lease-renewal (#4496) and dead-worker-detection (#4502) work that builds on it. - The heartbeatAt column is TIMESTAMP on PostgreSQL, CockroachDB, Oracle and H2 (they have no DATETIME); the manual DDL now gives per-database types. - Documented purge(status="interrupted") for pruning interrupted rows, instead of "use your own SQL". - Job timestamps are written with each server's application clock (Now()), not the database clock; corrected the clocks section. - Dropped "unless you turn it off" from the upgrade note: turning the auto-schema off needs jobsAutoCreateTables, which isn't built yet. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
…e guide processQueue() now runs through the job worker (#4495), so it reaps stale jobs, honours the per-host cap, and stops starting new jobs on a draining host. Document that as fact instead of a forward reference, note that it still doesn't refresh the host registry's lastSeenAt the way tick() does (its running count and drain state are read live), and drop #4495 from the "Coming in 4.2" list now that it has landed. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Accuracy re-check: head ee1c853, against develop at 47ce1a8 (#4495 now merged). All six earlier findings are resolved. One small new inaccuracy, in text added by the fix. Resolved
New LOW
Everything else I checked last round still holds on 47ce1a8. The PR diff against develop is docs-only (5 files). Neutral wording throughout. |
#4496 as approved sets a job's lease expiry at acquire and does not renew it from heartbeat() yet; the renewal is wired in #4496's develop merge. Word the liveness aside as "once #4496 lands, heartbeat() also renews an exclusive job's lease" so it's accurate until #4496 merges, then it becomes plain fact. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Re-check at e12ffec: the liveness aside now says lease renewal arrives with #4496 ("once #4496 lands, |
Fold the merged recurring-schedules feature into the multi-server jobs
guide as a full section, replacing the placeholder:
- The config/schedules.cfm DSL: schedule().job().cron() / .every(),
.timezone(), .catchUp() / .catchUpWindow(), .queue()/.priority()/
.data()/.enabled(). wheels new doesn't create the file.
- Running schedules with new wheels.JobScheduler().enqueueDue(): each
slot carries a uniqueKey, so calling it on every server yields one job
per slot.
- Catch-up ("latest" default within catchUpWindow, "none"), the
wheels_job_schedules table, code-vs-db schedule sources, and the
java.time/DST notes for non-UTC time zones.
Also correct the Background Jobs page, which said Wheels has no
recurring schedule, to point here, and add a 4.1 to 4.2 upgrade entry.
Remove recurring schedules from the "Coming in 4.2" list.
Refs #4485
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
|
Re-check at e9eea3e ("Recurring schedules", against #4493 merged as d1b8df2): accurate, no findings. Verified against
|
enqueueDue() is called manually today. Note the planned schedule-runner wiring: JobRunner.tick() and the CLI worker's poll will call it every jobsScheduleCheckSeconds (default 30), so a running worker fires schedules without a separate call. Clearly marked as not built yet. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
Fold in the improved SQL Server guidance: the filtered unique index needs compatibility level 100 or higher. Below that, either raise the level or use a plain unique index, but only once every server runs 4.2 (a plain index allows one NULL, so a 4.1 server writing NULL keys would collide). The framework's warning and Wheels.Job.UniqueKeyUnavailable now name the failed step and the per-database fix. Added to the guide's schema section and the 4.1 to 4.2 upgrade entry; removed from "Coming in 4.2". Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Re-check at 86790fe (#4494 folded in): accurate, no findings. Checked against
The 4.1→4.2 entry is consistent with this, and #4494 has been removed from "Coming in 4.2". |
Since #4495 routed processQueue() through the worker, a row another poller claims first is no longer counted as skipped — the atomic claim just fails and the loser moves on. Fix the stale "shows up in the skipped count" parenthetical, and add the fenced count (a current field) to the result-shape comment, with a pointer to the fencing guarantee. (leasesLost and the redefined skipped come with #4496.) Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
Fold in the merged lease-lock feature: - New "Exclusive jobs and concurrency keys" section: this.exclusive, this.concurrencyKey, and concurrencyKeyFor(data) (the method wins), the wheels_job_locks lease, a busy lease putting the job back to pending (counted as skipped), leasesLost, and the never-making- progress caution. - heartbeat() now renews an exclusive job's lease: flipped the forward-looking aside to plain fact. - Background Jobs: processQueue()'s result gains leasesLost, and its skipped/fenced/leasesLost counts are explained with links. - Added a 4.1 to 4.2 upgrade entry; noted wheels_job_locks and wheels_job_schedules as cheap auto-create tables; removed exclusive jobs from "Coming in 4.2". Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Re-check at da38a4d (#4496 folded in, against develop e63973b): accurate, one LOW nit. Verified
LOW
INFO
|
…ding Per rev1-r1: the exclusive-job lease window is sized from the timeout the job was CLAIMED with (the claimTimeout passed to $acquireJobLease), not this.timeout — they coincide only under processQueue. Reword to "the timeout it was claimed with plus the reap grace", matching the reaper language. And because that window outlasts the hard per-attempt timeout cap, the lease can't actually expire mid-run today, so drop the "expire out from under it" framing: heartbeat lease renewal is groundwork, like its effect on the reaper. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Re-check at 29a4f9a: the lease-window wording ('the timeout it was claimed with plus the reap grace') and the reworded heartbeat aside match the code. No open accuracy findings. |
Fold in the merged lifecycle-hooks feature: - New "Lifecycle hooks and failure handling" section: beforePerform/ afterPerform (run under the timeout; a throw fails the attempt), onSuccess(result) / onFailure(error, attempt, isFinal) (best-effort, after the outcome is recorded), isFinal covering reaped and interrupted jobs, the app-wide set(jobsOnFailure=...) hook with its data-free struct, and the stored wheels_jobs.result column. - Added the result column to the schema table and DDL. - Added a 4.1 to 4.2 upgrade entry. - Removed the now-empty "Coming in 4.2" section (remaining placeholders live in their own inline asides). Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Re-check at 7174b59 (#4498 folded in, against develop 87bbd01): accurate, with three LOW fixes. Verified
LOW
|
Per rev1-r1 on the #4498 fold: - jobsOnFailure is invoked with a named `event` argument, so the hook method must declare `function report(required struct event)`. Show the signature in the guide and note it in the upgrade entry. - Use `var order` in the ChargeCardJob example. - The stored result cap is 4000 characters on SQL Server but 4000 UTF-8 bytes on VARCHAR / Oracle VARCHAR2; reword to say so. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
…emantics Per rev1-r1: only Oracle VARCHAR2 counts bytes; PG/MySQL/SQLite VARCHAR count characters, and the code caps at 4000 UTF-8 bytes there as a safe bound. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
The groundwork the heartbeat section pointed at has landed: a job that heartbeats is now reclaimed once its beats stop for a heartbeat grace, even inside its timeout, instead of waiting out twice the timeout. - Reframe the heartbeat section from "groundwork" to the real behaviour: the grace is jobsHeartbeatGraceSeconds (default 300, floor 30, 0 = off) or this.heartbeatGrace; a job that never heartbeats keeps the full claim window; timeout stays the hard cap. Give the example a long timeout so the grace matters. - The exclusive-job lease is renewed by the grace on each beat, so the lease and the reclaim run on one clock; retitle and update the aside. - Drop the #4502 forward reference (it's this change) and add a 4.1 to 4.2 upgrade entry. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Re-check at 307364c (#4507 folded in, against develop a6c2d00): mostly accurate; one MEDIUM and one LOW. Verified
MEDIUM: the 3600-second example doesn't hold under
|
Per rev1-r1 on the #4507 fold: - MEDIUM: this.timeout is only the requested cap; the wheels jobs work worker caps every job at 300 s (Min(job, workerCap) with workerCap defaulting to 300 and this.perJobTimeout false), so a 3600 s job is still stopped at 300 s under the stock worker. Added a caution: run a long job through processQueue(timeout=) or JobRunner.tick(timeout=). - LOW: the exclusive lease is renewed to grace + max(60, grace) (600 s at the default), which outlasts the reclaim grace (300 s), so the claim and the lease don't "lapse together" — the reclaim happens first and the old lease frees shortly after. Reworded the aside and the exclusive-jobs line. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
Per rev1-r1: - processQueue() already honours a job's own timeout (perJobTimeout mode), so the timeout= arg is optional and only lowers the cap; only wheels jobs work caps at 300. Reword and link #4517, which tracks the CLI cap. - The exclusive lease renews to grace + max(60s, grace) — ~2x the grace (600s at the default), not "a little longer". After a reclaim the old lease frees about one grace later, and the retry waits deferred without spending an attempt. Reword the aside and the exclusive-jobs line. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Re-check at 2d2a264: the #4507 fold is accurate. The runner timeout caution (the CLI worker caps at 300 s, #4517; |
Per rev1-r1 reading the #4509 branch: jobsScheduleCheckSeconds defaults to 15, not 30; processQueue() also runs schedules (through the worker); and a draining host still enqueues due slots to run elsewhere. Still marked not-built-yet until #4509 merges. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
Two merged changes: - #4499: wheels jobs install writes a migration that creates (or tops up) the job tables from the running app's own schema, and set(jobsAutoCreateTables = false) turns off runtime CREATE/ALTER (then a missing table throws Wheels.Job.SchemaMissing). Restructure "The shared schema" to lead with install + the setting as the recommended multi-server path, keeping the hand-written DDL as the fallback; replace the "planned" aside; add an upgrade entry. - #4518 (fixes #4517): wheels jobs work now runs each job for its own timeout, with --job-timeout to cap, instead of stopping every job at 300 s. Rewrite the long-timeout caution accordingly and add an upgrade entry. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Re-check at 64c2a6f (#4499 and #4518 folded in, against develop c284bc0): accurate, with three LOW nits. Verified
LOW
|
Per rev1-r1: - Wheels.Job.SchemaMissing is thrown only for a missing wheels_jobs (and wheels_job_locks for exclusive jobs); a missing host-registry or schedules table just turns that feature off with a logged note. - Warn that the install migration's down() is destructive: it drops all four job tables, including adopted ones, and every job in them. - The --job-timeout note: --timeout stops the whole worker only on some CLI runtimes (where LuCLI's root option takes it), not universally. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
The #4518 line in background-jobs.mdx said --timeout stops the whole worker outright; it only does so on CLI runtimes where LuCLI's root option takes it. Match the wording used in the multi-server guide and upgrade entry. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
Fold in the merged token-gated tick route, replacing the placeholder with a real "Drive it over HTTP: the tick route" subsection: - POST /wheels/jobs/tick runs one JobRunner.tick() and returns JSON. - The token is the boundary: 404 until set(jobsRunnerToken = "..."), 403 without the right X-Wheels-Jobs-Token (constant-time), ?token= only with jobsRunnerTokenInQuery; any forwarding header is refused, so it's called on the server itself, not through a proxy. Answered before routing; overlapping ticks skipped. - wheels jobs tick runs one tick from the CLI. - Add a 4.1 to 4.2 upgrade entry. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Re-check at 4d76f39 (#4516 tick route folded in, against develop 85f0d8c): accurate on the security rules; one MEDIUM on throughput. Verified against
|
- #4509: a running worker checks schedules itself — wheels jobs work, processQueue() and JobRunner.tick() each enqueue due slots as they poll, at most once every jobsScheduleCheckSeconds (default 15, 0 = off) per application. Rewrite "Running the schedules" so enqueueDue() is only for when nothing polls the queue; note the draining-server behaviour and tick()'s scheduled count; update the upgrade entry. - #4516 review (rev1-r1): the tick route calls tick() with no args, so each call reaps, enqueues due schedules and runs at most ONE job (capped at 300 s), with no request parameter for either — say so and point volume at wheels jobs work or a CFML tick(maxJobs=n). Note GET and POST are both accepted. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Re-check at d9de7de: accurate, no findings.
Optional: link #4520 next to the one-job-per-call note, since it tracks making those limits configurable. |
The last multi-server piece. Rewrite "Clocks and skew" from "each server's own clock, keep NTP tight" to the real behaviour: wheels.JobClock keeps job timestamps on the database's UTC clock (read ~once a minute, skew-corrected, compared in SQL), so skewed or differently-zoned servers no longer disagree about stale/due. Document that H2 and SQLite use the server's own UTC (H2 needs a UTC JVM); the CFML API stays local (enqueueAt converts, status/stats convert back); and the install migration converts pre-4.2 rows once via this.convertLocalOffsetMinutes in one transaction with a marker (idempotent; SQLite skipped). Add the upgrade entry. With every placeholder now folded, this closes the guide. Closes #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Final re-check at 9a23903 (#4486/#4521 UTC clock folded in, against develop aa1ea2e): accurate on the clock; one MEDIUM on the upgrade path. Verified against
|
Per rev1-r1, enrich the pre-4.2 timestamp conversion guidance (the aside and the upgrade entry), from the generated migration comment and #4521's changelog: - The drain alternative: upgrade with the queue empty (no pending/ processing jobs) and leave convertLocalOffsetMinutes at 0. - The cost of doing neither on a non-UTC server: a delayed job runs early/late by the offset, and on a server behind UTC an in-flight job looks stale and is reclaimed and run again. - Servers already on UTC (and SQLite) need neither. - The offset sign: -300 for UTC-5, 60 for UTC+1. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
|
Final accuracy pass at bdd935f: no open findings. The upgrade path for rows written before 4.2 now gives all four points:
This matches the generated migration's comment and #4521's changelog. Across every fold (#4487–#4496, #4498, #4499, #4507, #4509, #4516, #4518, #4521), each API name, setting, default, return shape and guarantee I checked matches develop at aa1ea2e. I'm leaving the build/CI result to the PR checks (still running at the time of this comment). |
What
Adds a new 4.2 Digging Deeper guide, Running Jobs on Multiple Servers, as a companion to Background Jobs for when several app servers process one
wheels_jobstable.It documents the multi-server job features already on
developand clearly marks the parts still to come.Covered (merged on develop)
perform()must be idempotent.heartbeat()for long-running jobs, and whatWheels.Job.Fencedmeans.this.idempotentand theinterruptedstatus (at-most-once recovery for work that must not run twice).set(jobsMaxConcurrentPerHost = n)and thejobsHostNameoverride.processQueue(), andJobRunner.tick().wheels jobs drain --wait/wheels jobs resume, the host registry,jobsCodeVersion.ALTERthe table), with the column list and SQL.Marked as not-yet-built (each links its tracking issue)
The token-gated
/wheels/jobs/tickroute, a single UTC database clock, recurring schedules, exclusive jobs and concurrency keys, lifecycle hooks, andwheels jobs install.Also
wheels jobs drain/resumechanges on the 4.1 → 4.2 upgrade guide (next to the existingclaimTimeout, fencing, anduniqueKeyentries).Notes
uniqueKeydedupe (feat(job): de-duplicate enqueue with uniqueKey #4488),heartbeat()/idempotency (feat(job): heartbeat() and idempotency-aware reaping #4490), per-host cap +JobRunnerdrain/tick (feat(job): per-host concurrency cap, drain/resume and JobRunner.tick() #4491), CLI drain/resume (feat(cli): wheels jobs drain [--wait] and resume for deploys #4492),processQueue()through the worker (fix(job): run processQueue() through the job worker #4495), recurring schedules (feat(job): recurring schedules (cron and intervals) with one job per slot #4493) and the schedule runner (feat(job): run job schedules from the worker poll and JobRunner.tick() #4509), SQL Server compat (fix(job): explain uniqueKey index failures on old SQL Server levels #4494), exclusive jobs / concurrency keys (feat(job): exclusive jobs and concurrency keys on a shared lease lock #4496), lifecycle hooks +jobsOnFailure+ stored result (feat(job): lifecycle hooks, an app-wide failure hook, and a stored result #4498), heartbeat-grace reclaim (fix(job): reclaim a job when its heartbeats stop, not only after 2x its timeout #4507), per-job worker timeout (fix(cli): let wheels jobs work run each job with its own timeout #4518), the/wheels/jobs/tickroute (feat(job): token-gated /wheels/jobs/tick route and wheels jobs tick #4516),wheels jobs install+jobsAutoCreateTables(feat(cli): wheels jobs install and jobsAutoCreateTables #4499), and the database UTC clock (fix(job): write and compare job timestamps on a single UTC database clock #4486). No placeholders remain.pnpm --filter @wheels-dev/site-guides build(854 pages; new page and all internal anchors resolve).🤖 Generated with Claude Code