Repository navigation
feat(job): run job schedules from the worker poll and JobRunner.tick() - #4509
Conversation
Due schedule slots are now enqueued by every worker poll (wheels jobs work, processQueue()) and JobRunner.tick(), at most once every jobsScheduleCheckSeconds (default 15, 0 = off) per application, so schedules need no separate process. Each slot is still enqueued once however many servers check. The check only runs when the schedules table exists, and never throws. tick() reports `scheduled`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
Wheels Test Results147 895 tests 145 177 ✅ 59m 21s ⏱️ Results for commit 4e4fac2. ♻️ This comment has been updated with latest results. |
|
Review: head db35dcf. APPROVE (ready to merge). Two LOW, two INFO. Verified
Matrix 37849380151 at db35dcf:
PR CI: 23 pass, 0 fail, 2 skipped. LOW
INFO
|
Per rev1-r1 reading the #4509 branch: jobsScheduleCheckSeconds defaults to 15, not 30; processQueue() also runs schedules (through the worker); and a draining host still enqueues due slots to run elsewhere. Still marked not-built-yet until #4509 merges. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
…ng server A failing enqueueDue() leaves the poll running; with no schedules table nothing is checked (or created); a draining server's tick still enqueues due slots, which another server runs. Adds a $newScheduler() seam for the first two. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
…-runner Signed-off-by: Peter Amiri <peter@alurium.com> # Conflicts: # vendor/wheels/Job.cfc # vendor/wheels/events/onapplicationstart.cfc
|
Final review: head 4e4fac2. APPROVE (ready to merge).
|
- #4509: a running worker checks schedules itself — wheels jobs work, processQueue() and JobRunner.tick() each enqueue due slots as they poll, at most once every jobsScheduleCheckSeconds (default 15, 0 = off) per application. Rewrite "Running the schedules" so enqueueDue() is only for when nothing polls the queue; note the draining-server behaviour and tick()'s scheduled count; update the upgrade entry. - #4516 review (rev1-r1): the tick route calls tick() with no args, so each call reaps, enqueues due schedules and runs at most ONE job (capped at 300 s), with no request parameter for either — say so and point volume at wheels jobs work or a CFML tick(maxJobs=n). Note GET and POST are both accepted. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
* docs(web): add the 4.2 "Running Jobs on Multiple Servers" guide A companion to Background Jobs for when several app servers process one wheels_jobs table. It documents the multi-server features already on develop and marks the parts still to come. Covered (merged): - the guarantees: one recorded outcome per attempt (per-claim fencing), at-least-once for idempotent jobs, and why perform() must be idempotent - heartbeat() for long jobs, and Wheels.Job.Fenced - this.idempotent and the interrupted status (at-most-once) - set(jobsMaxConcurrentPerHost) and the host name override - runner options: the CLI worker, a scheduled processQueue(), JobRunner.tick() - draining a server before a deploy with wheels jobs drain --wait / resume - the shared schema to apply by hand on a live multi-server database - clock skew and NTP Marked as not-yet-built, each with its tracking issue: the token-gated /wheels/jobs/tick route, a single UTC database clock, recurring schedules, exclusive jobs and concurrency keys, lifecycle hooks, and wheels jobs install. Links both ways with Background Jobs, adds the sidebar and index entries, and records the heartbeat/idempotency, per-host cap, and jobs drain/resume changes on the 4.1 to 4.2 upgrade guide. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): correct the multi-server jobs guide per accuracy review Fixes from rev1-r1's pass on the new guide and the 4.1 to 4.2 upgrade entry: - heartbeat() does not extend a job's timeout. The timeout is a hard cap per attempt (perform() runs in a bounded thread and is stopped when it elapses); heartbeat() only records liveness. Reframed the section and the upgrade entry, and noted the lease-renewal (#4496) and dead-worker-detection (#4502) work that builds on it. - The heartbeatAt column is TIMESTAMP on PostgreSQL, CockroachDB, Oracle and H2 (they have no DATETIME); the manual DDL now gives per-database types. - Documented purge(status="interrupted") for pruning interrupted rows, instead of "use your own SQL". - Job timestamps are written with each server's application clock (Now()), not the database clock; corrected the clocks section. - Dropped "unless you turn it off" from the upgrade note: turning the auto-schema off needs jobsAutoCreateTables, which isn't built yet. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): fold the merged processQueue-through-worker change into the guide processQueue() now runs through the job worker (#4495), so it reaps stale jobs, honours the per-host cap, and stops starting new jobs on a draining host. Document that as fact instead of a forward reference, note that it still doesn't refresh the host registry's lastSeenAt the way tick() does (its running count and drain state are read live), and drop #4495 from the "Coming in 4.2" list now that it has landed. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): phrase the heartbeat lease-renewal note as forward-looking #4496 as approved sets a job's lease expiry at acquire and does not renew it from heartbeat() yet; the renewal is wired in #4496's develop merge. Word the liveness aside as "once #4496 lands, heartbeat() also renews an exclusive job's lease" so it's accurate until #4496 merges, then it becomes plain fact. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): document recurring job schedules (#4493) Fold the merged recurring-schedules feature into the multi-server jobs guide as a full section, replacing the placeholder: - The config/schedules.cfm DSL: schedule().job().cron() / .every(), .timezone(), .catchUp() / .catchUpWindow(), .queue()/.priority()/ .data()/.enabled(). wheels new doesn't create the file. - Running schedules with new wheels.JobScheduler().enqueueDue(): each slot carries a uniqueKey, so calling it on every server yields one job per slot. - Catch-up ("latest" default within catchUpWindow, "none"), the wheels_job_schedules table, code-vs-db schedule sources, and the java.time/DST notes for non-UTC time zones. Also correct the Background Jobs page, which said Wheels has no recurring schedule, to point here, and add a 4.1 to 4.2 upgrade entry. Remove recurring schedules from the "Coming in 4.2" list. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): mark the planned automatic schedule runner enqueueDue() is called manually today. Note the planned schedule-runner wiring: JobRunner.tick() and the CLI worker's poll will call it every jobsScheduleCheckSeconds (default 30), so a running worker fires schedules without a separate call. Clearly marked as not built yet. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): note the uniqueKey index on old SQL Server levels (#4494) Fold in the improved SQL Server guidance: the filtered unique index needs compatibility level 100 or higher. Below that, either raise the level or use a plain unique index, but only once every server runs 4.2 (a plain index allows one NULL, so a 4.1 server writing NULL keys would collide). The framework's warning and Wheels.Job.UniqueKeyUnavailable now name the failed step and the per-database fix. Added to the guide's schema section and the 4.1 to 4.2 upgrade entry; removed from "Coming in 4.2". Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): correct processQueue's skipped claim and add the fenced count Since #4495 routed processQueue() through the worker, a row another poller claims first is no longer counted as skipped — the atomic claim just fails and the loser moves on. Fix the stale "shows up in the skipped count" parenthetical, and add the fenced count (a current field) to the result-shape comment, with a pointer to the fencing guarantee. (leasesLost and the redefined skipped come with #4496.) Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): document exclusive jobs and concurrency keys (#4496) Fold in the merged lease-lock feature: - New "Exclusive jobs and concurrency keys" section: this.exclusive, this.concurrencyKey, and concurrencyKeyFor(data) (the method wins), the wheels_job_locks lease, a busy lease putting the job back to pending (counted as skipped), leasesLost, and the never-making- progress caution. - heartbeat() now renews an exclusive job's lease: flipped the forward-looking aside to plain fact. - Background Jobs: processQueue()'s result gains leasesLost, and its skipped/fenced/leasesLost counts are explained with links. - Added a 4.1 to 4.2 upgrade entry; noted wheels_job_locks and wheels_job_schedules as cheap auto-create tables; removed exclusive jobs from "Coming in 4.2". Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): correct the lease window source and soften the renewal wording Per rev1-r1: the exclusive-job lease window is sized from the timeout the job was CLAIMED with (the claimTimeout passed to $acquireJobLease), not this.timeout — they coincide only under processQueue. Reword to "the timeout it was claimed with plus the reap grace", matching the reaper language. And because that window outlasts the hard per-attempt timeout cap, the lease can't actually expire mid-run today, so drop the "expire out from under it" framing: heartbeat lease renewal is groundwork, like its effect on the reaper. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): document job lifecycle hooks and the failure hook (#4498) Fold in the merged lifecycle-hooks feature: - New "Lifecycle hooks and failure handling" section: beforePerform/ afterPerform (run under the timeout; a throw fails the attempt), onSuccess(result) / onFailure(error, attempt, isFinal) (best-effort, after the outcome is recorded), isFinal covering reaped and interrupted jobs, the app-wide set(jobsOnFailure=...) hook with its data-free struct, and the stored wheels_jobs.result column. - Added the result column to the schema table and DDL. - Added a 4.1 to 4.2 upgrade entry. - Removed the now-empty "Coming in 4.2" section (remaining placeholders live in their own inline asides). Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): fix jobsOnFailure signature, var scoping, and the result cap Per rev1-r1 on the #4498 fold: - jobsOnFailure is invoked with a named `event` argument, so the hook method must declare `function report(required struct event)`. Show the signature in the guide and note it in the upgrade entry. - Use `var order` in the ChargeCardJob example. - The stored result cap is 4000 characters on SQL Server but 4000 UTF-8 bytes on VARCHAR / Oracle VARCHAR2; reword to say so. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): clarify the result cap is a byte bound, not VARCHAR byte semantics Per rev1-r1: only Oracle VARCHAR2 counts bytes; PG/MySQL/SQLite VARCHAR count characters, and the code caps at 4000 UTF-8 bytes there as a safe bound. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): heartbeats shorten dead-worker detection (#4507) The groundwork the heartbeat section pointed at has landed: a job that heartbeats is now reclaimed once its beats stop for a heartbeat grace, even inside its timeout, instead of waiting out twice the timeout. - Reframe the heartbeat section from "groundwork" to the real behaviour: the grace is jobsHeartbeatGraceSeconds (default 300, floor 30, 0 = off) or this.heartbeatGrace; a job that never heartbeats keeps the full claim window; timeout stays the hard cap. Give the example a long timeout so the grace matters. - The exclusive-job lease is renewed by the grace on each beat, so the lease and the reclaim run on one clock; retitle and update the aside. - Drop the #4502 forward reference (it's this change) and add a 4.1 to 4.2 upgrade entry. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): qualify the long timeout and the lease-vs-reclaim windows Per rev1-r1 on the #4507 fold: - MEDIUM: this.timeout is only the requested cap; the wheels jobs work worker caps every job at 300 s (Min(job, workerCap) with workerCap defaulting to 300 and this.perJobTimeout false), so a 3600 s job is still stopped at 300 s under the stock worker. Added a caution: run a long job through processQueue(timeout=) or JobRunner.tick(timeout=). - LOW: the exclusive lease is renewed to grace + max(60, grace) (600 s at the default), which outlasts the reclaim grace (300 s), so the claim and the lease don't "lapse together" — the reclaim happens first and the old lease frees shortly after. Reworded the aside and the exclusive-jobs line. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): sharpen the runner cap and lease-window details Per rev1-r1: - processQueue() already honours a job's own timeout (perJobTimeout mode), so the timeout= arg is optional and only lowers the cap; only wheels jobs work caps at 300. Reword and link #4517, which tracks the CLI cap. - The exclusive lease renews to grace + max(60s, grace) — ~2x the grace (600s at the default), not "a little longer". After a reclaim the old lease frees about one grace later, and the retry waits deferred without spending an attempt. Reword the aside and the exclusive-jobs line. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): correct the schedule-runner placeholder default to 15 Per rev1-r1 reading the #4509 branch: jobsScheduleCheckSeconds defaults to 15, not 30; processQueue() also runs schedules (through the worker); and a draining host still enqueues due slots to run elsewhere. Still marked not-built-yet until #4509 merges. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): document wheels jobs install, and the fixed worker timeout Two merged changes: - #4499: wheels jobs install writes a migration that creates (or tops up) the job tables from the running app's own schema, and set(jobsAutoCreateTables = false) turns off runtime CREATE/ALTER (then a missing table throws Wheels.Job.SchemaMissing). Restructure "The shared schema" to lead with install + the setting as the recommended multi-server path, keeping the hand-written DDL as the fallback; replace the "planned" aside; add an upgrade entry. - #4518 (fixes #4517): wheels jobs work now runs each job for its own timeout, with --job-timeout to cap, instead of stopping every job at 300 s. Rewrite the long-timeout caution accordingly and add an upgrade entry. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): tighten the jobs-install schema notes Per rev1-r1: - Wheels.Job.SchemaMissing is thrown only for a missing wheels_jobs (and wheels_job_locks for exclusive jobs); a missing host-registry or schedules table just turns that feature off with a logged note. - Warn that the install migration's down() is destructive: it drops all four job tables, including adopted ones, and every job in them. - The --job-timeout note: --timeout stops the whole worker only on some CLI runtimes (where LuCLI's root option takes it), not universally. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): soften the --timeout worker note to runtime-dependent The #4518 line in background-jobs.mdx said --timeout stops the whole worker outright; it only does so on CLI runtimes where LuCLI's root option takes it. Match the wording used in the multi-server guide and upgrade entry. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): document the /wheels/jobs/tick route (#4516) Fold in the merged token-gated tick route, replacing the placeholder with a real "Drive it over HTTP: the tick route" subsection: - POST /wheels/jobs/tick runs one JobRunner.tick() and returns JSON. - The token is the boundary: 404 until set(jobsRunnerToken = "..."), 403 without the right X-Wheels-Jobs-Token (constant-time), ?token= only with jobsRunnerTokenInQuery; any forwarding header is refused, so it's called on the server itself, not through a proxy. Answered before routing; overlapping ticks skipped. - wheels jobs tick runs one tick from the CLI. - Add a 4.1 to 4.2 upgrade entry. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): schedules run from the worker (#4509); tick-route job limits - #4509: a running worker checks schedules itself — wheels jobs work, processQueue() and JobRunner.tick() each enqueue due slots as they poll, at most once every jobsScheduleCheckSeconds (default 15, 0 = off) per application. Rewrite "Running the schedules" so enqueueDue() is only for when nothing polls the queue; note the draining-server behaviour and tick()'s scheduled count; update the upgrade entry. - #4516 review (rev1-r1): the tick route calls tick() with no args, so each call reaps, enqueues due schedules and runs at most ONE job (capped at 300 s), with no request parameter for either — say so and point volume at wheels jobs work or a CFML tick(maxJobs=n). Note GET and POST are both accepted. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): link #4520 from the tick-route one-job-per-call note The route runs one job per call today; #4520 will make its batch size and timeout configurable. Both orch1 and rev1-r1 asked for the forward link. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): job timestamps on the database's UTC clock (#4486) The last multi-server piece. Rewrite "Clocks and skew" from "each server's own clock, keep NTP tight" to the real behaviour: wheels.JobClock keeps job timestamps on the database's UTC clock (read ~once a minute, skew-corrected, compared in SQL), so skewed or differently-zoned servers no longer disagree about stale/due. Document that H2 and SQLite use the server's own UTC (H2 needs a UTC JVM); the CFML API stays local (enqueueAt converts, status/stats convert back); and the install migration converts pre-4.2 rows once via this.convertLocalOffsetMinutes in one transaction with a marker (idempotent; SQLite skipped). Add the upgrade entry. With every placeholder now folded, this closes the guide. Closes #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> * docs(web): spell out the UTC conversion upgrade path Per rev1-r1, enrich the pre-4.2 timestamp conversion guidance (the aside and the upgrade entry), from the generated migration comment and #4521's changelog: - The drain alternative: upgrade with the queue empty (no pending/ processing jobs) and leave convertLocalOffsetMinutes at 0. - The cost of doing neither on a non-UTC server: a delayed job runs early/late by the offset, and on a server behind UTC an in-flight job looks stale and is reclaimed and run again. - Servers already on UTC (and SQLite) need neither. - The offset sign: -300 for UTC-5, 60 for UTC+1. Refs #4485 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com> --------- Signed-off-by: Peter Amiri <peter@alurium.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Refs #4481 (recurring schedules, merged in #4493). This is the follow-up that runs them.
Problem
Since #4493 an app can define schedules (
config/schedules.cfm,schedule()), but nothing calledJobScheduler.enqueueDue(). Due slots were only enqueued if the app called it itself.Change
Due schedule slots are now enqueued from the normal job processing:
JobWorker.processNext()checks right after its reap, sowheels jobs work,processQueue()andJobRunner.tick()all cover it.JobRunner.tick()checks before it dispatches, so a slot due now can run in the same tick. It reports the slots it enqueued asscheduled.Throttle. At most one check every
jobsScheduleCheckSeconds, default 15;0turns it off. The throttle is per application, under a short exclusive lock, so concurrent polls on one server don't stack checks.Several servers. Any number of servers can check at once: each slot is enqueued exactly once, through its
uniqueKey(#4493).Apps without schedules. They are unaffected: the check only runs when the schedules table exists, so it never creates one.
Never breaks a poll. A failing check, or a schedule error, is logged and the poll carries on. It also skips inside a Wheels transaction.
Tests
JobScheduleRunnerSpec(4):tick()enqueues it and reportsscheduled = 1;jobsScheduleCheckSecondsdoes nothing;0turns it off.On
develop:scheduledkey and no throttled check.Local results:
Docs
The guide's schedules section (#4485) says that schedules run from the worker or a tick and covers
jobsScheduleCheckSeconds. Changelog fragment included.🤖 Generated with Claude Code