diff --git a/web/sites/guides/src/content/docs/v4-2-0/digging-deeper/background-jobs.mdx b/web/sites/guides/src/content/docs/v4-2-0/digging-deeper/background-jobs.mdx index bcaebcb434..81b1f9086b 100644 --- a/web/sites/guides/src/content/docs/v4-2-0/digging-deeper/background-jobs.mdx +++ b/web/sites/guides/src/content/docs/v4-2-0/digging-deeper/background-jobs.mdx @@ -131,7 +131,7 @@ wheels jobs enqueue billing.InvoiceJob --queue=billing --priority=5 | `--at` | Run it at this time, in ISO 8601: with `Z` or an offset (`2026-10-06T11:00:00+02:00`), or without one for this machine's local time (`2026-10-06 09:00`). A time in the past is refused. | | `--format=json` | Print the enqueued job as JSON. | -"Scheduling" here means one delayed run (`--in` / `--at`, the same as `enqueueIn()` / `enqueueAt()`). Wheels has no recurring, cron-style schedule; for that, run `wheels jobs enqueue` or `processQueue()` from your system's scheduler. +"Scheduling" here means one delayed run (`--in` / `--at`, the same as `enqueueIn()` / `enqueueAt()`). For a *recurring*, cron-style schedule, declare it in `config/schedules.cfm` — see [Recurring schedules](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#recurring-schedules). The class must be a component under `app/jobs/` (or a `jobClassPrefixes` path, the same list the worker accepts) that extends `wheels.Job`. Anything else is refused before it's loaded, and an unknown name lists the job classes it found. Enqueueing writes a row, so like `wheels jobs work` it only talks to this project's own server and sends the reload password. @@ -220,10 +220,14 @@ wheels jobs work --quiet # suppress per-job output; failur wheels jobs work --job-timeout=600 # cap each job's run time at 600 seconds (default: each job's own timeout) ``` -Each job runs with its own `timeout` (set in `config()`, default `300` seconds), so a job with `this.timeout = 900` gets its full 15 minutes. `--job-timeout=` caps that: a job with a longer `timeout` is cut off at the cap, and one with a shorter `timeout` keeps its own. (Don't use `--timeout`: that is the CLI runtime's own option, which stops the whole worker after that many seconds.) Before 4.2 the worker stopped every job at 300 seconds, whatever its `timeout`. +Each job runs with its own `timeout` (set in `config()`, default `300` seconds), so a job with `this.timeout = 900` gets its full 15 minutes. `--job-timeout=` caps that: a job with a longer `timeout` is cut off at the cap, and one with a shorter `timeout` keeps its own. (Use `--job-timeout`, not `--timeout`: on some CLI runtimes `--timeout` is the runtime's own root option and would stop the whole worker after that many seconds.) Before 4.2 the worker stopped every job at 300 seconds, whatever its `timeout`. In production, run the worker under a process supervisor (systemd, Docker's restart policy, Kamal accessory containers): when it loses the server it exits non-zero and the supervisor restarts it. Run several workers for parallelism — the framework uses optimistic row locking, so two workers never run the same job twice. `--queue` accepts a comma-delimited list (`--queue=critical,default`); jobs across the listed queues run in `priority DESC, runAt ASC` order. + + Check queue health without tailing logs: ```bash title="your shell" @@ -243,7 +247,7 @@ component extends="Controller" { function drain() { worker = new wheels.Job(); result = worker.processQueue(queue="mailers", limit=10); - // result = {processed: n, failed: n, skipped: n, errors: [...]} + // result = {processed: n, failed: n, skipped: n, fenced: n, leasesLost: n, errors: [...]} renderWith(data=result); } } @@ -253,7 +257,7 @@ component extends="Controller" { - `processQueue(queue="mailers")` — only drain the `mailers` queue. - `processQueue(limit=100)` — bigger batch for one-shot catch-ups. -In production, schedule the call at whatever cadence matches your latency tolerance — every minute is plenty for emails. Run it from multiple schedulers for parallelism: the base class uses optimistic row locking, so two pollers won't process the same job twice (a row claimed between the SELECT and the claim shows up in the `skipped` count). +In production, schedule the call at whatever cadence matches your latency tolerance — every minute is plenty for emails. Run it from multiple schedulers for parallelism: the base class uses optimistic row locking, so two pollers won't process the same job twice — the atomic claim means the poller that loses the race simply doesn't get that row and moves on. The extra counts cover the multi-server cases: `fenced` is attempts whose result was discarded because the job had been reaped and reclaimed, `skipped` is [exclusive jobs](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#exclusive-jobs-and-concurrency-keys) whose lease was held (put back to `pending`), and `leasesLost` is runs that finished but had lost their lease. See [the guarantees](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#the-guarantees). Check queue health without tailing logs — `queueStats()` returns per-status counts: @@ -447,6 +451,11 @@ Run the purge from a cron or scheduled task. A weekly purge of completed rows ol ## Related guides + + + Read [Background Jobs](/v4-2-0/digging-deeper/background-jobs/) first. This page assumes you know how to define a job, enqueue it, and process the queue with `wheels jobs work` or `processQueue()`. It builds on those; it doesn't repeat them. + + +## The guarantees + +Several workers polling the same table is safe by design: each job is claimed with an atomic, guarded `UPDATE` (`WHERE id = :id AND status = 'pending'`), so two workers never start the same row. Beyond that, 4.2 makes the recovery path safe when a worker dies or stalls mid-job. The guarantees are worth stating exactly, because the two things they ask of your code follow directly from them. + +- **One job runs on one worker at a time.** The claim is atomic. A row in `processing` is owned by whichever worker claimed it. +- **A crashed worker's job is recovered.** A worker killed mid-run leaves its row stranded in `processing`; the next poll on that queue reaps it (past a grace margin built from the claiming worker's own `timeout`) and requeues it for retry, or fails it once retries are exhausted. This is covered in full under [Crashed workers and the `timeout`](/v4-2-0/digging-deeper/background-jobs/#crashed-workers-and-the-timeout). +- **One recorded *outcome* per attempt — not one execution.** A worker that was only stalled (a long GC pause, a hung remote call) can come back to life after its job was reaped and claimed again. Every claim writes a fresh `claimToken` to the row, and a worker may only complete, retry, or fail a job while the row still carries *its* token. The late attempt's result is discarded and logged as `Wheels.Job.Fenced`. Its `perform()` still ran to the point it stalled, though — so the work may have happened twice even though only one outcome was recorded. + +Two consequences for your code: + +1. **`perform()` must be idempotent.** Recovery is at-least-once: a reaped job runs again, and a fenced attempt already did part of its work. Write `perform()` so running it twice is the same as running it once — check-before-write, store an idempotency key, lean on a unique database constraint. This is the single most important rule for jobs on more than one server. +2. **The guarantees hold only once every worker is on 4.2.** Fencing lives in the claim and the terminal updates. A worker still on 4.1 claims and completes a job *without* a token, so it can still overwrite a newer attempt. During a rolling upgrade, expect the old behaviour until the last 4.1 worker is gone. Jobs already `processing` when a worker upgrades have no token and are reaped the old way. + + + +## Record liveness with `heartbeat()` + +A job's `timeout` is a **hard cap on a single attempt**: `perform()` runs in a bounded thread and is stopped when the timeout elapses, however much work is left (the attempt is then retried or failed like any timeout). `heartbeat()` does **not** extend that cap — a job can't run past its `timeout`. What it does is let a **long** job's worker-death be noticed quickly. Call it from inside `perform()` as the work proceeds, and it stamps the row's liveness; a job that has heartbeated is reclaimed once its beats stop for the **heartbeat grace** — even inside its `timeout`, so its recovery no longer waits out a long timeout. + +```cfm {test:compile} title="app/jobs/RebuildSearchIndexJob.cfc" +component extends="wheels.Job" { + function config() { + super.config(); + this.queue = "maintenance"; + this.timeout = 3600; // a full rebuild may legitimately run up to an hour + } + + public void function perform(struct data = {}) { + var job = this; + var onBatch = function(batch) { + reindex(batch); + job.heartbeat(); // "still alive" — a dead worker is now caught within the grace, not an hour + }; + model("Product").findInBatches(batchSize = 500, callback = onBatch); + } +} +``` + +`heartbeat()` stamps liveness on **this attempt's claim only** (it carries the same `claimToken` guard as a completion). Call it between units of work — after each batch, each file, each page — not in a tight inner loop. + + + +The grace is `jobsHeartbeatGraceSeconds` (default `300`, floor `30`, `0` turns it off), set app-wide with `set(jobsHeartbeatGraceSeconds = n)` or per class with `this.heartbeatGrace`. With it, a heartbeating job whose worker dies is reclaimed about that many seconds after its last beat, instead of waiting for the full claim window (`timeout` plus the reap grace). A job that **never** heartbeats keeps that full claim window — heartbeating is what opts a long job into faster recovery. + +If the claim is gone — the job was already reaped and handed to another worker while this one stalled — `heartbeat()` throws `Wheels.Job.Fenced`. That's your signal to **stop and return from `perform()`**: another attempt owns the job now, and anything more this attempt does will be discarded. Let the exception propagate; don't catch it to keep working. + + + + + +## Idempotency and `this.idempotent` + +Some work genuinely must not run twice, and can't be made idempotent — a job that posts to a ledger, say, where a replay is a real double-post. For those, `this.idempotent` changes what happens when a reaped job's worker is presumed dead: + +```cfm {test:compile} title="app/jobs/PostLedgerEntryJob.cfc" +component extends="wheels.Job" { + function config() { + super.config(); + this.queue = "ledger"; + this.idempotent = false; // do NOT re-run after a reap + } + + public void function perform(struct data = {}) { + // A replay here would double-post. Better to stop than to risk it. + postEntry(arguments.data.entryId); + } +} +``` + +- `this.idempotent = true` (the **default**) — recovery is **at-least-once**. A reaped job is retried, the attempt counts against `maxRetries`, and your idempotent `perform()` absorbs the replay. This is what every job does unless you say otherwise. +- `this.idempotent = false` — recovery is **at-most-once**. A reaped attempt is **not** re-run: the row ends in the new status `interrupted`, and stops there. You've chosen "didn't finish" over "might run twice". + +`interrupted` rows are counted in `queueStats()` and `getStats()` alongside `pending` / `processing` / `completed` / `failed`, so you can alert on them: + +```cfm {test:compile} title="watch for interrupted work" +component extends="Controller" { + function metrics() { + var jobs = new wheels.Job(); + var stats = jobs.queueStats(); + // {pending, processing, completed, failed, interrupted, total} + renderWith(data = stats); + } +} +``` + +An `interrupted` row is a job that may not have finished and deliberately won't be retried, so treat a non-zero `interrupted` count as something to look at — decide per entry whether the work needs re-queuing by hand. Like `failed` rows, `interrupted` rows persist (`purgeCompleted()` only touches `completed` rows). Once you've dealt with them, clear the old ones with `purge()`: + +```cfm {test:compile} title="prune interrupted rows" +new wheels.JobWorker().purge(status = "interrupted", days = 30); // interrupted rows older than 30 days +``` + +`purge(status, days, queue)` accepts `"completed"`, `"failed"`, or `"interrupted"`, so it's also how you prune failed rows once you've extracted what you need from `lastError`. + +## Lifecycle hooks and failure handling + +A job can hook its own lifecycle, and the application can hear every failure in one place — which matters across servers, where a job you enqueued runs on a machine you aren't watching. + +```cfm {test:compile} title="app/jobs/ChargeCardJob.cfc" +component extends="wheels.Job" { + public void function perform(struct data = {}) { + var order = model("Order").findByKey(arguments.data.orderId); + order.chargeCard(); + } + + // Best-effort, after the outcome is recorded: + public void function onSuccess(any result) { /* ... */ } + public void function onFailure(any error, numeric attempt, boolean isFinal) { + if (arguments.isFinal) notifyBilling(arguments.error); // gave up for good + } +} +``` + +Per-job hooks, all optional: + +- **`beforePerform(data)`** and **`afterPerform(data, result)`** run *as part of the work* — inside `perform()`'s timeout, on the same attempt. A throw from either **fails the attempt**, like a throw from `perform()`. Use them for setup and teardown that must succeed for the job to count. +- **`onSuccess(result)`** runs once the completion is recorded, and **`onFailure(error, attempt, isFinal)`** after each recorded failure. Both are **best-effort**: a throw is logged, not propagated, and never changes the recorded outcome. + +`isFinal` is `true` when the job won't be tried again — the last retry failed, **or** a reaper gave up on it: a reaped last attempt, and a non-idempotent job the reaper ends as `interrupted` (see [Idempotency](#idempotency-and-thisidempotent)), both fire `onFailure(..., isFinal = true)`. So `onFailure` with `isFinal` is a reliable "this job is done trying" signal even when the giving-up happened on another server's reaper, not in `perform()`. + +For centralized alerting, set an **application-wide failure hook** that hears every job's failures: + +```cfm {test:compile} title="config/settings.cfm" +set(jobsOnFailure = "app.jobs.FailureNotifier.report"); +``` + +It names `component.path.method`. The method is invoked with a single **named** argument, `event`, so it must declare that parameter: + +```cfm {test:compile} title="app/jobs/FailureNotifier.cfc" +component { + public void function report(required struct event) { + // event = {jobId, jobClass, queue, attempt, maxRetries, isFinal, error} + if (arguments.event.isFinal) alert("Job #arguments.event.jobClass# gave up: #arguments.event.error.message#"); + } +} +``` + +The `event` struct deliberately carries **no job `data`** (that can hold secrets), so it's safe to route to logs or an alerting service. + +### The stored result + +`perform()`'s return value is stored on the row, in a `wheels_jobs.result` column, and handed to `onSuccess(result)`. A simple value is stored as-is; a struct or array is stored as JSON. It's capped — 4000 characters on SQL Server (`NVARCHAR`), 4000 **UTF-8 bytes** on the other databases (a safe bound; Oracle's `VARCHAR2` counts bytes) — truncated on a character boundary with a visible marker if it's longer, so return a small summary, not a large payload. The `result` column is added by the same guarded auto-`ALTER` as the other 4.2 columns; without it, jobs run exactly as before and nothing is stored. + +## Cap how many jobs a server runs at once + +By default a server runs as many jobs as its workers claim. To bound the load one server puts on a shared database or an external API, set a per-host cap: + +```cfm {test:compile} title="config/settings.cfm" +set(jobsMaxConcurrentPerHost = 4); // this server runs at most 4 jobs at once; 0 (default) = no cap +``` + +The cap is enforced by every runner that goes through the worker — the CLI worker (`wheels jobs work`), `JobRunner.tick()`, and `processQueue()` (which runs through the worker as of [#4495](https://github.com/wheels-dev/wheels/issues/4495)). A server at its cap claims nothing new until a running job finishes; other servers keep working to their own caps. The count is per *host*, so the cap governs all the workers and pollers on one server together, not each process on its own. + + + +## Exclusive jobs and concurrency keys + +The per-host cap bounds *how many* jobs a server runs. A different need is making sure two runs of a *particular* job never overlap across the whole fleet — a nightly billing run that must not double, a per-account sync that mustn't race itself. Declare that in the job class: + +```cfm {test:compile} title="app/jobs/NightlyBillingJob.cfc" +component extends="wheels.Job" { + function config() { + super.config(); + this.exclusive = true; // at most one run of this class at a time, across all servers + } + public void function perform(struct data = {}) { + runBilling(); + } +} +``` + +- **`this.exclusive = true`** (default `false`) — at most one run of this job class at a time, on any server. While a run holds it, another server that claims the same class puts the job **back to `pending`** (its attempt is handed back, not spent) and tries again once the lease frees. +- **`this.concurrencyKey = "..."`** — instead of locking the whole class, share one lease among jobs with the same key. Two jobs with different keys run concurrently; two with the same key don't. +- **`concurrencyKeyFor(struct data)`** — a key computed from the job's data, for per-record exclusion. It **wins over** `this.concurrencyKey` when both are set: + +```cfm {test:compile} title="app/jobs/SyncAccountJob.cfc" +component extends="wheels.Job" { + public string function concurrencyKeyFor(struct data = {}) { + return "account:" & arguments.data.accountId; // one sync per account at a time + } + public void function perform(struct data = {}) { + syncAccount(arguments.data.accountId); + } +} +``` + +The lease lives in a `wheels_job_locks` table (auto-created on first use, never inside a transaction), owned by the running attempt's claim token, and lasts as long as the attempt's reap window — the timeout it was claimed with plus the reap grace — so it covers the run. A long run that [`heartbeat()`](#record-liveness-with-heartbeat)s renews the lease on each beat, to about twice the reclaim grace, so a job that stops beating is reclaimed after the grace while its old lease lingers about one grace more — a retry that finds it still held is deferred without spending an attempt. If a run does lose its lease before it finishes — taken over after expiry — that's logged and counted (as `leasesLost` in a `processQueue()` result), and the run is **not** failed for it; make the work idempotent so a rare overlap is harmless. + + + +## Running the queue on each server + +[Background Jobs](/v4-2-0/digging-deeper/background-jobs/#process-the-queue) covers the two ways to process the queue. Across servers, run one of them on **each** server; they coordinate through the shared table. + +- **The CLI worker** — `wheels jobs work` under a process supervisor (systemd, Docker's restart policy, a Kamal accessory). When it loses its app server it exits non-zero and the supervisor restarts it. Run one (or several) per app server. This is the simplest option and the one to reach for first. +- **A scheduled `processQueue()`** — call `processQueue()` from a CFML scheduled task or a cron-invoked script on each server, at whatever cadence your latency allows. No long-lived process to supervise. As of [#4495](https://github.com/wheels-dev/wheels/issues/4495) it runs through the worker, so it reaps stale jobs first, honours the per-host cap, and stops starting new jobs on a draining host — the same safety as the CLI worker. (It doesn't refresh the server's `lastSeenAt` / code version in the host registry the way `tick()` does, but its running count and drain state are read live, so `wheels jobs status` and `wheels jobs drain --wait` still see its in-flight jobs.) +- **`JobRunner.tick()`** — one call that registers the server in the host registry, reaps stale jobs, and runs a bounded batch, all honouring the per-host cap and drain flag. It's the right building block for a scheduled task, because it keeps the host registry (and so `wheels jobs status` and drain-aware deploys) current: + +```cfm {test:compile} title="a scheduled task on each server" +// Run every minute from a CFML scheduled task or a cron-invoked script. +new wheels.JobRunner().tick( + queues = "default,mailers", // empty = all queues + maxConcurrent = 2, // -1 (default) uses set(jobsMaxConcurrentPerHost); 0 = no cap + timeout = 300, // seconds a single job may run + maxJobs = 5 // most jobs to run in this tick +); +``` + +`tick()` runs its jobs synchronously, one after another, and does nothing beyond the cap or while the host is draining. Run it as often as your latency target wants; each call does a bounded amount of work and returns. + +### Drive it over HTTP: the tick route + +When your scheduler can only make an HTTP request — an engine scheduled task, a cron `curl`, a systemd timer — point it at `/wheels/jobs/tick`. Each call runs one `JobRunner.tick()` on that server with its defaults — so it reaps, enqueues any due schedules, and runs **at most one job** (capped at 300 s) — and returns a JSON summary. There's no request parameter to raise either limit yet ([#4520](https://github.com/wheels-dev/wheels/issues/4520) will make the route's batch size and timeout configurable), so one call a minute is one job a minute per server: fine for light, steady queues, but for volume run `wheels jobs work` (a continuous loop) or call `JobRunner.tick(maxJobs = n, timeout = …)` from CFML instead. Turn the route on by setting a token: + +```cfm {test:compile} title="config/settings.cfm" +set(jobsRunnerToken = "a-long-random-secret"); // the route is 404 until this is set +``` + +Then hit it with the token in a header, once a minute, from each server: + +```bash title="a cron entry / systemd timer" +curl -fsS -X POST http://127.0.0.1:8080/wheels/jobs/tick -H "X-Wheels-Jobs-Token: a-long-random-secret" +``` + +The **token is the boundary**, not the caller's address: + +- The route returns **404** unless `jobsRunnerToken` is set, and **403** without the right token (compared in constant time). The token goes in the `X-Wheels-Jobs-Token` header; `?token=` is accepted only with `set(jobsRunnerTokenInQuery = true)`, for a scheduler that can't set headers. +- A request carrying **any forwarding header** (`X-Forwarded-*`, `Forwarded`, `X-Real-IP`, `CF-Connecting-IP`, …) is refused — call it on the server itself (localhost), not through your proxy, so a public request can't reach it. +- It's answered before routing and the public-component gate, so it doesn't depend on your routes or middleware. An overlapping tick on the same server is skipped. `GET` and `POST` are both accepted. + +`wheels jobs tick` runs one tick from the CLI, for a cron job that has the `wheels` binary instead of an HTTP client. + +## Recurring schedules + +Everything above runs a job *once*. For work that recurs — a nightly digest, a health check every fifteen minutes — declare a schedule. On several servers this is where deduplication would normally bite (every server's cron fires the same minute); the scheduler handles it for you, producing **one job per time slot** no matter how many servers run it. + +Declare schedules in `config/schedules.cfm` with a small DSL (`wheels new` doesn't create the file — add it when you need one): + +```cfm title="config/schedules.cfm" +schedule("digest") + .job("DigestJob") + .cron("0 7 * * MON") + .timezone("America/New_York"); + +schedule("health-check") + .job("HealthCheckJob") + .every(15, "minutes") + .queue("maintenance"); +``` + +- **`.job(class)`** — the job class under `app/jobs/`, as for `enqueue`. +- **`.cron(expression)`** — a standard 5-field cron expression (`minute hour day-of-month month day-of-week`), read in the schedule's time zone. Lists, ranges, steps (`*/5`), and names (`MON`, `JAN`) work, as do the `@hourly` … `@yearly` aliases. Seconds and Quartz extensions are refused. +- **`.every(count, "minutes" | "hours")`** — a fixed interval, aligned to the epoch. For a daily or longer cadence, use `.cron()` instead. +- **`.timezone(zone)`** — the IANA zone `.cron()` is read in (default `UTC`), e.g. `Europe/London`. +- **`.catchUp(mode)`**, **`.catchUpWindow(seconds)`**, **`.queue()`**, **`.priority()`**, **`.data(struct)`**, **`.enabled(boolean)`** — tune catch-up (below), and set the same options you'd pass to `enqueue`. + +### Running the schedules + +**A running worker checks schedules for you.** `wheels jobs work`, `processQueue()`, and `JobRunner.tick()` each enqueue any due slots as they poll — at most once every `jobsScheduleCheckSeconds` (default `15`, `0` turns it off) per application. So once a schedule is in `config/schedules.cfm` and *anything* is draining the queue, it just runs; there's no separate scheduler to stand up. The check only runs when the schedules table exists, never throws (a failed check leaves the poll going), and `tick()`'s result reports how many slots it `scheduled`. + +Each due slot is enqueued with a `uniqueKey` of `":"`, so every server's worker can check and you still get **exactly one job per slot** — the duplicates are counted, not run (this rests on the [`uniqueKey`](/v4-2-0/digging-deeper/background-jobs/#enqueue-at-most-once-with-uniquekey) dedupe). A **draining** server still enqueues due slots; they run on another server. The enqueued jobs are then processed like any other. + +You only call `enqueueDue()` yourself when **nothing** is polling the queue — a box dedicated to scheduling, say: + +```cfm {test:compile} title="only needed when no worker polls this queue" +new wheels.JobScheduler().enqueueDue(); +// {checked, enqueued, duplicates, skipped, errors} +``` + +### Catch-up: missed slots + +If nothing calls `enqueueDue()` for a while — a deploy, an outage — slots are missed. `catchUp` decides what happens when it runs again: + +- **`"latest"`** (the default) — run only the **newest** missed slot, and only if it's within `catchUpWindow` (default 3600 seconds). Older missed slots are skipped. This is what you want for an idempotent digest: you get one run, not a backlog. +- **`"none"`** — run a slot only if it's current; never a missed one. + +Either way, a slot at or before the last one already enqueued never runs again, and a brand-new schedule starts from *now* — it doesn't fire slots from before it existed. Keep `catchUpWindow` below your [purge retention](#idempotency-and-thisidempotent) so a caught-up slot's `uniqueKey` row still exists to dedupe against. + +### Where schedules live + +Schedules are rows in a `wheels_job_schedules` table (auto-created on first use, never inside a transaction). `config/schedules.cfm` is synced into it at application start as *code* schedules: edit the file and restart, and the table follows. A code schedule you remove from the file is **disabled, not deleted**, so its history survives. An app can also insert its own rows at runtime (for example from an admin page) as *db* schedules; the start-up sync never touches those, and on a name clash the db row wins. + + + +## Drain a server before a deploy + +The problem a rolling deploy creates: you need to stop a server, but it may be running jobs. Killing it mid-job strands that work (it's reaped and retried elsewhere, but only after its timeout). `wheels jobs work --stop-when-empty` stops one worker loop, not the server, and not any other worker on it. **Draining** is the clean answer: mark the server so it starts nothing new, let its running jobs finish, then stop it. + + + +1. **Drain the server, and wait for it to go quiet:** + + ```bash title="on the server you're about to deploy" + wheels jobs drain --wait + ``` + + `drain` marks this server draining in the host registry: it starts no new jobs, while jobs already running finish and stale-job reaping continues. `--wait` then polls until the server has no running jobs and exits `0` — at which point it's safe to stop. If jobs are still running when the wait runs out (`--wait` alone allows 600 seconds; `--wait=120` sets your own limit), it exits non-zero so your deploy script can wait longer or investigate rather than stop mid-job. + +2. **Deploy** — stop the server, roll out the new build, start it. + +3. **Resume**, so the server takes jobs again: + + ```bash title="after the new build is up" + wheels jobs resume + ``` + + + +A drain lifts itself after `--expires=` (default 3600) even if nothing resumes it, so a deploy that dies between `drain` and `resume` doesn't leave the server idle forever. Pass `--expires=0` to drain until an explicit `resume`. The same actions are available in CFML for a scripted deploy — `new wheels.JobRunner().drain(expiresInSeconds = 600)` and `.resume()` — and `status()` (or `wheels jobs status`) reports a server's running count, cap, and drain state from its registry row. + + + +## The shared schema + +The jobs table and its auxiliary tables [auto-create on first use](/v4-2-0/digging-deeper/background-jobs/#the-jobs-table), and the multi-server columns are added to an existing table the same way — a guarded `ALTER`, run once, outside any transaction. That's convenient on a single server. **On several servers sharing a production database, put the schema under your own control instead**, for two reasons: an auto-`ALTER` runs when the first upgraded server happens to poll, against a live, in-use table (on PostgreSQL and CockroachDB especially, you want that DDL on *your* schedule, not at an arbitrary moment under load); and if the application's database user lacks `ALTER` privilege — a common production hardening — the auto-upgrade can't run at all, and the features that need the columns quietly fall back (unfenced claims, `updatedAt` heartbeats). + +Two tools make that clean: + +- **`wheels jobs install`** writes a migration — `app/migrator/migrations/_CreateWheelsJobTables.cfc` — from the running app's own schema. It creates the job tables, or tops up the ones Wheels already auto-created, idempotently. Nothing touches the database until `wheels migrate latest`, so the jobs schema applies like any other migration, in your maintenance window. This is the recommended path for a multi-server deploy. (The app's server must be running, so the migration matches its framework version.) Its `down()` is **destructive**: it drops all four job tables — including any Wheels auto-created and the migration adopted — and every job in them. +- **`set(jobsAutoCreateTables = false)`** turns off every runtime `CREATE` and `ALTER` of the job tables, so no server changes the schema under load. With it set, a missing **`wheels_jobs`** table (or **`wheels_job_locks`**, which exclusive jobs need) makes `enqueue()` and a worker poll throw `Wheels.Job.SchemaMissing` pointing you at `wheels jobs install`, instead of failing like an empty queue. A missing host-registry or schedules table isn't fatal — that feature just stays off, with a one-time log note. + +So on production: `wheels jobs install`, apply the migration, and set `jobsAutoCreateTables = false`. The columns and indexes it builds are below, for reference — or to apply by hand where a migrator isn't in play. + +The columns the multi-server guarantees use, on the `wheels_jobs` table: + +| Column | Type | Used for | +|---|---|---| +| `claimTimeout` | `INT` | Reaping a stale job on the *claiming* worker's timeout, so workers with different timeouts can share a queue | +| `claimToken` | `VARCHAR(36)` | Fencing — a terminal update only lands while the row still carries the attempt's token | +| `claimedBy` | `VARCHAR(128)` | The host that owns a claim (and the per-host cap and drain) | +| `heartbeatAt` | `DATETIME` / `TIMESTAMP` | `heartbeat()` stamping a long job's liveness | +| `uniqueKey` | `VARCHAR(255)` | Enqueue-once dedupe (nullable, with a unique index) | +| `result` | `VARCHAR(4000)` | `perform()`'s stored return value (`NVARCHAR(4000)` on SQL Server) | + +Adjust types per database. `heartbeatAt` is a timestamp column, so it's `DATETIME` on MySQL, SQL Server, and SQLite but **`TIMESTAMP` on PostgreSQL, CockroachDB, Oracle, and H2** (PostgreSQL and CockroachDB have no `DATETIME` type). SQL Server omits `COLUMN`; Oracle uses `VARCHAR2` for the string columns and the `ADD (col TYPE)` form. For MySQL (and SQLite, which ignores the exact spelling): + +```sql +ALTER TABLE wheels_jobs ADD COLUMN claimTimeout INT; +ALTER TABLE wheels_jobs ADD COLUMN claimToken VARCHAR(36); +ALTER TABLE wheels_jobs ADD COLUMN claimedBy VARCHAR(128); +ALTER TABLE wheels_jobs ADD COLUMN heartbeatAt DATETIME; -- TIMESTAMP on PostgreSQL / CockroachDB / Oracle / H2 +ALTER TABLE wheels_jobs ADD COLUMN uniqueKey VARCHAR(255); +ALTER TABLE wheels_jobs ADD COLUMN result VARCHAR(4000); -- NVARCHAR(4000) on SQL Server +UPDATE wheels_jobs SET uniqueKey = id; +CREATE UNIQUE INDEX idx_wjobs_unique_key ON wheels_jobs (uniqueKey); +``` + +On PostgreSQL and CockroachDB, write `ADD COLUMN heartbeatAt TIMESTAMP`. On SQL Server, make the unique index filtered — `CREATE UNIQUE INDEX idx_wjobs_unique_key ON wheels_jobs (uniqueKey) WHERE uniqueKey IS NOT NULL` — because SQL Server allows only one NULL in a plain unique index. The filtered index needs SQL Server **compatibility level 100 or higher**; below that it can't be built. On such a database, either raise the compatibility level, or create a *plain* unique index (no `WHERE`) — but only **once every server runs 4.2** (and so always writes a key), because a plain unique index allows just one NULL and a server still on 4.1 writes NULL keys that would collide. When the framework's own upgrade can't build the index, its warning and the `Wheels.Job.UniqueKeyUnavailable` error now name the failed step and give the fix for your exact database and SQL Server level. On a large, busy table, do the `uniqueKey` backfill and index build in a maintenance window: it touches every row. See [The jobs table](/v4-2-0/digging-deeper/background-jobs/#the-jobs-table) for the full column list and the per-database notes. The auxiliary tables — `wheels_job_hosts` (the host registry), `wheels_job_locks` (exclusive-job leases), and `wheels_job_schedules` (recurring schedules) — have no backfill and are cheap to auto-create, so you can leave those to the framework; only the `wheels_jobs` column changes above are worth applying by hand up front. + +The hand-written DDL above is the fallback — for a DBA applying it directly, or a database with no migrator. Where you can run migrations, prefer `wheels jobs install` ([#4499](https://github.com/wheels-dev/wheels/issues/4499)): it emits the same schema, per database, with a `down()`, and leaves nothing to get wrong by hand. + +## Clocks and skew + +Job timestamps — reap windows, heartbeats, `runAt` comparisons — are kept on **one UTC clock: the database's** ([#4486](https://github.com/wheels-dev/wheels/issues/4486)). `wheels.JobClock` reads the database's UTC time (`SYSUTCDATETIME`, `UTC_TIMESTAMP`, `clock_timestamp()`, `SYS_EXTRACT_UTC(…)`) at most once a minute per application and tracks the skew against the server's own UTC; every timestamp the jobs code writes or compares is corrected by that skew, and the comparisons still run in SQL. So two servers with clocks minutes apart no longer disagree about whether a job is stale or due — which removes the main reason app-server clock skew used to matter. (A clock roughly right is still worth keeping; a failed clock read just reuses the last good skew, logging once.) + +Two things to know: + +- **H2 and SQLite use the server's own UTC**, not a database clock — they're single-host databases, so there's nothing to reconcile. On **H2**, timestamp arithmetic runs in the JVM's time zone, so **run its JVM in UTC** (`-Duser.timezone=UTC`); a non-UTC JVM shifts stored job times. SQLite is unaffected. +- **The CFML API stays in local time.** `enqueueAt(runAt = …)` takes a local time and converts it to UTC for storage, and `JobRunner.status()` and the queue stats convert back, so your code and dashboards read local time as before. Only the stored column values are UTC. + + + +## Upgrading from 4.1 + +All of this is additive: existing jobs keep working, the new settings default to today's behaviour, and the schema is added for you on first use. The one rule that matters during a rolling upgrade is that **fencing only holds once every worker is on 4.2** — so expect the old recovery behaviour until the last 4.1 worker is gone. The job-related entries on the [4.1 → 4.2 upgrade guide](/v4-2-0/upgrading/4x-1-to-4x-2/) cover each change, the columns added, and what fails safe if skipped. + +## Related guides + + + + + + + diff --git a/web/sites/guides/src/content/docs/v4-2-0/upgrading/4x-1-to-4x-2.mdx b/web/sites/guides/src/content/docs/v4-2-0/upgrading/4x-1-to-4x-2.mdx index 6fb4b570fd..c2b2fc04f6 100644 --- a/web/sites/guides/src/content/docs/v4-2-0/upgrading/4x-1-to-4x-2.mdx +++ b/web/sites/guides/src/content/docs/v4-2-0/upgrading/4x-1-to-4x-2.mdx @@ -381,7 +381,55 @@ Every job claim now writes a fresh `claimToken` (and the claiming host, `claimed `enqueue()`, `enqueueIn()` and `enqueueAt()` accept `uniqueKey`, and only one job is written per key ([#4478](https://github.com/wheels-dev/wheels/issues/4478)). A duplicate returns `{enqueued: false, duplicate: true}` with the existing job's `id`. Every `enqueue()` result now includes `enqueued` and `duplicate`. Jobs enqueued without a key behave as before. -The `wheels_jobs` table gains a nullable `uniqueKey` column with a unique index, filtered to non-NULL keys on SQL Server. On an existing table, the first worker poll or `enqueue(uniqueKey=...)` after the upgrade adds the column, sets it to each row's `id`, and builds the index under the migration lock. It doesn't wait if another instance holds the lock. Servers still on 4.1 keep enqueuing during a rolling upgrade: their rows get a NULL key, which never collides. On a large table, or where the database user can't `ALTER`, apply the change yourself first; the statements are in [The jobs table](/v4-2-0/digging-deeper/background-jobs/#the-jobs-table). +The `wheels_jobs` table gains a nullable `uniqueKey` column with a unique index, filtered to non-NULL keys on SQL Server. On an existing table, the first worker poll or `enqueue(uniqueKey=...)` after the upgrade adds the column, sets it to each row's `id`, and builds the index under the migration lock. It doesn't wait if another instance holds the lock. Servers still on 4.1 keep enqueuing during a rolling upgrade: their rows get a NULL key, which never collides. On a large table, or where the database user can't `ALTER`, apply the change yourself first; the statements are in [The jobs table](/v4-2-0/digging-deeper/background-jobs/#the-jobs-table). On a SQL Server database below **compatibility level 100**, the filtered index can't be built; the upgrade's warning and the `Wheels.Job.UniqueKeyUnavailable` error now name the failed step and the fix for your level — raise the level, or (once every server runs 4.2) use a plain unique index ([#4489](https://github.com/wheels-dev/wheels/issues/4489)). + +### Jobs can call `heartbeat()`, and `this.idempotent` gates retry after a reap + +A job can now call `heartbeat()` from inside `perform()` to record that this attempt is still alive ([#4477](https://github.com/wheels-dev/wheels/issues/4477)). It stamps a new `wheels_jobs.heartbeatAt` column (added on first use, like `claimTimeout`; without it, `updatedAt` is stamped instead) for the current claim only, and throws `Wheels.Job.Fenced` if the claim was already reaped and reissued. It does **not** extend the attempt's `timeout` — the `timeout` is a hard cap per attempt, and `perform()` is stopped when it elapses however many times the job has heartbeated. Jobs also gain `this.idempotent`, **`true` by default**, which keeps today's at-least-once retry after a reap. A class that sets `this.idempotent = false` is **not** re-run after a reap: the reaped attempt ends in a new status, `interrupted`, counted in `queueStats()` and `getStats()` and prunable with `purge(status="interrupted")`. Nothing changes unless you add `heartbeat()` or set `this.idempotent = false`. See [Running Jobs on Multiple Servers](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#record-liveness-with-heartbeat). + +### A per-host concurrency cap, a host registry, and drain/resume for deploys + +Three additions let several servers share one queue under control ([#4480](https://github.com/wheels-dev/wheels/issues/4480)), all opt-in and defaulting to today's behaviour: + +- `set(jobsMaxConcurrentPerHost = n)` caps how many jobs one server runs at once (default `0`: no cap). The cap is per host — the name a worker records in `claimedBy`, which `set(jobsHostName = "...")` overrides when several app servers share a machine. +- A `wheels_job_hosts` registry table (one row per host; auto-created on first use, never inside a transaction) tracks each server's running count, cap, drain state, and code version. Set a release tag or git SHA with `set(jobsCodeVersion = "...")` to see which build each server runs during a rollout. +- `new wheels.JobRunner()` exposes `tick()` (register, reap, run a bounded batch), `drain(expiresInSeconds)` / `resume()`, and `status()`. A draining host starts nothing new while its running jobs finish. + +### `wheels jobs drain` and `wheels jobs resume` for rolling deploys + +`wheels jobs drain` marks this server draining so it starts no new jobs while its running jobs finish; `wheels jobs resume` lifts it ([#4480](https://github.com/wheels-dev/wheels/issues/4480)). `wheels jobs drain --wait` polls until the server has no running jobs and exits non-zero if any remain when the wait runs out, so a deploy script can stop the server safely. `--expires=` (default 3600, `0` = until resumed) sets when the drain lifts itself if nothing resumes it. `wheels jobs status` now also reports this server's running count, cap, and drain state. See [Drain a server before a deploy](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#drain-a-server-before-a-deploy). + +### Recurring job schedules + +Jobs can now run on a recurring, cron-style schedule, declared in a new `config/schedules.cfm` ([#4481](https://github.com/wheels-dev/wheels/issues/4481)): `schedule("digest").job("DigestJob").cron("0 7 * * MON").timezone("America/New_York")`, or `.every(15, "minutes")`. Schedules sync into a new `wheels_job_schedules` table at application start (auto-created, never inside a transaction). A running worker checks them for you — `wheels jobs work`, `processQueue()`, and `JobRunner.tick()` each enqueue due slots as they poll, at most once every `jobsScheduleCheckSeconds` (default 15, 0 = off) per application ([#4509](https://github.com/wheels-dev/wheels/issues/4509)) — so once anything drains the queue, a schedule just runs; call `new wheels.JobScheduler().enqueueDue()` yourself only when nothing polls the queue. Each slot is enqueued with a `uniqueKey`, so every server checking produces one job per slot. Nothing changes for an existing app: there's no schedule until you add `config/schedules.cfm` or call the scheduler. `wheels new` does not create the file. See [Recurring schedules](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#recurring-schedules). + +### Exclusive jobs and concurrency keys + +A job can now hold a cross-server lease while it runs ([#4496](https://github.com/wheels-dev/wheels/issues/4496)): `this.exclusive = true` lets at most one run of the class happen at a time on any server, and `this.concurrencyKey` or a `concurrencyKeyFor(struct data)` method (the method wins) scopes that to a key. A run that finds its lease held is put back to `pending` (its attempt isn't spent) and retries when the lease frees; `heartbeat()` renews an exclusive run's lease. The lease lives in a new `wheels_job_locks` table, auto-created on first use outside any transaction. `processQueue()`'s result now includes `leasesLost`, and its `skipped` count now counts lease-deferred runs. Nothing changes unless a job opts in — an existing job holds no lease. See [Exclusive jobs and concurrency keys](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#exclusive-jobs-and-concurrency-keys). + +### Job lifecycle hooks, an app-wide failure hook, and a stored result + +Jobs gain optional lifecycle hooks ([#4498](https://github.com/wheels-dev/wheels/issues/4498)): `beforePerform(data)` and `afterPerform(data, result)` run as part of the work (a throw fails the attempt), while `onSuccess(result)` and `onFailure(error, attempt, isFinal)` run best-effort after the outcome is recorded — `isFinal` is true when the job won't be retried, including a reaped last attempt or an `interrupted` non-idempotent job. `set(jobsOnFailure = "component.path.method")` adds an application-wide failure hook, whose method takes a single named `event` argument — `{jobId, jobClass, queue, attempt, maxRetries, isFinal, error}`, with no job data. `perform()`'s return value is stored in a new `wheels_jobs.result` column (`VARCHAR(4000)`, `NVARCHAR(4000)` on SQL Server), added by the guarded auto-`ALTER`; without the column, jobs run as before and nothing is stored. Nothing fires unless a job defines a hook or you set `jobsOnFailure`. See [Lifecycle hooks and failure handling](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#lifecycle-hooks-and-failure-handling). + +### Heartbeating jobs are reclaimed on a heartbeat grace + +A job that calls `heartbeat()` is now reclaimed once its beats stop for a **heartbeat grace**, even inside its `timeout`, instead of waiting for twice the timeout ([#4502](https://github.com/wheels-dev/wheels/issues/4502)). The grace is `set(jobsHeartbeatGraceSeconds = n)` (default `300`, floor `30`, `0` turns it off), or `this.heartbeatGrace` per class. A job that never heartbeats keeps the old claim window (`timeout` plus the reap grace), and `timeout` is still the hard cap on a single attempt. An exclusive job's heartbeat renews its lease by the same grace. Nothing changes unless a job heartbeats. See [Record liveness with `heartbeat()`](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#record-liveness-with-heartbeat). + +### `wheels jobs work` runs each job for its own timeout + +The `wheels jobs work` worker used to cap every job at 300 seconds, whatever its `this.timeout` ([#4517](https://github.com/wheels-dev/wheels/issues/4517)). It now claims and runs each job with its own class timeout, as `processQueue()` does, so a job with `this.timeout = 900` gets its full fifteen minutes. `wheels jobs work --job-timeout=` caps that (not `--timeout`, which on some CLI runtimes is the runtime's own root option and would stop the whole worker). A job class with a long `timeout` that relied on being cut off at 300 now runs longer; set `--job-timeout` or lower its `this.timeout`. + +### Installing the job schema as a migration + +`wheels jobs install` writes a migration (`app/migrator/migrations/_CreateWheelsJobTables.cfc`) that creates the job tables — `wheels_jobs`, `wheels_job_hosts`, `wheels_job_locks`, `wheels_job_schedules` — or tops up the ones Wheels auto-created, idempotently, generated from the running app's own schema ([#4499](https://github.com/wheels-dev/wheels/issues/4499)). `set(jobsAutoCreateTables = false)` then stops every runtime `CREATE`/`ALTER` of those tables, so no server changes the schema under load; with it set and a table missing, `enqueue()` and a worker poll throw `Wheels.Job.SchemaMissing` with the install guidance. The default is `true` (today's auto-create behaviour), so nothing changes until you opt out. On a multi-server production database, run `wheels jobs install`, apply the migration, and set `jobsAutoCreateTables = false`. See [The shared schema](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#the-shared-schema). + +### A token-gated `/wheels/jobs/tick` route + +A scheduler that can only make an HTTP request can now run jobs inside the app server: each `POST /wheels/jobs/tick` runs one `JobRunner.tick()` and returns a JSON summary ([#4482](https://github.com/wheels-dev/wheels/issues/4482)). The route is **off (404) until `set(jobsRunnerToken = "...")`** is set; it then needs that token in the `X-Wheels-Jobs-Token` header (or `?token=` with `set(jobsRunnerTokenInQuery = true)`), compared in constant time, and refuses any request carrying a forwarding header — so call it on the server itself, not through a proxy. `wheels jobs tick` does one tick from the CLI. Nothing is exposed unless you set the token. See [Drive it over HTTP: the tick route](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#drive-it-over-http-the-tick-route). + +### Job timestamps are stored in UTC on the database's clock + +Job times are now kept on the database's UTC clock rather than each server's local clock, so servers with skewed clocks or different time zones no longer disagree about whether a job is stale or due ([#4486](https://github.com/wheels-dev/wheels/issues/4486)). `wheels.JobClock` reads the database's UTC time about once a minute and corrects every job timestamp by the skew. The **CFML API is unchanged** — it stays in local time: `enqueueAt()` converts to UTC for storage, and `JobRunner.status()` and the queue stats convert back. Two things to check: on **H2**, run the JVM in UTC (`-Duser.timezone=UTC`), since H2 does timestamp arithmetic in the JVM zone (SQLite and the client/server databases are unaffected); and a database **upgraded from before 4.2 holds local-time rows**. On a non-UTC server, either upgrade once the queue is empty (no `pending`/`processing` jobs) and leave the offset at `0`, or set `this.convertLocalOffsetMinutes` in the `wheels jobs install` migration to the old servers' offset from UTC (`-300` for UTC-5, `60` for UTC+1) to convert existing rows once, in a single transaction with a marker so it's safe to re-run. Do neither and the old rows are misread: a delayed job runs early or late by the offset, and on a server behind UTC an in-flight job looks stale and is reclaimed. SQLite rows and servers already on UTC need neither. See [Clocks and skew](/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/#clocks-and-skew). ### CSRF accepts the `X-CSRF-Token` header on any request diff --git a/web/sites/guides/src/sidebars/v4-2-0.json b/web/sites/guides/src/sidebars/v4-2-0.json index 701babb18a..0476038bf3 100644 --- a/web/sites/guides/src/sidebars/v4-2-0.json +++ b/web/sites/guides/src/sidebars/v4-2-0.json @@ -88,6 +88,7 @@ { "label": "Authorization & Filters", "link": "/v4-2-0/digging-deeper/authorization-and-filters/" }, { "label": "Authorization Policies", "link": "/v4-2-0/digging-deeper/authorization-policies/" }, { "label": "Background Jobs", "link": "/v4-2-0/digging-deeper/background-jobs/" }, + { "label": "Running Jobs on Multiple Servers", "link": "/v4-2-0/digging-deeper/running-jobs-on-multiple-servers/" }, { "label": "Caching", "link": "/v4-2-0/digging-deeper/caching/" }, { "label": "Sending Email", "link": "/v4-2-0/digging-deeper/sending-email/" }, { "label": "File Uploads & Downloads", "link": "/v4-2-0/digging-deeper/file-uploads-and-downloads/" },