[bpamiri]

Closes #4484

Problem

The jobs schema is created and altered automatically at runtime, with no migration. Apps with migration discipline, and their DBAs, can't own or review it. On several servers, the automatic ALTERs run on every host at startup.

Change

wheels.JobSchema: the schema, defined once

Each job table is listed once, with:

• its columns in creation order: name, type (string / integer / text / datetime), limit, nullability and default;
• its indexes: columns, unique, and whether the index is optional.

Auto-create builds its DDL from it. Job.$ensureJobTable() and the column upgrades (claimTimeout, claimToken/claimedBy, uniqueKey) now build their CREATE TABLE, ALTER TABLE ... ADD and CREATE INDEX from the schema:

• Types: the same per-database types as before.
• Oracle claimTimeout: the claimTimeout upgrade on Oracle now adds INT, which Oracle stores as NUMBER(38). That is the type CREATE TABLE always used; the upgrade used to add NUMBER(10). Both hold any timeout, and existing columns aren't changed.
• Indexes: each index is now built in its own try. Before, one failing optional index skipped the rest.

Catalog checks. It also answers hasTable / hasColumn / hasIndex from each database's catalog (information_schema, user_tab_columns / user_indexes, sys.indexes, pg_indexes, sqlite_master / pragma_table_info). It never runs a failing probe query, so the checks are safe inside a migration's transaction: a failed statement aborts a PostgreSQL transaction.

wheels jobs install

It writes app/migrator/migrations/<timestamp>_CreateWheelsJobTables.cfc.

How it's generated. The source comes from the running app's JobSchema (bridge command jobsInstallSource, read-only), so it matches the framework version the app runs. Nothing touches the database until wheels migrate latest.

What it contains. The migrator DSL (createTable, the column helpers, addColumn, addIndex), so the same file runs on every supported database. A dev-SQLite / prod-PostgreSQL app runs one file. execute() is used only where the DSL can't express the step:

• SQL Server's filtered unique index (WHERE uniqueKey IS NOT NULL);
• the uniqueKey = id backfill when it adds that column, as the automatic upgrade does.

Existing installs. up() adopts tables that auto-create already built: it creates only missing tables, columns and indexes. Running it again changes nothing.

down(). It drops the job tables, including ones the migration adopted rather than created, and every job in them. That is what rolling back a create-table migration means; the generated file says so in its header.

Datasource. The migration's catalog checks run against the migrator's datasource and credentials ($migratorDataSource()), so a TenantMigrator checks each tenant's database.

Re-running the command. --force rewrites an existing jobs migration under the same file name, so a migration that already ran keeps its version. Without --force the command leaves it alone and says so.

jobsAutoCreateTables

set(jobsAutoCreateTables = false), for an app that ran the migration, stops every runtime CREATE and ALTER of the job tables.

• Missing table. Once the database catalog confirms the table is missing (a probe failing for another reason, such as a lost connection, surfaces as that error), enqueue and a worker's poll throw Wheels.Job.SchemaMissing, whose message points to wheels jobs install and wheels migrate latest. It is also logged once per application. A worker doesn't pass the poll off as an empty queue: wheels jobs work exits with the message.
• Missing uniqueKey. The enqueue(uniqueKey=) error says the same.
• Default. It is true, set with the other framework defaults: auto-create is unchanged for apps that don't opt out.

Why multi-server apps should use it (for the guide)

With auto-create, every server runs the jobs DDL at startup.

There is also a hypothesis, seen only in the test suite and not reproduced in a deployment. On PostgreSQL and CockroachDB, an automatic ALTER on a live table could make other servers' prepared job statements fail once per statement and connection (cached plan must not change result type).

Applying the schema once, through this migration, before deploying avoids both.

Tests

JobSchemaSpec (8):

• auto-create builds every column and index the schema lists, and the catalog checks answer correctly;
• the install migration, generated and run exactly as the command writes it, builds the same columns with the same nullability, plus every index, and the table works: a keyed enqueue de-duplicates;
• an older auto-created table without the fencing columns, uniqueKey or indexes is topped up, existing rows get their uniqueKey, and a second run is a no-op;
• up, down and up again through the Migrator (its transaction and lock). The migrated table has the same type family for every column (string / text / integer / datetime, read from the catalog) and the same nullability as auto-create's, down() drops it, and every index is back after the second up();
• with jobsAutoCreateTables = false, $ensureJobTable(), enqueue and a worker poll each throw Wheels.Job.SchemaMissing with the install guidance and create nothing;
• with jobsAutoCreateTables = false, a probe failure on a table the catalog says exists surfaces as its own error, not as a missing schema;
• with jobsAutoCreateTables = false, an old table isn't altered;
• each database's DDL.

JobsCommandSpec: install and --force parse; the command writes the server's source once; a re-run leaves it alone; --force rewrites it in place.

CliBridgeSpec: declares jobsInstallSource.

On develop:

• the new core specs error, because wheels.JobSchema doesn't exist;
• the jobsAutoCreateTables = false spec fails, because develop creates the table anyway;
• CliBridgeSpec fails, because there is no jobsInstallSource.

Local results:

• Full core suite, Lucee 7 + SQLite: 7147 pass, 0 fail, 0 error (582 bundles).
• Jobs specs, Lucee 7: green on SQLite, H2, MySQL, PostgreSQL, CockroachDB, SQL Server and Oracle; PostgreSQL and CockroachDB also green on a second run against the same database.
• Jobs specs, BoxLang: green on SQLite, PostgreSQL, CockroachDB, SQL Server and Oracle.
• Jobs specs, Adobe 2023: green on SQLite and MySQL.
• Jobs specs, Adobe 2025: green on SQLite, PostgreSQL and SQL Server.
• Migrator specs, Lucee 7 + SQLite: green.
• CLI-bridge core specs: green on Lucee 7. The CLI module's own suite (JobsCommandSpec) runs in PR CI.
• Complexity gates (core and CLI): pass.

Develop merge (4.2.0): every job table now in JobSchema

Merged develop with a merge commit. develop now has the whole series: #4490 heartbeat, #4491/#4492 hosts and drain, #4493 schedules, #4494, #4495, #4496 lease lock and #4498 hooks. This PR lands last, so JobSchema adopts every job table and column, and auto-create and install both cover them.

What JobSchema adds:

• wheels_jobs columns: heartbeatAt (feat(job): heartbeat() and idempotency-aware reaping #4490) and result (feat(job): lifecycle hooks, an app-wide failure hook, and a stored result #4498).
• Tables: wheels_job_hosts (feat(job): per-host concurrency cap, drain/resume and JobRunner.tick() #4491), wheels_job_schedules (feat(job): recurring schedules (cron and intervals) with one job per slot #4493) and wheels_job_locks (feat(job): exclusive jobs and concurrency keys on a shared lease lock #4496).
• Column types: decimal (whole numbers, for the epoch-millisecond columns), and unicodeString for result. unicodeString is NVARCHAR on SQL Server, whose VARCHAR mangles non-ASCII text.

Auto-create. Every auto-create path now builds its DDL from JobSchema: the wheels_jobs columns, hosts, schedules and locks. Each one also respects jobsAutoCreateTables:

• Locks. With it false, a missing locks table throws Wheels.Job.SchemaMissing when an exclusive job runs.
• Hosts and schedules. A missing hosts or schedules table logs once and disables only that feature (the cap/drain, or schedule enqueueing).

The generated migration. It covers all four tables, each with its own create-or-top-up block:

• NVARCHAR on SQL Server. The migration column helpers would make result a VARCHAR there, so the migration sets NVARCHAR with execute().
• Transactions. It runs with this.useTransaction = false. It is idempotent, and on Lucee + CockroachDB a multi-DDL step inside the migrator's transaction can fail under read_committed (fix(migrator): multi-DDL migration steps fail on Lucee + CockroachDB after a read_committed model transaction #4506).

CLI. The install verb, --force and the bridge command sit beside #4492's drain and resume.

Spec. JobSchemaSpec now compares auto-create and the install migration for every table: columns and nullability, and type families through the migrator round trip. On SQL Server it also checks that result is NVARCHAR.

Local results on this head:

• Full core suite, Lucee 7 + SQLite: 7233 pass, 0 fail, 0 error.
• Complexity gates (core and CLI): pass.

From the trial merge of the same job and CLI code:

• Jobs specs, Lucee 7: green on all seven databases. PostgreSQL and CockroachDB were also green on a second run against the same database (CockroachDB re-run after the useTransaction fix).
• Jobs specs, BoxLang: green on SQLite, PostgreSQL, CockroachDB, SQL Server and Oracle.
• Jobs specs, Adobe 2023: green on SQLite and MySQL.
• Jobs specs, Adobe 2025: green on SQLite, PostgreSQL and SQL Server.

The per-attempt history table (#4497) joins behind --attempts once it exists.

Docs

The guide section (wheels jobs install, jobsAutoCreateTables, the multi-server recommendation) goes into the 4.2 guide (#4485). Changelog fragment included.

🤖 Generated with Claude Code

[github-actions]

Wheels Test Results

36 965 tests   36 230 ✅  12m 17s ⏱️
 2 965 suites     735 💤
     5 files         0 ❌

Results for commit 11db051.

♻️ This comment has been updated with latest results.

[bpamiri]

Review: draft, head e805d6b. REQUEST CHANGES: two MEDIUM, both on items this change set out to cover. CI is fully green and the shared-schema design is sound. Not for merge before the 4.2.0 cut anyway.

MEDIUM

1. With jobsAutoCreateTables = false and no job table, the "run wheels jobs install" guidance only reaches the log (Job.cfc $ensureJobTable / enqueue / JobWorker).

• enqueue(). The INSERT fails, $ensureJobTable() logs missingSchemaMessage(...) to wheels_jobs.log and returns false, then $throwEnqueueFailed(error = e) throws Wheels.Job.EnqueueFailed. The exception carries the raw "table doesn't exist" database message, and its extendedInfo is about transactions and second datasources. So the developer is pointed at the wrong cause.
• Worker. processNext() re-runs the candidate SELECT, fails again, and returns skipped = true with the raw error on every poll, so wheels jobs work just looks idle.
• UniqueKeyUnavailable. With auto-create off, extendedInfo now prepends the install hint but goes on to say "The framework adds them on a worker poll or an enqueue…", which is no longer true.
• Fix:
  • when auto-create is off and the table/column is missing, throw (or return as the worker's error) a typed error whose message is missingSchemaMessage() (e.g. Wheels.Job.SchemaMissing);
  • drop the "framework adds them" sentence in that mode;
  • spec the enqueue error and the worker result.

2. The generated migration's down(), and up() through the migrator, are untested (JobSchemaSpec).

• The specs call migration.up() directly on an instance: no migrator, no $runMigrationStep transaction, no down().
• Running inside the migrator's transaction matters for this migration:
  • on PostgreSQL its catalog checks run after DDL in the same transaction;
  • on MySQL/Oracle the DDL commits implicitly.
• down() is never executed on any database.
• Fix: run the generated file through the migrator, up → down → up (as wheels migrate latest && down && up does), and assert the tables after each step. The catalog-compare spec also compares only column names and nullability. Comparing a type family per column (string/int/text/timestamp) would catch the migrator DSL and auto-create drifting (e.g. t.datetime vs DATETIME/DATETIME2/TIMESTAMP).

LOW

• The catalog checks may look at a different database from the one the migration changes.
  • In the generated migration, new wheels.JobSchema() defaults to application.wheels.dataSourceName and the default credentials.
  • The migration DSL runs on $migratorDataSource(), which a tenant migration sets to the tenant's datasource and credentials.
  • Example: under TenantMigrator, with the job tables already in the app database and missing from a tenant's, hasTable() says yes, up() creates nothing on the tenant, and the version is still recorded.
  • Pass the migrator's datasource (e.g. new wheels.JobSchema(datasource = $migratorDataSource()) in the generated source, or have JobSchema.init honour request.wheels.migratorDataSource).

INFO

• down() also drops tables the migration adopted (auto-created before it ran), together with any queued jobs in them. That's defensible for an install migration; worth one line in the generated docblock.
• jobsAutoCreateTables has no registered default (it's read as true when absent) and isn't in the settings docs or guides yet.
• claimTimeout's ALTER on Oracle is now INT (from columnType) rather than NUMBER(10). Both work. Noted as a deliberate unification.
• The spec writes the generated CFC under vendor/wheels/tests/_assets/jobs_install at runtime and removes it in afterAll. Engines sharing a bind mount write identical content, so this is fine.

Verified

• Auto-create parity.
  • createTableSql() produces the same column list, order, types, defaults and NOT NULLs per database as the old inline DDL.
  • Each index is now created separately. Previously, one failing optional index skipped the rest. A non-optional failure (uniqueKey) is still logged, and $recordUniqueKeyIndexVerified() is only set on success.
  • The ALTER builders go through addColumnSql: Oracle ADD (...), SQL Server ADD, others ADD COLUMN.
• Unique index. SQL Server's unique index is filtered WHERE uniqueKey IS NOT NULL, both in auto-create and in the generated migration (via execute). Elsewhere it's a plain addIndex(unique = true).
• Adoption. The generated up() adds only missing nullable columns, backfills uniqueKey = id, and adds missing indexes. Running it again is a no-op (specced on all 32 legs).
• Catalog queries. There's one per database, keyed on $detectDatabaseType(), with no failing probe SELECT. CockroachDB takes the postgresql branch (information_schema/pg_indexes, both supported).
• Auto-create off with a table present. $ensureJobTable() returns true with no column upgrades, and every $ensure* upgrade path returns early (specced).
• CLI.
  • install needs a running server, takes the source from the app's own framework version, and writes <timestamp>_CreateWheelsJobTables.cfc once.
  • --force rewrites the same file name, so a migration that already ran keeps its version.
  • JobsCommandSpec covers parsing and the write/rewrite.
  • The bridge command is read-only.

Compat matrix 37729374931: 32/32 Lucee/Adobe/BoxLang legs green, 0 fail / 0 error on every database. JobSchemaSpec 5/5 on every leg. RustCFML 7122/0/0.

PR CI: 20 pass, 0 fail, 2 skipped. Smoke: testing (Adobe 2023) failed once before any test ran (the container never answered); its single re-run passed.

[bpamiri]

Final review: head a74786d. APPROVE (ready). Still a draft for v4.3.0; it stays held until after the 4.2.0 cut.

Earlier findings, all resolved

• MEDIUM 1, clear errors with auto-create off.

  • With jobsAutoCreateTables = false and no job table, $ensureJobTable(), enqueue() and a worker poll now all throw Wheels.Job.SchemaMissing, whose message is the install guidance. It's logged once per app.
  • The worker's ensure wrapper rethrows it instead of reporting an empty queue.
  • A probe failure is only called a missing schema when the catalog agrees (JobSchema.hasTable()). Otherwise the probe runs again outside the try, so a lost connection or a permission error surfaces as itself.
  • Specs cover all three callers plus the mocked "table present, probe fails" case.

• MEDIUM 2, migration coverage. A new spec writes the generated migration into a scratch migrations directory and runs up → down → up through a real wheels.Migrator (its transaction and lock). After each step it asserts:

  • the table exists or doesn't;
  • catalog type families (string/text/integer/datetime) and nullability equal auto-create's;
  • every index is present after the second up.

  The spec reads $typeFamily() per database, including SQLite's string→text folding, H2 1.4's TYPE_NAME fallback and CockroachDB's bigint.

• LOW, migrator datasource. The generated up()/down() build JobSchema(datasource = $migratorDataSource(), credentials = $migratorDataSourceCredentials()), and type detection uses Job.$databaseTypeOf(ds). Under TenantMigrator, the checks now run on the database the migration changes.

• INFO items handled:

  • down() dropping adopted tables (and their jobs) is stated in the generated header, the PR and the changelog;
  • jobsAutoCreateTables = true is registered in onapplicationstart;
  • the Oracle INT (NUMBER(38)) vs NUMBER(10) difference is noted in columnType().

• Also fixed: Adobe's compile-time rejection of DirectoryCreate(dir, true).

Note (non-blocking)

Inside a PostgreSQL transaction, a failed probe aborts the transaction, so the hasTable() confirmation then errors with "current transaction is aborted" rather than SchemaMissing. With a missing table inside a transaction any path errors there, so the clearer message is simply unavailable in that one spot.

Evidence

• Full matrix 37729374931 at e805d6b: 32/32 JVM legs green on every database, plus RustCFML.
• Narrowed matrix 37734386995 at 10ddf9b (5 JVM engines × sqlite/mysql/postgres/sqlserver/oracle, plus RustCFML):
  • 25/25 legs green, 0 fail / 0 error, JobSchemaSpec 7/7 on every leg;
  • RustCFML 7124/0/0.
• Run 37735590780 at a74786d (lucee7 + boxlang × h2/cockroachdb/sqlite): 5/5 legs green, JobSchemaSpec 8/8 on each. That covers the H2 TYPE_NAME path, the CockroachDB family mapping, and the migrator up/down/up on both.
• Delta check. 10ddf9b→a74786d2d is the hasTable() confirmation in $ensureJobTable(), which behaves the same on every engine, plus one spec. That spec's MockBox prepareMock/$() usage has prior art in 20 core spec files, and it ran green on Lucee 7, BoxLang and RustCFML at this head.
• PR CI at a74786d: 22 pass, 0 fail, 3 skipped. Lucee 7 + SQLite (LuCLI) is path-filtered (no CLI change since e805d6b, where it and JobsCommandSpec passed); the other two skips are reviewer jobs.

[bpamiri]

Review at 17b100d (develop merge 4284bfc, reviewed as code). REQUEST CHANGES: one MUST-FIX, an intermittent spec failure that turns the compat-matrix gate red. The schema adoption itself is sound.

MUST-FIX

1. JobSchemaSpec "with jobsAutoCreateTables = false, reports a probe failure as itself when the table exists" fails on boxlang-sqlite in matrix 37824635569 (Compatibility Matrix Gate red).

• Symptom. The failed assertion is Len(probed.message) > 0 ("the probe's own error is rethrown"), so $ensureJobTable() returned without throwing.

• Why that points at the spec. With hasTable mocked to true, the only no-throw path is that the re-run SELECT … FROM wheels_jobs WHERE 1=0 succeeded, i.e. wheels_jobs still existed. The spec's $dropJobTable() swallows any DROP TABLE error, so a failed drop (SQLite table lock or busy handle, plausibly after the merged-in specs) leaves the table in place and the spec runs against the wrong precondition.

• It's intermittent. PR CI's own boxlang + sqlite leg at the same head passed (0 failures, 7383 specs), and the spec was 8/8 on boxlang-sqlite at a74786d (37735590780).

• Fix:

  • make the precondition explicit: after $dropJobTable(), assert new wheels.JobSchema().hasTable("wheels_jobs") is false (or retry the drop) before mocking hasTable, so a failed drop fails as "couldn't set up", not as a behaviour regression;
  • better, make $dropJobTable() stop swallowing errors when the table still exists afterwards;
  • then find out why the drop fails on BoxLang + SQLite at that point in the suite (an unclosed result or handle from an earlier jobs spec?).

  Not a CI re-run: a spec that's order- or timing-dependent will come back.

LOW

• The hosts warning overstates what's lost. With auto-create off and no wheels_job_hosts, $warnAuxTableMissingOnce("wheels_job_hosts", "the per-host cap and drain are unavailable") is logged. The cap still works without the registry: $runningOnHost() counts processing rows by claimedBy in wheels_jobs. Only drain, the registry row and wheels jobs status' host view are lost. Suggest: "drain and the host registry are unavailable (the per-host cap still applies)".

Verified (as code)

• One definition for every table. JobSchema defines all four job tables, and each matches the DDL it replaces column for column (types, defaults, NOT NULLs, PKs):
  • wheels_jobs, now with heartbeatAt and result as unicodeString (NVARCHAR(4000) on SQL Server);
  • wheels_job_hosts;
  • wheels_job_schedules, with decimal precision 15 → DECIMAL(15,0);
  • wheels_job_locks (= LeaseLock.createTableSql() for the job table; the migrator's own lock table is untouched).
• Every auto-create path uses JobSchema and respects jobsAutoCreateTables:
  • jobs table → SchemaMissing (catalog-confirmed);
  • column upgrades skipped;
  • locks → SchemaMissing;
  • hosts/schedules → logged once, feature off;
  • $ensureJobLockTable's in-transaction message now points at wheels jobs install.
• Generated migration:
  • it covers all four tables and sets this.useTransaction = false, which the Migrator honours (Migrator.cfc ~1093) and which is justified (idempotent, mixed DDL commit semantics);
  • unicodeString → t.string and then ALTER COLUMN … NVARCHAR on SQL Server at create, or ADD … NVARCHAR when adopting;
  • decimals pass precision, scale = 0;
  • adoption still adds only missing nullable columns, plus the uniqueKey backfill.
• CLI. install sits beside drain/resume, and the bridge jobsInstallSource is unchanged.

Evidence

• Full matrix 37824635569 at 17b100d (all 32 legs; I re-fetched the artifacts after a partial first download):
  • 31/32 green, 0 fail / 0 error, 591 bundles each, with JobSchemaSpec 8/8 on all of those (Oracle, CockroachDB, H2 and SQL Server included, so the NVARCHAR check ran);
  • boxlang-sqlite: 7235 pass, 1 fail (MUST-FIX 1);
  • RustCFML 7208/0/0.
• PR CI at 17b100d: 22 pass, 0 fail, 4 skipped (including boxlang + sqlite: 0 failures). Merge state is clean.

[bpamiri]

Final review: head 4a60553. APPROVE (ready to merge).

• MUST-FIX resolved. The probe-failure spec no longer depends on table state: it fails the probe with a datasource that doesn't exist, with hasTable mocked to true, and asserts that the datasource's own error surfaces rather than SchemaMissing. $dropTableOrFail() checks the catalog before and after the DROP and throws Spec.DropFailed with the DROP's error instead of swallowing it, so a spec can no longer run on the wrong precondition.
• Root cause tracked. An earlier spec leaves an uncommitted transaction on a pooled BoxLang + SQLite connection. test: an earlier spec leaves an open transaction on a pooled BoxLang + SQLite connection #4508 tracks it, with the evidence and bisect criteria.
• LOW resolved. The hosts-missing warning now says drain/resume and the host record are unavailable and the per-host cap still works (claimedBy).
• Evidence:
  • full matrix 37849356846 at 4a60553: 32/32 legs green, 0 fail / 0 error, 591 bundles, JobSchemaSpec 8/8 on every leg (including boxlang-sqlite, the leg that failed before); RustCFML 7208/0/0;
  • PR CI: 24 pass, 0 fail, 2 skipped.
• Everything verified as code at 17b100d still holds: the delta is the spec and the one warning string.

[bpamiri]

Delta-check at a6a52cb (develop a6c2d00 merged, including #4507): PASS. The final approval stands.

• Change set. Every file the PR touches has the same added/removed lines against its new base as the approved 4a60553 had against 87bbd01 (compared per file, ignoring context).
• The one conflict. In the onapplicationstart defaults, both settings were kept: jobsAutoCreateTables = true and fix(job): reclaim a job when its heartbeats stop, not only after 2x its timeout #4507's jobsHeartbeatGraceSeconds.
• fix(job): reclaim a job when its heartbeats stop, not only after 2x its timeout #4507 interaction. It adds no columns (it reads the existing heartbeatAt), so JobSchema and the generated migration need no change.
• Narrowed matrix 37854454930 at a6a52cb (5 engines × sqlite/postgres/cockroachdb/sqlserver):
  • 20/20 legs green, 0 fail / 0 error, 592 bundles;
  • JobSchemaSpec 8/8 on every leg;
  • RustCFML 7215/0/0.
• Evidence for the unchanged code: full matrix 37849356846 at 4a60553 (32/32).
• PR CI at a6a52cb: 24 pass, 0 fail, 2 skipped.

[bpamiri]

Delta-check at 11db051 (develop 7f4dc07 merged, including #4518): PASS. The final approval stands.

• Unchanged files. Every file except cli/lucli/Module.cfc has the same added/removed lines against its new base as the approved a6a52cb.
• Module.cfc. The only difference is the resolved jobs usage line, which now carries both install [--force] (this PR) and --job-timeout=<seconds> (fix(cli): let wheels jobs work run each job with its own timeout #4518). jobsArgSpec has both the job-timeout option and the install force flag, and $parseJobsArgs() maps both (jobTimeout = parsed["job-timeout"], force = parsed.force).
• SQLite matrix 37857378288 at 11db051 (all 6 engines):
  • 5/5 JVM legs green, 0 fail / 0 error, 593 bundles;
  • JobSchemaSpec 8/8;
  • RustCFML 7218/0/0.
• PR CI at 11db051: 24 pass, 0 fail. Deploy CLI tests + verb smoke: 2448 pass, 0 fail, 0 error (includes the install and job-timeout CLI specs). Merge state is clean.
• Earlier evidence for the unchanged framework code: 37854454930 and the full 37849356846.