Skip to content

feat(migrations): the startup policy, serve --migrate and [server] catalog_upgrade - #741

Open
LeeroyHannigan wants to merge 16 commits into
mainfrom
feat/catalog-startup-policy
Open

feat(migrations): the startup policy, serve --migrate and [server] catalog_upgrade#741
LeeroyHannigan wants to merge 16 commits into
mainfrom
feat/catalog-startup-policy

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #681, the third PR of the catalog migration work: what serve does at startup with a catalog that is behind the binary.

  • [server] catalog_upgrade sets it. refuse, the default in a production build, exits 3 as before, and the message now ends with "or start this server with --migrate". apply, the default in a dev build, migrates the catalog and then serves. serve --migrate sets apply for one run, and EXTENDDB__SERVER__CATALOG_UPGRADE sets it from the environment. Every other status is refused under both values with the code it had.
  • Under apply the server migrates through the same locked run as migrate --yes: it waits up to [server] catalog_upgrade_lock_timeout (60 s by default) for another migration and for running servers, reads the status again under the lock, and serves a catalog that another process migrated meanwhile. A catalog found current under the lock is served without calling the backend's run. A lock still held at the timeout exits 6.
  • serve reads the status, and under apply migrates, before it forks, so in daemon mode the exit code and the message reach the invoking shell. While it migrates it prints a line on stderr for each step and one every 5 s, each starting with "catalog migration: ". The catalog check runs before the bind.
  • The npm launcher counts its startup timeout from the last migration progress line, passes progress to onProgress, stops a server that fails to start, and reports its exit code or signal.
  • MongoDB releases the migration lock with the lock's own client.
  • The lock check scripts check the startup policy on every backend, and the upgrade manual and deployment guide describe serve --migrate, catalog_upgrade and the deployment recipes, Kubernetes included.

Testing

clippy (1.97.0) and rustfmt pass on the changed crates, and their unit tests pass (extenddb-storage 213, extenddb-app 51, extenddb-config 11). The integration jobs and the lock check scripts run in CI.

@LeeroyHannigan
feat(migrations): apply a catalog's pending migrations at startup
cf9329b 
[server] catalog_upgrade sets what serve does with a catalog that is
behind the binary. refuse, the default in a production build, exits 3
as before. apply, the default in a dev build, migrates the catalog and
then serves. serve --migrate sets apply for one run, and
EXTENDDB__SERVER__CATALOG_UPGRADE sets it from the environment. Every
other status is refused under both values with the code it had, and no
value serves a catalog the binary cannot use.

Under apply the server migrates through run_with_lock, which migrate
--yes now calls too: it takes the migration lock, waiting up to
[server] catalog_upgrade_lock_timeout (60 s by default, at most a day)
for another migration and for running servers, reads the status again
under the lock, runs detect for every pending code body, takes the
SQLite copy, applies the migrations and releases the lock. A catalog
that another process migrated while this one waited is served as it
is. When the lock is still held at the timeout, the status is read once
more: a catalog that is current by then is served, and any other exits
6. Under apply serve does not read the notes migrate --yes checks
before it takes the lock, so a body that fails on a table the role does
not own stops the run with the same note.

serve reads the status, and under apply runs the migrations, before it
forks, on a current-thread runtime it shuts down first, so in daemon
mode the exit code and the message reach the invoking shell. After the
fork the server reads the status again and registers, and exits 6 when
a migration holds the lock. While it migrates it prints a line on
stderr for each step, each starting with "catalog migration: ", and one
every 5 s until the run ends.

A server's refusal of a catalog behind it ends with "or start this
server with --migrate". The SQLite factory no longer migrates a Behind
catalog in a dev build: the startup policy does that before the factory
runs. An in-memory SQLite database is created by its server and is not
read before it starts. MongoCatalogMigrations can name its databases,
so the integration tests run the policy on databases of their own. init
writes both keys, commented out, into the config it generates.
@LeeroyHannigan
feat(npm): count the startup timeout from the last migration progress…
5d7e9b1 
… line

A dev server applies a catalog's pending migrations before it serves,
which on a large database takes longer than the launcher's startup
timeout. The server prints a line on stderr for each step of the run,
each starting with "catalog migration: ", and one every 5 s while it
goes on. The launcher now counts startupTimeoutMs, 15 s by default,
from the later of its start and the last such line, so it waits for a
server that reports progress and gives up on one that prints nothing.

The wait moves to startup.js, which the package ships. The launcher test
runs a stand-in server that answers /health after 20 s: with a progress
line every 2 s it is waited for, and with none the wait ends at 15 s.
@LeeroyHannigan
ci: check serve's startup policy in the catalog lock scripts
46333c7 
On each backend the catalog lock script puts the catalog behind the
release binary and checks that serve exits 3 with a message that names
--migrate. In daemon mode the exit code and the message reach the
invoking process. The script then holds the migration lock and checks
that serve --migrate waits until catalog_upgrade_lock_timeout and exits
6. Once the lock is free serve --migrate applies the migration and
serves.

The SQLite script also checks catalog_upgrade = "apply" in the config
file, EXTENDDB__SERVER__CATALOG_UPGRADE=apply and serve --migrate in
daemon mode. It checks that serve --migrate writes nothing to a current
catalog, exits 5 with nothing written on a catalog that needs a person,
and that two serve --migrate started together on a released catalog
migrate it once, with one copy.

devtools/run-mongodb-tests runs the startup policy tests against its
replica set and the PostgreSQL job without pgvector runs them too.
@LeeroyHannigan
docs: serve --migrate, catalog_upgrade and the deployment recipes
9ee12f3 
The upgrade manual describes the startup policy: [server]
catalog_upgrade and its defaults, serve --migrate, the environment
variables, the steps serve takes under apply, the progress lines, and
the statuses serve refuses under both values. It gives the recipes for
a single host, for Docker Compose, which needs no change, and for
Kubernetes: one replica with the Deployment strategy Recreate and a
migration Job applied in the same release, with the reasons the recipe
uses no Helm pre-upgrade hook and does not migrate in the pod. It lists
the exit codes of serve, which in daemon mode reach the invoking shell.

The deployment guide gives the Kubernetes recipe, and its upgrade
checklist names serve --migrate. docker/README.md says what the serving
container does with a catalog behind its image. The admin guide lists
the two keys, and the sample config, the process lifecycle in the
architecture guide and the troubleshooting entry for exit 6 follow the
change.
@LeeroyHannigan
fix(mongodb): release the migration lock with the lock's own client
de4b0bf 
MongoCatalogMigrations now overrides release: it deletes the lock
document through the client that took the lock, on the caller's runtime,
and sets an atomic released flag, so dropping the lock afterwards does
nothing. Before, the release in run_with_lock dropped the lock, and Drop
blocked the runtime's thread, connected a new client on a thread of its
own and ignored a failed delete. A failed delete left the document until
its lease ended, up to 60 s, and the server serve --migrate starts after
its run found the document when it registered and exited 6.
@LeeroyHannigan
fix(migrations): serve a catalog found current under the lock without…
1642e83 
… a run

The startup policy's run under apply changes in these ways.

- When the status read under the lock is Current or AheadCompatible, the
  server releases the lock and serves the catalog without calling the
  backend's run, so on PostgreSQL it makes no pgvector attempt and opens
  no runner session. run_with_lock takes a WhenCurrent argument for this:
  Release on the server's path, and Run for migrate --yes, which keeps
  its pgvector attempt on every run.
- After a run that applied nothing, the status is read again, and a
  catalog ahead of this binary by additive migrations gets the warning
  that migrate --yes prints after such a run.
- Before it takes the lock, the server checks the status notes that
  migrate --yes checks (before_lock). A note that refuses runs stops the
  server with exit 1 and the same notes before any change, and the
  refusal names serve with --migrate or catalog_upgrade = "apply", which
  before_lock now takes as an argument. Notes that do not refuse are
  printed as progress lines. The PostgreSQL ownership note names both
  commands.
- A dev build refuses, before the server starts, an uninitialized store
  that its server would refuse, as Uninitialized with exit 1. The new
  CatalogMigrations::server_creates_catalog says which stores: on SQLite,
  a database file that holds tables. In daemon mode the refusal and its
  message now reach the invoking shell; before, the factory refused the
  file after the fork, and the shell saw exit 1 with no message.
- A progress line comes before the first status read, which runs detect
  for every pending code body, and before the status read that follows a
  wait that reached the lock timeout.
- The PostgreSQL pgvector notice names the full command, extenddb
  migrate --yes --pg-user <role>, so the notice is correct when serve
  prints it.

The doc comments now say that the lines of what a run did come when the
run ends, with a heartbeat line every 5 s while it goes on. The startup
unit tests pass std::slice::from_ref to the fake, as clippy's
cloned_ref_to_slice_refs requires, and cover each change: a dev build
refuses every status its server cannot serve, the fallback after a wait
serves a catalog ahead by additive migrations, a run on a catalog found
current under the lock calls no backend run, and a refusing note stops
the run before the lock. The PostgreSQL ownership test checks the
server's refusal too.
@LeeroyHannigan
fix(serve): check the catalog before the bind, log during it, bound i…
9df3f5d 
…ts shutdown

serve now starts in this order: the config checks, the refusal of daemon
mode on a platform without it, the catalog check, the bind, the banner
and the fork. The platform check used to come after the catalog check,
so serve without --foreground on such a platform read the status, and
under apply migrated, before it failed. The bind used to come before the
catalog check, so during a long migration the kernel accepted
connections that got no answer until the server started. A port
conflict is still reported before the fork, and the banner still
follows the bind.

The catalog check runs under a stderr subscriber built as the one
migrate --yes uses, set with tracing::subscriber::with_default for the
check alone, so the warnings the backends log during a run reach stderr
and the global default stays free for the server. Before, the run came
before the server installed its subscriber, and PostgreSQL's lock retry
warnings and Cassandra's lease warnings were dropped.

The check times the shutdown of its runtime. When the shutdown reaches
its 10 s cap, a blocking thread may still be running, and a lock that
thread holds would stay held in the child after a fork. In daemon mode
serve then prints the cause on stderr and exits 1 instead of forking.

Each line of a progress message is printed with the "catalog migration: "
prefix, so a message that spans lines counts as progress on each line.
The check takes its inputs in ServeStart, and its core takes the backend
and the output, so the tests run it on a fake. They cover --migrate over
catalog_upgrade = "refuse", the prefix on every line, the exit code of a
refusal, which is never 0, the cap on the shutdown, the bound of a day
on catalog_upgrade_lock_timeout, and the prefix the npm launcher matches
in packaging/npm/startup.js.
@LeeroyHannigan
test(migrations): rename the two-run startup tests, and add the missi…
b23da5e 
…ng ones

The four two_servers_started_together tests ran two prepare_to_serve
calls and never started a server, and counted a run whose first status
read already saw the catalog current as one that waited for the lock.
They are now two_prepare_to_serve_runs_under_apply_migrate_once, with
migrate_a_legacy_catalog_once at the end on Cassandra. Each holds the
lock as another process until both runs wait for it, then frees it, and
asserts that one run applied the migrations and the other took the lock
after it and applied nothing: a report with no step.

New tests on PostgreSQL: a zero lock timeout exits 6 at once through the
server's run and through the run migrate --yes makes, while another
session holds the migration key or the serving key; and a run that waits
for both keys exits 6 once one timeout has passed in all. New tests on
SQLite, in process: a catalog another process migrates during the wait
is served with nothing written and no backend run, and a dev build's
policy refuses a file that holds another program's tables with the
refusal the factory gives.
@LeeroyHannigan
fix(npm): stop a server that fails to start, and report its exit code
b0d5906 
Each /health probe has a 1 s limit that starts before the connect. The
old limit started once the socket connected, so a probe of a port that is
bound but does not accept waited for the agent's socket timeout instead:
5 s with Node 22, measured against a listener with a full accept queue.

start() stops the server before it rejects, because the caller has no
handle to a server that failed to start. When the server exits first, the
error carries its exit code as exitCode and its signal as signal. The
error is built on "close", after the server's stderr has ended, so it
holds the last lines the server printed. A new onProgress option receives
each catalog migration line while the server starts. extenddb start writes
those lines to its stderr and exits with the server's exit code.

test/start.test.js runs start() and extenddb start against
test/fake-server.js through a replaced child_process.spawn, with
startupTimeoutMs unset. A server that reports progress for 20 s is waited
for, one that prints nothing is given up on at 15 s and stopped, and the
exit code and the signal reach the error. The silent stand-in answers
after 60 s, so a deadline check that runs late still ends in a timeout.
The scenario that called waitForStartup directly leaves
test/launcher.test.js. The launcher workflow and the npm build matrix run
the new file.

The README says when the startup timeout counts from, documents
onProgress and the error's fields, and says that a migrated database
keeps the copy taken before the migration beside it.
@LeeroyHannigan
ci: bound the lock waits and clean up daemons in the catalog lock scr…
1b0386f 
…ipts

Each lock script checks that serve --migrate waits at least the 3 s set in
EXTENDDB__SERVER__CATALOG_UPGRADE_LOCK_TIMEOUT and less than 30 s, so a
serve that ignored the setting and waited the 60 s default fails. The
MongoDB check of that wait also looks for the lock in the output, and its
two exit 3 checks have different names.

The SQLite script compares the database's content, read with iterdump,
before and after serve --migrate exits 5, so a write left in the WAL
fails the check. On a current catalog the check is named for what it
compares: serve --migrate adds no ledger row and takes no copy. Of two
serve --migrate started together, exactly one serves, and the other finds
nothing to apply or exits 6; an exit 1 from it fails the check.

Each script stops, in its cleanup, a daemon a check left running, found
through the daemon's PID file and only when that PID runs the binary under
test. Daemon mode runs go through run_split, which reports a command still
running after 120 s as a failed check. healthy() gives each curl probe a
2 s limit. The integration workflow's comment names both ports the SQLite
script uses.
@LeeroyHannigan
docs: describe the startup policy as the server applies it
7fff9d6 
The upgrade manual's part on applying migrations at startup becomes its
own section after the rollback procedure, outside the region where a
merge with pull request #523 conflicts. It says that under apply serve
reads the PostgreSQL ownership notes before it takes the lock and exits 1
on them, and that catalog_upgrade_lock_timeout bounds all the waits of a
run together on every backend, with 0 making one attempt. It says that a
catalog found current under the lock is served with nothing written, what
two servers started together do on each backend, which progress lines
serve prints and when, that serve binds its port after the catalog check,
and that SQLite servers from 0.1.14 on are detected.

The Kubernetes recipe in the upgrade manual and in the deployment guide
runs the preview first and migrate --yes only on exit 3, so the Job of a
release with nothing to apply succeeds when the new pod registers first.
Both say that the Job's --lock-timeout has to be longer than the old pod
takes to stop, and that a pod that exited 3 is restarted after its
CrashLoopBackOff delay, which grows up to 5 minutes. The reason against a
Helm pre-upgrade hook covers 1.0 and the upgrade from 0.1.x.

The architecture guide's process lifecycle binds the port after the
catalog check. ADR-0023 describes the startup policy. The admin guide
names the statuses serve refuses under both values, the troubleshooting
entry says what exit 6 means for serve, and the two install guides run
migrate --yes.
@LeeroyHannigan
test(app): give the startup tests' applied steps an outcome
205017f 
AppliedStep gained an outcome field in 86154f6, on the branch this one
is stacked on, after these tests were written.
@LeeroyHannigan
fix(devtools): the PostgreSQL lock check calls pid_file under another…
3dde176 
… name

main() also assigns a local pid_file, the path of the daemon's PID file, so
the call that stops the daemon raised "TypeError: 'str' object is not
callable" in CI's "Check serve and migrate on one catalog" step. The
function is imported as pid_file_for, as the Cassandra check does.
@LeeroyHannigan
LeeroyHannigan marked this pull request as ready for review November 2, 2026 01:35
@LeeroyHannigan LeeroyHannigan mentioned this pull request Nov 2, 2026
Open
4 of 8 tasks
Base automatically changed from feat/catalog-migrations-cassandra-mongodb to main November 2, 2026 22:45
@LeeroyHannigan
Merge branch 'main' into feat/catalog-startup-policy
c36d717 
Main merged #666 and #681, which add migration 5, drop_continuous_backups,
on every backend.

The architecture guide keeps this branch's text on catalog_upgrade and
--migrate, drops the release's migration number as main did, and adds
main's sentence on what a server that refuses its catalog prints. The admin
guide keeps this branch's text with main's example of a catalog at
migration 4 that needs migration 5. The upgrade manual's status paragraph
keeps this branch's serve --migrate route with main's migrations 1 to 5 and
its note on migration 5, and main's last rollback paragraph now comes
before this branch's section on applying migrations at startup.

The MongoDB startup tests expect six steps and migration 5 after a run on
a released catalog.
@LeeroyHannigan
Merge origin/main into feat/catalog-startup-policy
d0154a6 
crates/config/src/lib.rs: main added the backup module (#671) at the end
of the file, where this branch adds the catalog_upgrade module and its
re-exports. The result declares both.

docs/manuals/05-admin-guide.md: main rewrote the region row of the
[server] table (CreateBackup is refused under a region outside the ARN
class). This branch adds the catalog_upgrade and
catalog_upgrade_lock_timeout rows below it. The result keeps main's
region row and this branch's two rows.

docs/manuals/07-upgrade-manual.md: in the Current Status paragraph main
counts migrations 1 to 6 and names migration 6, the backup catalog
index. This branch adds the serve --migrate alternative to the
migrate --yes step and says that a server refuses pending migrations by
default. The result keeps main's count and its migration 6 sentence with
this branch's clause and link.
@LeeroyHannigan
test(storage-mongodb): count migration 6 in the startup tests
d8428eb 
main's migration 6 adds one step to the migration of a 0.1.14 catalog.
apply_migrates_a_released_catalog_and_refuse_writes_nothing and
two_prepare_to_serve_runs_under_apply_migrate_once expected 6 applied
steps and a catalog at version 5. They now expect 7 steps, the count
main's lock test uses, and a catalog at MigrationId::LATEST.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@amrith amrith Awaiting requested review from amrith amrith is a code owner
@c33howard c33howard Awaiting requested review from c33howard c33howard is a code owner
@jcshepherd jcshepherd Awaiting requested review from jcshepherd jcshepherd is a code owner
@pdf-amzn pdf-amzn Awaiting requested review from pdf-amzn pdf-amzn is a code owner
@yesyayen yesyayen Awaiting requested review from yesyayen
@robinnsc robinnsc Awaiting requested review from robinnsc robinnsc is a code owner
@eliseong eliseong Awaiting requested review from eliseong eliseong is a code owner
@hitesh96db hitesh96db Awaiting requested review from hitesh96db hitesh96db is a code owner
@navareamol navareamol Awaiting requested review from navareamol navareamol is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@LeeroyHannigan