Skip to content

fix: create SQLite database files 0600 and set wider ones to 0600 when opened - #691

Open
LeeroyHannigan wants to merge 14 commits into
mainfrom
fix/657-sqlite-file-modes
Open

LeeroyHannigan wants to merge 14 commits into
mainfrom
fix/657-sqlite-file-modes

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

What

crates/storage-sqlite/src/file_mode.rs adds connect(location, SqlitePoolOptions), and the bootstrapper pool, catalog_pool and SqliteEngine::new now open their pools through it. It runs prepare before the pool's first connection and restrict after it. The migration runner opens its own connection when it writes: for migrate --yes, and for the first open of a dev-mode serve. It calls prepare and restrict around that connection. Its read-only connection for the catalog status changes no mode.

  • prepare creates the database file with mode 0600 when the connection would create it, then narrows the file and any -wal, -shm or -journal file left by an earlier run, so the sidecars SQLite creates on this connection take 0600 from the database file. The file is created with create_new, which refuses any existing entry, a symlink included. For a dangling symlink at the database name, the link's target is created.
  • restrict narrows the database file and each sidecar present again once the pool has connected.

Which files are changed

  • Only a regular file with a permission bit outside 0600 is changed, and it is set to 0600. A regular file at 0600, 0400, 0200 or 0000 is left alone with no attempt to change its mode, so a read-only mount whose files are already 0600 opens as before.
  • A symlink at the database name is followed, as SQLite follows it. A symlink at a sidecar name is left alone and its target keeps its mode. SQLite opens its files with O_NOFOLLOW, so if it needs that sidecar, its own open fails and the command reports SQLite's error.
  • A directory, FIFO, socket or device at any of these names keeps its mode, and so does the directory holding the database.
  • On Linux each name is opened with O_PATH | O_NOFOLLOW, which opens no file for reading or writing, so nothing waits on a FIFO or a device. The type and mode are read from that handle and the mode is changed with chmod of /proc/self/fd/<n>, so the check and the change act on the same file. Closing an O_PATH handle keeps the POSIX locks the process's SQLite connections hold on the file. Closing a descriptor opened for reading releases them, which section 2.2 of How To Corrupt An SQLite Database File lists as a cause of corruption. serve opens the catalog pool while the engine's pool is connected, and connect narrows the files while its own pool is connected, so the narrowing runs while connections hold locks on the file.
  • On other Unix systems (macOS) the type and mode come from lstat and the mode is changed with fchmodat(AT_SYMLINK_NOFOLLOW), which opens nothing.
  • A regular file wider than 0600 that cannot be narrowed is an error, cannot set the mode of SQLite file <path>: <error>. A database file prepare created whose mode cannot be set gives cannot set the mode of SQLite database <path>: <error>. The .lock file already follows this policy (fix(sqlite): one extenddb process per database file #391).
  • On platforms other than Unix both steps do nothing.

Which file the connection opens

prepare and restrict find the file the connection opens, and prepare decides whether the connection creates it, by reading the location as sqlx 0.8.6 and the bundled SQLite 3.46.0 read it. sqlx reads the connection string's query as form-urlencoded: mode=rwc creates unless a mode=ro is also given, and the last vfs and the last immutable win. A file: URI's own query is read by a new module, crates/storage-sqlite/src/sqlite_uri.rs, which follows SQLite's sqlite3ParseUri: %HH is decoded, ?, #, & and = separate only when they appear raw, the last vfs wins, and each mode is applied in order. As in SQLite, a mode that asks for more access than the one before it fails the open, so no file is created. A file: name combined with vfs or immutable is left to the connection, because sqlx then wraps it so that SQLite opens a relative path spelled file:....

serve_lock::database_file reads a file: path through sqlite_uri and finds mode=memory through the same form-urlencoded parse. For a file: URI with %HH in its path, the lock file is now next to the file SQLite opens, and an encoded mode=memory (for example mod%65=memory) now takes no lock, like the other in-memory forms. For other locations the lock path stays where it was, and the existing lock path tests pass without changes.

The chmod loop in SqliteEngine::new is removed. catalog_pool returns its error as a String, serve_lock::resolve_dangling_symlinks is pub(crate), and the crate takes the workspace url dependency, which Cargo.lock already had through sqlx. docs/troubleshooting.md gains an entry for the two errors and says that settings and verify set these modes, and docs/manuals/10-security-model.md names the database files.

Why

extenddb init --backend sqlite leaves the database file and its -wal and -shm files at mode 0644, while the config file and the .lock file are 0600. The database holds the encryption key next to the access-key secrets and password hashes it protects, so any local user can read them until the first serve narrows the files.

Measured on main (1c0523c9) on 30 October 2026 with a binary built with --no-default-features --features sqlite, under umask 022, and for the first three items under umask 002 as well:

  • After init: extenddb.toml 600, x.sqlite 644, x.sqlite-wal 644, x.sqlite-shm 644, x.sqlite.lock 600. Umask 002 gives 644 as well, because SQLite creates a database at 0644 less the umask.
  • After the first serve start all five files are 600. The serve engine set the modes after its pool connected, on the files present at that moment.
  • migrate over a copy of the database at 0644 with no sidecars creates -wal and -shm at 644, and over a copy at 0600 it creates them at 600. SQLite gives a new sidecar the mode of the database file.
  • SQLite resolves symlinks before it names the sidecars: through link.sqlite -> real.sqlite it uses real.sqlite-wal and real.sqlite-shm.
  • init with a sqlite: URL as the path also leaves the files at 644.

The bootstrapper pool (init, migrate, destroy) and the catalog pool (settings, verify, and the management and diagnostics stores) open the file without setting a mode. The serve engine set modes only after connecting. It appended -wal and -shm to the configured string, which for a sqlite: URL names files that do not exist, and it ignored errors.

Closes #657

Testing done

Measured with a binary built from this branch on 31 October 2026, under umask 022 and 002: after init all five files are 600, and they stay 600 through the first serve, management writes, CreateTable, PutItem, a clean stop, a second serve and migrate. migrate over copies at 0600 and 0644 with no sidecars leaves the database and its new sidecars at 600.

Unit and integration tests

Unit tests in crates/storage-sqlite/src/file_mode/tests.rs, independent of the test process's umask:

  • A missing database is created at 0600, also at the target of a dangling symlink. The mode the new file has before it is narrowed, read through its descriptor, equals the mode of a file the test creates with mode 0600 under the same umask.
  • An existing 0644 database and its sidecars become 0600, through a plain path and through a sqlite: URL. Inside after_connect, before restrict runs, the database and the sidecars SQLite created are 0600, for a missing database and for an existing 0644 one. Files set to 0644 inside after_connect are 0600 once connect returns.
  • A 0755 directory at the database name keeps its mode, with and without a trailing /. A symlink to a 0644 file at -journal, a FIFO at -wal and a socket node at -shm keep theirs, prepare and restrict each return within ten seconds, and the link's target stays 0644.
  • With a mode call that fails with EROFS, prepare and restrict succeed on files at 0600 and 0400 without the call being made, and a 0644 file gives an error that names the file and stays at 0644. The test puts the failing call in place of the system call, so it changes no file outside its own temporary directory, also when it runs as root.
  • On Linux, a WAL pool stays open while its files go from 0644 to 0600 through restrict. Afterwards an F_OFD_GETLK query still finds the connection's read lock on the database file's shared range and on the -shm file's DMS byte.
  • Nothing is created for sqlite:///path without a query, mode=ro, mode=rwc&mode=ro, vfs=memdb, :memory:, an encoded mod%65=memory, a file: name with vfs or immutable, and file: URIs whose own query does not create (mode=ro, mode=rw, mode=memory, vfs=memdb, and mode=ro followed by mode=rwc). A repeated vfs that ends with unix, and file: URIs whose own query (written with %3F) is cache=shared or mode=rwc, create the database at 0600 with its sidecars. file:<dir>/a%2520b.sqlite reaches a b.sqlite and leaves no stray a%20b.sqlite.

sqlite_uri.rs has unit tests for the decoding, the separators, %00, empty option names, the authority, and creates_file over 13 queries. serve_lock.rs has new tests for the lock path of a decoded file: path, an encoded mode key, and a file: name with vfs (unchanged).

Integration test binary crates/storage-sqlite/tests/file_modes.rs, which sets umask 022 in its own process:

  • init (the bootstrapper sequence the CLI runs), serve through the server components factory, a first write (create-account, CreateTable, TagResource), a stop and a second serve: after each step each regular file in the deployment directory is 0600.
  • A dev-mode serve that bootstraps a missing database creates 0600 files.
  • Files set to 0644 after init are 0600 after a serve and after migrate --yes.
  • Each pool alone on a missing path, with the modes read before any other pool opens the file: the bootstrapper pool, the engine pool, and the catalog pool through the settings store factory.
  • Control: a pool opened outside these openers leaves 0644 files under that umask.
Negative controls

Each mutation was made on a clean committed tree and reverted afterwards. The first two ran on this branch before the commits that change how files are narrowed and how locations are read; the others ran on those commits.

Mutation What fails
connect without prepare the unit tests that read modes inside after_connect (sidecars at 0644); the integration tests pass, because restrict repairs the end state
connect without restrict the unit test that sets 0644 inside after_connect
no regular-file check the directory test (set to 0600) and the FIFO, socket and symlink test
O_PATH without O_NOFOLLOW the FIFO, socket and symlink test (the -journal link followed)
files at 0600 changed too the owner-only test (the failing mode call is made)
a descriptor opened for reading in place of O_PATH the FIFO, socket and symlink test (blocked for ten seconds) and the lock test (lock released)
a descriptor opened with O_NOFOLLOW | O_NONBLOCK the lock test (lock released) and the FIFO, socket and symlink test (ENXIO on the socket node)
the mode error discarded the error test
the new file created at 0644 the creation-mode unit test; the integration tests pass, because the narrowing repairs the end state
catalog_pool without file_mode the catalog pool integration test
SqliteEngine::new without file_mode the engine pool integration test
the first vfs wins the location test with vfs=memdb&vfs=unix
a file: URI with its own query left to SQLite the location test with %3Fcache=shared
no %HH decoding the %2520 tests for the file and the lock path, and two sqlite_uri tests
mode=memory found with a raw split the encoded mode key tests for the file and the lock path
a file: name with vfs or immutable not left alone the test of locations SQLite does not open as named

Gate

  • cargo +1.97.0 fmt --all -- --check: no changes.
  • cargo +1.97.0 clippy --locked -p extenddb-storage-sqlite --all-targets -- -D warnings: no warnings.
  • cargo test --locked -p extenddb-storage-sqlite on stable (1.99.0) and on 1.88.0: the crate's 185 unit tests and 7 integration tests pass, and 1 test is ignored, as on main.
  • cargo check --locked --workspace --all-targets: no warnings.
  • sqlite_uri.rs compiled with rustc +1.97.0 --target x86_64-pc-windows-msvc beside a caller that uses parse and path, as serve_lock does on Windows: no warnings, with and without --test. Before the last two commits it gave seven dead-code warnings.
  • After the last two commits, on 1.97.0: fmt and clippy are clean, and the crate's 185 unit tests and 7 integration tests pass with 1 ignored.
  • After the merge of main (0e9a6ed), fmt and clippy on 1.97.0 are clean. The crate's 258 unit tests and 7 integration tests pass, with 1 ignored.

These tests ran on Linux. The macOS version of the narrowing has not been run. The npm build workflow on this pull request compiled it for both macOS targets, and both builds passed.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

CI ran the workspace tests on Rust 1.88.0 and on stable, and the integration suites on all four backends. All of them passed. Locally I ran the SQLite crate's tests as listed under Gate. Clippy ran with -D warnings, without the pedantic group. The change sets file modes and adds no command, flag or file format.

ADR / RFC: n/a

Breaking changes

  • A mode that cannot be set is now an error for init, migrate --yes, settings, verify and serve. On main the serve engine ignored a failed chmod and the other paths set no mode. destroy carries on when its catalog read fails, as on main, and then removes the files while holding the lock file.
  • Files left at 0644 by an earlier version are set to 0600 by the first settings, verify, migrate --yes or serve their owner runs.
  • A user other than the owner of the database files is now refused by settings and verify, and by serve and migrate --yes when no lock file is present. On main the lock file already refused that user for serve, migrate and destroy whenever it existed.
  • On a read-only mount, settings and verify work when the files are 0600 and are refused when the files are wider. One command run by the owner on a writable mount, or chmod 600, sets the files so that the read-only mount opens.
  • When the user cannot search the directory that holds the database, settings and verify now fail with cannot set the mode of SQLite file <path>: Permission denied (os error 13), where main printed SQLite's unable to open database file. Both exit 1 and leave the files' modes unchanged.

Measured on 31 October 2026 with a main binary (1c0523c9) and a binary built from this branch. Each command ran in an amazonlinux:2023 container on a bind mount of a deployment directory the owner created with init. The read-only rows ran as the owner. The group member is uid 65534 with the owner's group, and the other user is uid 65532.

Case Command main this branch
Read-only mount, files 0600, -wal and -shm present settings list, settings get, verify rc 0 rc 0
Read-only mount, files 0644 settings list, verify rc 0 rc 1, Read-only file system (os error 30)
Read-only mount, after the owner ran verify on a writable mount settings list, verify rc 0 rc 0
Read-only mount, after the owner ran chmod 600 settings list rc 0 rc 0
Read-only mount, files 0600, no -wal or -shm settings list rc 1, SQLite unable to open database file rc 1, same error
Group member, files 0660, directory 0770 settings list, verify rc 0 rc 1, Operation not permitted (os error 1)
Group member, files 0660, directory 0770 migrate --yes, serve, destroy --yes rc 1, cannot set the mode of lock file rc 1, same error
Group member, no lock file migrate --yes rc 0 rc 1, cannot set the mode of SQLite file
Group member, no lock file serve starts and listens exits with rc 1, cannot set the mode of SQLite file
Other user, files 0666, directory 0777 settings list, verify rc 0 rc 1, Operation not permitted (os error 1)

For a read-only mount, the new troubleshooting entry has the owner run one command on a writable mount or chmod 600 the files. For files that belong to another user, it has extenddb run as that user.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

…QLite opens it

`extenddb init --backend sqlite` left the database file and its -wal and
-shm files at mode 0644. SQLite's unix VFS creates a database file with
mode 0644 less the umask and gives each new sidecar the mode of the
database file, and the bootstrapper pool that init and migrate use never
changed either mode. The engine set the modes after connecting, on the
configured string with a suffix appended, and ignored failures.

Every pool opener on a file database (the bootstrapper, the engine and the
catalog pool) now creates a missing database file with mode 0600 before it
connects, when the connection would create the file, and sets the file and
any -wal, -shm and -journal file present to 0600 before and after it
connects. The sidecar names come from the resolved path, because SQLite
follows symlinks before naming them. A mode that cannot be set is an
error, as it is for the lock file. Other platforms are unchanged.
…fter init, serve and a write

Unit tests for file_mode: prepare creates a missing database at 0600 and
sets an existing database and its sidecars to 0600; a sqlite: URL without
mode=rwc, or with mode=ro, creates nothing; a WAL pool opened after prepare
gets 0600 sidecars from SQLite; a dangling symlink has its target created.

An integration test binary sets a umask of 022 in its own process and runs
the bootstrapper sequence init uses, the server components factory, a
management write, CreateTable and TagResource, a clean stop and a second
start, checking that every file in the deployment directory is 0600 with
the WAL sidecars present while a server is up. It also covers a dev-mode
serve on a missing path, and serve and migrate over files left at 0644. A
control test shows a connection opened outside the crate's openers leaves
0644 files under that umask.
…ly connect

The bootstrapper pool, the serve engine and the catalog pool now open
their pools through file_mode::connect. It runs prepare before the pool's
first connection and restrict after it, so an opener cannot run one step
without the other.

The new unit tests read the modes of the database file and its -wal and
-shm files inside the connection setup, before restrict runs. A database
left at 0644 with no sidecars gets 0600 sidecars only because prepare set
it to 0600 before SQLite opened it.
…ons prepare leaves to SQLite

A pool whose connection setup sets the database file and its sidecars to
0644 finds them at 0600 once connect returns. prepare creates nothing for
a file: URI with its own query or for a VFS outside the unix family.
…ation

A configured sqlite: URL names the same database file and sidecars as the
path inside it. The engine's old post-connect chmod appended the suffixes
to the configured string, which for a URL named files that do not exist.
The tests move unchanged from file_mode.rs to file_mode/tests.rs, so the
module stays under 600 lines as the tests grow.
…gh a handle that follows no link

The mode was set by path on whatever the database name and each sidecar
name resolved to. A database path naming a directory set the directory to
0600, a symlink planted at a sidecar name set its target to 0600, and a
read-only mount failed the read-only commands although every file was
already 0600.

Each name is now opened with O_PATH | O_NOFOLLOW on Linux, which opens no
file for reading or writing: a symlink yields a handle on the link itself
and a FIFO or a device is not opened, so nothing waits. The type and mode
are read from that handle, and only a regular file with a permission bit
outside 0600 is changed, by chmod of /proc/self/fd/<n>, so the check and
the change act on the same file. Closing an O_PATH handle keeps the POSIX
locks the process's SQLite connections hold on the file, which closing a
descriptor opened for reading would release. On other Unix systems the
type and mode come from lstat and the change is fchmodat with
AT_SYMLINK_NOFOLLOW, which opens nothing.

A regular file wider than 0600 that cannot be narrowed is still an error.
The calls that change a mode can be replaced in tests, which use that to
check the error without changing any file the test did not create.
…he locks of its open connection

A WAL connection holds a read lock on the database file's shared range and
on the DMS byte of the -shm file. The test narrows the files of a database
whose connection is open and asks, with an open file description lock,
whether a write lock on either range would still conflict.
…se file is created with

The integration tests opened every pool in sequence and checked the end
state, so a later opener's narrowing hid an opener that skipped its own.
Three tests now open one pool each on a missing path and read the modes
before any other pool opens the file: the bootstrapper's pool, the engine's
pool and the catalog pool through the settings store factory.

A unit test reads the mode of the database file prepare creates as the file
is created, before prepare narrows it, and compares it with a file the test
creates with mode 0600 under the same umask.
…ad them before creating the database

prepare decided whether to create the database file, and which file, from a
reading of the location that differed from sqlx's and SQLite's in four
ways. With several vfs values it required all of them to be in the unix
family, where sqlx uses the last. It left every file: URI with its own
query to SQLite, which created the file at 0644 when that query did not
narrow the open. database_file did not decode %HH in a file: URI path, so
prepare created a stray file and the database SQLite opened stayed 0644;
it matched mode=memory with a raw split, so an encoded key such as mod%65
made prepare create a file SQLite never used; and a file: name combined
with vfs or immutable, which sqlx wraps in a second file: URI so that
SQLite opens a relative path spelled file:..., got a stray file at the
path the URI names.

A new module, sqlite_uri, reads a file: URI as the bundled SQLite does:
the authority, the path and query split at raw separators, %HH decoded,
%00 ending a part, the last vfs, and each mode applied in order with the
rule that a mode may not ask for more access than the one before it.
prepare creates the file only when SQLite would create it on disk.
database_file reads the path through sqlite_uri and mode through
form_urlencoded, and file_mode leaves a file: name with vfs or immutable
alone. The lock file moves only where the decoding changes the path: for
a file: URI with %HH in its path it is now next to the file SQLite opens,
and an encoded mode=memory takes no lock, as every other in-memory form.
…hen one cannot be set

The security model left file permissions to the operator and did not
mention the SQLite database files, and the troubleshooting guide
described settings and verify as commands that only read. With the
SQLite backend, init, migrate, settings, verify and serve now create the
database file with mode 0600 and set it and its -wal, -shm and -journal
files to 0600 when they are wider, and they fail when a mode cannot be
set.

The security model now names those files. The lock entry in the
troubleshooting guide says that settings and verify set these modes
although they take no lock. A new entry covers the errors "cannot set
the mode of SQLite file" and "cannot set the mode of SQLite database"
and their causes: a read-only mount holding files wider than 0600, and
database files that belong to another user.
creates_file and the SQLITE_OPEN_* flags it reads are used only by the
Unix file mode code, so a Windows build reported seven dead-code
warnings for them. They are now compiled on Unix only, and the options
field they read allows dead code on other platforms.
The SQLite file message also appears when the database file or one of
its sidecars cannot be looked up, for example when the user cannot
search the directory that holds the database. The SQLite database
message comes from a filesystem that refuses to set the mode of the
file the command created. The troubleshooting entry now says so, and
lists the directory case with its fix.
@LeeroyHannigan LeeroyHannigan added bug Something isn't working security sqlite Related to the SQLite backend specifically. labels Oct 31, 2026
Cargo.lock: main added tracing-subscriber to the dependencies of
extenddb-storage-sqlite, and this branch added url. The merged file is
main's, with url added back by cargo.

crates/storage-sqlite/Cargo.toml: main added libsqlite3-sys and this
branch added url. The merged file keeps both.

crates/storage-sqlite/src/bootstrapper.rs: main replaced the pool's
pragmas with set_connection_pragmas, and this branch opens the pool
through file_mode::connect. The merged pool runs main's pragmas and
opens through file_mode::connect.

crates/storage-sqlite/src/lib.rs: main moved catalog_pool onto
set_connection_pragmas, and this branch opens it through
file_mode::connect and returns its error as a String. The merged
catalog_pool keeps main's pragmas and this branch's opener.

crates/storage-sqlite/src/store.rs: main moved the engine's pragmas onto
set_connection_pragmas and dropped the CATALOG_VERSION import. This
branch opens the engine's pool through file_mode::connect and removes
the chmod loop and the sqlite_url import. The merged engine keeps
main's pragmas and imports and this branch's opener, without the chmod
loop.

docs/troubleshooting.md: main names migrate --yes as the command that
takes the lock and adds migrate without --yes to the read-only
commands. This branch added a sentence on the modes settings and
verify set. The merged entry keeps main's text with this branch's
sentence at the end.

Main moved migrate --yes, and the first open of a dev-mode serve, onto
the migration runner's own connection, which this branch's openers do
not cover. SqliteMigrations::open_with now calls file_mode::prepare
and file_mode::restrict around that connection when it opens the
database for writing, so migrate --yes sets files left at 0644 to 0600
as this branch describes. The read-only status connection is
unchanged. file_mode::prepare and file_mode::restrict are pub(crate)
for this.

tests/file_modes.rs called bootstrapper methods that main removed. Its
init step now follows main's init (the migrations under the lock, then
the rows a server needs), its migrate step takes the lock and calls
run_locked, and the bootstrapper pool test opens the pool through
list_table_names.

The new troubleshooting entry and the security model name migrate
--yes where they named migrate, and say that a command that creates a
missing database file creates it with mode 0600. On main, migrate and
verify create no missing database file.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working security sqlite Related to the SQLite backend specifically.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

storage-sqlite: init creates the database and WAL sidecars at 0644; only serve tightens the main file

1 participant