perf(build): write the database in one transaction

[jeffreylouden]

Part of AUREO-1470

Summary

• scripts/build-sqlite.ts: populateDatabase runs inside one db.transaction(...)() with synchronous = OFF, after schema.sql has switched the file to WAL. VACUUM and ANALYZE move to buildDatabase after the commit (they cannot run inside a transaction). On error the handle is closed, the partial file is removed and the error rethrown, so a failed build never leaves a database that looks complete.
• .github/workflows/url-health.yml: git add -f on .github/url-cache.json. The file is in .gitignore for local pnpm validate-urls runs, so git add without -f exited 1 under set -e and the weekly cache has never been committed; every Monday rechecked about 28k URLs cold inside the 60-minute timeout.
• CLAUDE.md: a bullet under Catalog Schema Compatibility recording why the schema runs before the transaction and why journal_mode = OFF is not an option (better-sqlite3's defensive mode answers it with delete).

Measured

Full corpus, 21,164 entries, 17,075 products, time pnpm build:

	seconds
before	43.1
after	23.2

The remaining time is YAML parsing and markdown rendering, not SQLite.

Verified against a build from the original code: sqlite_master identical (187 objects), every table count equal, FTS queries identical, integrity_check ok. Only created_at/updated_at defaults and FTS5 internal segment tables differ. The WAL flag in the file header (bytes 18 and 19) is unchanged, so Studio reads the file as before. No schema change, no schema_version bump.

Test plan

• pnpm build, pnpm test (37 files, 761 tests), pnpm typecheck, pnpm validate, pnpm format:check pass locally
• CI validate and build jobs green on this PR
• Next Monday's URL health run commits chore: update URL cache

From the 2026-10-16 import pipeline review (racks session).

🤖 Generated with Claude Code

https://claude.ai/code/session_013X2d8vEWjRnkZydpXo94JF

---

Generated by Claude Code

Summary by CodeRabbit

• Improvements
  • Database builds now load records in a single transaction and apply final optimization steps afterward, improving build efficiency.
  • If a build fails, incomplete database output is removed rather than left behind.
  • Validated URL-health cache updates are now preserved during automated commits. These changes improve build and maintenance workflows without adding new end-user functionality.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 3a1df6e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: existential-engineering/catalog/.coderabbit.yaml
• Review profile: CHILL
• Plan: Advanced
• Run ID: 12bff0c8-6902-45e6-bbdd-e6ec076f05ee

📥 Commits

Reviewing files that changed from the base of the PR and between c7cbc66 and 3a1df6e.

📒 Files selected for processing: 3

• .github/workflows/url-health.yml
• CLAUDE.md
• scripts/build-sqlite.ts

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour.

Walkthrough

The URL health workflow now force-stages its ignored cache file. The SQLite build applies the schema before population, inserts rows in one transaction, runs optimization afterward, and closes the database and deletes the output file if the build fails.

Changes

URL Cache Staging

Layer / File(s)	Summary
Force-stage the URL cache .github/workflows/url-health.yml	The commit step uses git add -f to stage the ignored cache file after the existing checks.

SQLite Database Build

Layer / File(s)	Summary
Transactional database build scripts/build-sqlite.ts, CLAUDE.md	buildDatabase applies the schema before population, runs inserts in one transaction with synchronous writes disabled, then runs VACUUM and ANALYZE. On error, it closes the database, deletes the output file, and rethrows. populateDatabase no longer applies the schema or runs optimization. The documentation records this build flow.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant buildDatabase
  participant SQLite
  participant populateDatabase
  participant outputFile
  buildDatabase->>SQLite: Apply schema and disable synchronous writes
  buildDatabase->>SQLite: Start transaction
  buildDatabase->>populateDatabase: Populate rows
  populateDatabase-->>SQLite: Insert rows within caller transaction
  buildDatabase->>SQLite: Run VACUUM and ANALYZE after transaction
  alt Build error
    buildDatabase->>SQLite: Close database
    buildDatabase->>outputFile: Delete partial output file
  end

Loading

Merge Risk: ⚪ Minimal · up to 3a1df

The cache is generated by the scheduled or manually triggered validator, and the database build commits population before optimization and cleans up caught failures. No material user-facing failure is indicated, so it is ready to merge under normal checks.

Security architecture risk: 🔵 Low · up to 3a1df

Cache publication now succeeds, making its existing validation weakness relevant to future runs. The exposure is bounded to shared URL-check data: ordinary HTTP responses cannot directly choose arbitrary cache lifetimes, and no new path to repository code execution was established. The SQLite transaction change improves handled-failure containment.

Retained concerns

• Medium · security · inferred: Force-staging activates durable publication behind a container-only validation gate. If an attacker gains control of the producer-written cache, identity-consistent entries with forged statuses and excessive finite TTLs can be committed and reused. This expands persistence of the retained validation weakness; it does not establish that ordinary URL responses can forge those fields.

Security review details

Security Blast Radius

• inferred — Control of the producer-written artifact could affect the entire shared cache map and subsequent URL-check results, not merely one attacker-hosted URL. The observed publication commands stage one fixed cache path; the trace does not establish arbitrary repository-file writes, credential disclosure, code execution, or cross-tenant exposure.

Security Findings and Attack Paths

• inferred — The retained finding concerns insufficient validation of cache entries. The PR adds durable publication and future reuse of that data. A forged status with an excessive finite TTL could suppress rechecks if an attacker can write the artifact. Ordinary response handling instead derives TTL locally, so the inspected source does not establish arbitrary cache forgery from external HTTP input alone. Raw cache consumption by the issue-reporting step already existed before this PR.

Trust Boundaries and Controls

• observed — The workflow runs on schedule or manual dispatch, not pull requests. The producer has contents-read and issues-write permissions; only the separate commit job has contents-write. The commit job installs no dependencies, validates the downloaded JSON container, stages the fixed cache path, and disables hooks for commit and push. These controls constrain provenance and authority but do not enforce the entry-level cache policy.

Resilience and Maintainability Implications

• inferred — Failed or overlapping runs need not publish their latest cache update: the handoff accepts an existing file after validation failure, and unsuccessful pushes are only logged. These states limit confidence in cache freshness and recovery, but do not independently establish a new security-control bypass.

Hardening Proposals

• proposed — Enforce a data-only entry contract at privileged publication: validate key-to-URL identity, field types, timestamps, allowed statuses, and bounded status-derived TTLs. This would reduce reliance on producer integrity without executing repository code in the write-capable job.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the primary change: writing the database in one transaction to improve build performance.
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. (2 skipped: 2 …
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}