Add file storage for resources behind a pluggable object store

[dborovcanin]

Closes #127.

What

Files (profile photos, product images, documents) stored beside Atom's identities and policies: metadata and access control are Atom's, bytes live in a pluggable object store.

• A file is a resource of kind file: tenant, owner (uploader by default), object groups, attributes, soft delete/restore, audit and resource.* events (with kind: "file", size and type in the details).
• Server-owned storage metadata in file_objects (migration 005, PostgreSQL + SQLite, native repository adapters). The storage key is never an attribute, so no GraphQL or REST input can point a file at another object.
• Provider-agnostic storage (src/storage/): a BlobStore trait with Atom-owned types (BlobKey, BlobMeta, BlobError, byte stream), BlobError → AppError in one place, and a StorageResolver::for_tenant seam for per-tenant buckets later.
  • memory adapter (tests; always compiled).
  • object_store adapter behind Cargo features: storage-s3 (default, includes local; MinIO/R2/SeaweedFS/B2/Wasabi via endpoint), storage-gcs, storage-azure. Multipart for large bodies, aborted on failure.
  • Shared conformance suite (storage::conformance::run) run against memory, local, object_store's InMemory, and S3 when ATOM_TEST_S3_BUCKET is set.
  • check-db-boundary.sh now also fails if any code outside the adapter names object_store::.
  • Startup writes, reads back and deletes a probe object; an unreachable store fails startup.
• REST (mounted only when ATOM_STORAGE_BACKEND is set; GraphQL carries no bytes):
  • POST /files — streamed upload, ATOM_FILE_MAX_BYTES enforced from Content-Length and while streaming, SHA-256 on the way through, tenant quota checked under the tenant lock. Needs manage/write in the tenant.
  • GET /files/{id} — public, read on the file/tenant, or a signed URL. ETag = SHA-256, If-None-Match, single byte Range.
  • PUT /files/{id} (write) replaces bytes, same id/URL, new ETag. DELETE /files/{id} (delete) soft-deletes.
  • POST /files/{id}/signed-url — HMAC-SHA256 under a key derived from the KEK, bounded by ATOM_FILE_SIGNED_URL_MAX_TTL_SECS; verification touches no database.
  • Shares the custom_endpoints rate-limit bucket (application traffic).
• Content safety: type sniffed from bytes (PNG, JPEG, GIF, WebP, PDF); a file declaring one of those without its signature is refused; other types need ATOM_FILE_ALLOWED_TYPES. Every download sends nosniff; only raster images and PDF are inline, everything else (SVG, HTML) is an attachment with Content-Security-Policy: sandbox.
• Consistency: a blob_deletions queue. Uploads queue their own key before writing and unqueue it in the committing transaction; triggers on file_objects queue old keys on replacement and on every physical delete (resource purge, retention purge, tenant purge cascades). A worker claims keys older than the grace period, deletes them, and requeues failures. An upload whose key was already claimed fails with 503 and requeues it, so bytes are either referenced or collected, never both.
• GraphQL: read-only Resource.file { sizeBytes contentType sha256 public url updatedAt }.
• Contracts/docs: OpenAPI, GraphQL SDL, 22 new env vars in deployment-config.json (count test 168 → 190), re-pinned contracts-v1.0.0.sha384, new operations/file-storage docs page (config, Supabase comparison, authorization, key layout, export, deletion, adding a provider), README and AGENTS.md.
• Container image: ships an atom-owned /app/data to mount a volume on (e.g. ATOM_STORAGE_LOCAL_PATH=/app/data/files).

Departures from the issue

• No pending rows or separate sweeper. The deletion queue covers unfinished uploads: each key is queued before it is written and unqueued on commit, so one worker handles abandoned uploads, replacements and purges.
• Public caching is public, max-age=300, not immutable: replacement keeps the URL, so an immutable response would pin stale bytes for a year. The ETag makes revalidation cheap.
• Replace/delete also accept manage, consistent with the GraphQL resource mutations.
• No action_applicability change: resources already allow read/write/delete/manage.
• Bootstrap YAML unchanged: storage is env-only, so no secrets can land in YAML.
• No MinIO CI job: the S3 conformance test runs when ATOM_TEST_S3_BUCKET is set and skips otherwise.

Testing

• tests/m53_file_storage.rs (11 tests): round trip, sniffing, 304/206, public vs private, signed URLs (other file, tampered, expired, TTL bounds), tenant permissions, SVG as sandboxed attachment, oversized with and without length, quota under 6 concurrent uploads, replacement + worker, attributes cannot redirect a file, soft delete/restore keeps bytes, resource and tenant purge delete bytes, the upload/grace-period race, routes absent without storage.
• Full DB-gated suite, CI-style (fresh database per binary), on PostgreSQL and SQLite: all pass except m41_pki_est, which needs the external ATOM_EST_CLIENT locally.
• fmt, clippy (default and --no-default-features), boundary/parity check, v1 contract gate and regressions, GraphQL SDL diff.

[dborovcanin]

Reviewed c59fa7e...f519ba4. Three verified findings below.

Validation: all 11 existing file integration tests passed on SQLite; three focused review probes reproduced the authorization and deletion failures. Six storage unit/conformance checks passed (memory, object_store InMemory, local); live S3 was not run. PostgreSQL was inspected but not rerun locally. Rust/API CI is green. The frontend job fails at pnpm audit on the unchanged Next.js dependency, separately from these changes.

[dborovcanin]

Re-reviewed the update f519ba4..5bcf41b against all three prior findings. All three are verified fixed; posted verification details and resolved their threads. No new blocking findings in the update.

Validation: all 14 file integration tests passed on SQLite, including policy denies, inactive/deleted-tenant downloads, interrupted deletion recovery, and the upload/grace-period race. Six memory/local storage unit and conformance checks passed; cargo fmt --check and database/storage boundary/schema parity checks passed. PostgreSQL changes were inspected but not executed locally; live S3 was not exercised. CI is still in progress. No approval submitted.

[arvindh123]

Approved for the feature merge at 88c7cac, with the four deferred reliability and observability findings tracked in #130.

The follow-up covers incomplete multipart upload cleanup, returning replacement metadata before commit, failure observation for file mutations, and non-blocking local storage setup. Multipart cleanup is the most urgent before sustained production S3 use; this approval does not establish production readiness.

Validation: local formatting, patch/whitespace, and database/storage boundary/schema parity checks passed. Rust and API/docs CI passed. Local Rust tests could not run because the pinned object_store dependency was unavailable offline; live S3 was not exercised. The UI job still fails its dependency audit on the unchanged Next.js dependency and remains a separate CI concern.

[felixgateru]

All review comments addressed in follow up issue:
#130