feat!: pluggable storage backends for 5.0 by johnf · Pull Request #132 · crate-works/arocapi

johnf · 2026-10-21T06:53:18Z

Summary

A consumer passes one Storage backend instead of two handlers:

 await fastify.register(arocapi, {
   prisma, search, entityVisibilityFilter, searchVisibilityFilter, access,
-  fileHandler: { get, head },
-  roCrateHandler: { get, head },
+  storage: DirectoryStorage({ root: './data' }),
+  // or mix halves:
+  // storage: { metadata: EntityMetaMetadata(), content: (await OcflStorage({ root })).content },
 });
-import { OpenSearchBackend } from 'arocapi/opensearch';
+import { OpenSearchBackend } from 'arocapi/search/opensearch';

GET and HEAD make the same single call, after visibility and access:

sequenceDiagram
    participant Client
    participant Route as /file/:id, /entity/:id/metadata
    participant Half as storage.content / storage.metadata
    Client->>Route: GET or HEAD
    Route->>Route: load Record (visibility), enforce access
    Route->>Half: one call (file, entity with meta, contentDisposition)
    Half-->>Route: Delivery: stream | path | redirect | json (Metadata only) | null
    Route-->>Client: headers from storage, and bytes on GET only (302 for redirect, 404 for null, logged 500 for a throw)

 src/
+├── storage/
+│   ├── backend.ts          # contract: StorageBackend, deliveries, meta shapes (root exports)
+│   ├── deliver.ts          # turns a Delivery into the response
+│   ├── location.ts         # default meta keys or `locate`, refuses absolute / `..` Storage keys
+│   ├── results.ts          # missing → null, path deliveries from stat
+│   ├── directory/          # arocapi/storage/directory   DirectoryStorage
+│   ├── entity-meta/        # arocapi/storage/entity-meta EntityMetaMetadata
+│   ├── s3/                 # arocapi/storage/s3          S3Storage (presigned redirect, or stream)
+│   └── ocfl/               # arocapi/storage/ocfl        OcflStorage (LDaCA, 0002, 0003, 0004)
+│       └── ocflFs.ts       # @ocfl/ocfl-fs workarounds, removed under #117
-└── types/fileHandlers.ts

The decisions are in ADR 0004 and the glossary's "How it is stored" section. The docs cover Configuration → Storage (replacing Handlers), Reference → Storage backend, the Entity meta key table, Strategies → Storage and the tutorial.

Where this differs from the spec:

locate is one object, { metadata?(entity), content?(file, entity) }.
Metadata stream/path deliveries carry no contentType, because the route always sends application/ld+json.
A null meta value is a wrong shape (500) everywhere; only a missing key is a 404.

Evidence

Before: HEAD and GET were separate consumer functions, so they could disagree. raspbocrate's HEAD describes compact JSON while its GET sends it pretty-printed. Content-Length came from a possibly stale File.size.
After: a shared storage contract suite runs every reference backend through the full plugin. For Directory, S3 (redirect and stream) and OCFL it checks both halves; for EntityMetaMetadata, mixed with Directory Content, it checks the Metadata half:
```
for each backend × { metadata, content }
  GET  → stored bytes, Content-Type (File.mediaType | application/ld+json), Content-Disposition
  HEAD → headers equal to GET's
  S3 redirect → follow the presigned URL to VersityGW → same bytes and headers
  missing item → 404 · absolute or `..` key → 500 · custom locate → honoured
```
pnpm test: 972 tests pass, with 100% coverage on lines, branches, functions and statements, against Postgres, OpenSearch and VersityGW.
pnpm lint: biome, types, knip, the docs examples and the packed-tarball consumer check all pass. The consumer check proves that arocapi/search/* and arocapi/storage/* resolve, that arocapi/opensearch and arocapi/postgres no longer do, and that a content half can't return json.
pnpm --filter website build passes, including Mermaid parsing and internal link checks.

Merge Danger

Door: one-way

Every push to main publishes to npm under beta (see RELEASING.md). Because of the feat!: footers, this merge publishes 5.0.0-beta.1, and a published version can't be taken back. The code itself reverts in one step.

Blast Radius: consumers

What breaks for consumers: every consumer must replace fileHandler/roCrateHandler with storage, and change the search import paths. The BREAKING CHANGE: footers give the mapping. latest (4.x) isn't affected until 5.0 stable.
What every consumer gets: Content headers now come from storage, not File.size. Metadata is always application/ld+json. HEAD on a redirect answers 302.
Left as they are:
- A presigned S3 URL refuses HEAD with 403. This is documented, and follow-up S3Storage: presign a HeadObject URL for HEAD /file/:id #131 tracks a fix.
- There's no OCFL inventory cache.
- /file/:id loads the entity's full meta. With EntityMetaMetadata that includes the stored crate.
- S3 answers 403 for a missing key without s3:ListBucket. This is documented.
@ocfl/ocfl-fs workarounds: isolated for Before 5.0 stable: adopt upstream @ocfl/ocfl-fs fixes #117 and reported upstream in Storage layouts: 0003 encapsulation directory doesn't follow the spec, and hashed layouts from storageLayout() can't map until storage is loaded Language-Research-Technology/ocfl-js#8.
After merge: close docs: record pluggable storage backends in ADR 0004 and the glossary #122 unmerged; its ADR commit is the first commit here.

Closes #123, closes #124, closes #125, closes #126, closes #127, closes #128, closes #129, closes #130

Adds ADR 0004 (storage backends replace fileHandler and roCrateHandler) and a "How it is stored" glossary section (Storage backend, Storage key, Delivery), and rewords ADR 0002 to refer to storage backends. Refs #121

The search backends now sit under a grouped subpath beside storage. Export names are unchanged. BREAKING CHANGE: `arocapi/opensearch` and `arocapi/postgres` are removed with no aliases. Import `OpenSearchBackend` from `arocapi/search/opensearch` and `PostgresBackend` from `arocapi/search/postgres` instead.

The required `storage` option takes a storage backend, `{ metadata, content }`. Each half makes one call per request, shared by HEAD and GET, and returns a `stream`, `path`, `redirect` or (Metadata only) `json` delivery, or `null` for a 404. `content` receives the file's entity with its `meta` and the computed `contentDisposition`. HEAD on a redirect now answers 302. `DirectoryStorage({ root, locate? })`, from `arocapi/storage/directory`, serves both halves from below `root`, reading `meta.storagePath` relative to it. Absolute and `..` keys are refused. BREAKING CHANGE: `fileHandler` and `roCrateHandler` are replaced by the required `storage` option: `fileHandler` becomes `storage.content` and `roCrateHandler` becomes `storage.metadata`. Each half is a single function replacing the separate `head` and `get`, and returns `null` where a handler returned `false`. The `file` result is now a `path` delivery, and `accelPath` (`X-Accel-Redirect`) is gone. The `FileHandler`, `RoCrateHandler`, `GetFileHandler`, `HeadFileHandler`, `GetRoCrateHandler`, `HeadRoCrateHandler`, `FileResult`, `FileRedirectResult`, `FileStreamResult`, `FilePathResult` and `FileMetadata` exports are removed.

arocapi/storage/s3 exports S3Storage({ client, bucket, locate?, delivery?, expiresIn? }). Content redirects to a presigned GetObject URL carrying File.mediaType and the request's Content-Disposition, or streams through arocapi with delivery: 'stream'. Metadata always streams, reading headers with HeadObject and fetching that version with If-Match. @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner are optional peer dependencies. Tests run against VersityGW (test-s3) locally and in CI.

arocapi/storage/entity-meta exports EntityMetaMetadata({ key? }), a Metadata half that delivers the JSON object at Entity.meta.rocrate (or another key) as a json delivery. A missing key is a 404; a value that isn't a JSON object throws naming the entity id and key. It runs through the storage contract mixed with DirectoryStorage's content half, which gains a keylessMetadata harness option.

arocapi/storage/ocfl exports the async OcflStorage({ root, objectRoot?, locate? }). It reads ocfl_layout.json at startup, implements 000N-path-direct-storage-layout, and throws on any other layout unless objectRoot is given. Each request loads the object's inventory through @ocfl/ocfl-fs (an optional GPL-3.0 peer dependency) and serves the head version's file as a path delivery with a strong ETag from its inventory digest. The OCFL fixtures are built by scripts/buildOcflFixtures.ts with node:fs and node:crypto only, and checked in under src/test/fixtures/ocfl.

Compare DirectoryStorage, S3Storage and OcflStorage, set out where Metadata comes from, and the trade-offs of stream, path and redirect, including HEAD on a presigned redirect. Drop the nginx X-Accel-Redirect pattern.

…flStorage OcflStorage now maps object ids for 0002-flat-direct, 0003-hash-and-id-n-tuple and 0004-hashed-n-tuple storage roots from ocfl_layout.json and the extension config, without objectRoot. The fixture script builds a storage root for each layout with its own id mapping. Refs #129

… findings - One locator in src/storage/location.ts resolves meta keys or `locate` and checks Storage keys once; shared `inside`, `nullIfMissing` and `pathDelivery` helpers. - Metadata `stream`/`path` deliveries no longer carry `contentType`; the metadata route alone sends application/ld+json. - The delivery responder closes an opened body if headers fail, and refuses unknown delivery types. - A `null` meta value is a wrong shape (500) on every backend. - OCFL checks the logical path, not the object id, and refuses a manifest target that isn't a file; startup checks the root marker without listing the root. - The storage contract splits into Metadata and Content suites; faster S3, OCFL and route tests. - AWS SDK peers are optional and accept any v3. - Docs, ADR 0004 and CLAUDE.md match.

github-actions · 2026-10-21T06:56:06Z

Coverage Report

Status	Category	Percentage	Covered / Total
🟢	Lines	100% (🎯 100%)	1130 / 1130
🟢	Statements	100% (🎯 100%)	1190 / 1190
🟢	Functions	100% (🎯 100%)	271 / 271
🟢	Branches	100% (🎯 100%)	780 / 780

File Coverage

File	Stmts	Branches	Functions	Lines
Changed Files
src/app.ts	100%	100%	100%	100%
src/routes/file.ts	100%	100%	100%	100%
src/routes/metadata.ts	100%	100%	100%	100%
src/storage/deliver.ts	100%	100%	100%	100%
src/storage/location.ts	100%	100%	100%	100%
src/storage/results.ts	100%	100%	100%	100%
src/storage/directory/directory.ts	100%	100%	100%	100%
src/storage/entity-meta/entity-meta.ts	100%	100%	100%	100%
src/storage/ocfl/ocfl.ts	100%	100%	100%	100%
src/storage/ocfl/ocflFs.ts	100%	100%	100%	100%
src/storage/s3/s3.ts	100%	100%	100%	100%

Generated in workflow #138 for commit 48f6d23 by the Vitest Coverage Report Action

Copilot

🟡 Changes recommended

One or more issues must be addressed before approval.

2 open issues

1 file not reviewed

pnpm-lock.yaml: Generated file

What changed in this PR

Introduces pluggable storage for arocapi 5.0, replacing separate file and RO-Crate handlers with a shared Metadata/Content delivery contract.

Changes:

Adds storage contracts, delivery handling, and Directory, Entity-meta, S3 and OCFL support.
Migrates examples and documentation to storage and grouped search imports.
Adds storage test helpers, OCFL fixtures and S3 test infrastructure.

A root-containment issue was identified in DirectoryStorage: symlinks can expose files outside its configured root.

For brevity, table paths use these prefixes:

W = website/src/content/docs
E = website/examples
F = src/test/fixtures/ocfl
P = F/path-direct/https_example.com/storage
LONG = F/hash-and-id-n-tuple/f3/45/18/94/abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij-f3451894fc18f49bb7cb434c3d7a687bc6e2b82afc978649034abecd8f9ef19a27fee98f4b835469daef8b3f80d0437776caeee792e19d828c9ebcddc5ec9046

File	Description
website/src/content/docs/strategies/search.mdx	Updated as part of this pull request.
website/src/content/docs/strategies/access.mdx	Updated as part of this pull request.
website/src/content/docs/reference/routes.mdx	Updated as part of this pull request.
website/src/content/docs/reference/errors.mdx	Updated as part of this pull request.
website/src/content/docs/reference/entity-meta.mdx	Updated as part of this pull request.
website/src/content/docs/introduction/what-is-arocapi.mdx	Updated as part of this pull request.
website/src/content/docs/introduction/pilars.mdx	Updated as part of this pull request.
website/src/content/docs/introduction/crate-works.mdx	Updated as part of this pull request.
website/src/content/docs/getting-started/first-api.mdx	Updated as part of this pull request.
website/src/content/docs/configuration/transformers.mdx	Updated as part of this pull request.
website/src/content/docs/configuration/search.mdx	Updated as part of this pull request.
website/src/content/docs/configuration/data-sources.mdx	Updated as part of this pull request.
website/src/content/docs/configuration/access.mdx	Updated as part of this pull request.
website/src/content/docs/concepts/record-metadata-content.mdx	Updated as part of this pull request.
website/package.json	Updated as part of this pull request.
website/examples/strategies/storage-signed-url.ts	Updated as part of this pull request.
website/examples/strategies/storage-s3-delivery.ts	Updated as part of this pull request.
website/examples/strategies/storage-ro-crate.ts	Updated as part of this pull request.
website/examples/strategies/storage-presigned.ts	Updated as part of this pull request.
website/examples/strategies/storage-mixing-ocfl.ts	Updated as part of this pull request.
website/examples/strategies/storage-local.ts	Updated as part of this pull request.
website/examples/strategies/storage-accel.ts	Updated as part of this pull request.
website/examples/reference/storage-backend-signed-redirect.ts	Updated as part of this pull request.
website/examples/reference/entity-meta-storage-path.ts	Updated as part of this pull request.
website/examples/introduction/what-is-arocapi-options.ts	Updated as part of this pull request.
website/examples/getting-started/src/search.ts	Updated as part of this pull request.
website/examples/getting-started/src/first-api-login.ts	Updated as part of this pull request.
website/examples/getting-started/src/first-api-index.ts	Updated as part of this pull request.
website/examples/getting-started/scripts/first-api-seed.ts	Updated as part of this pull request.
website/examples/configuration/storage-s3.ts	Updated as part of this pull request.
website/examples/configuration/storage-ocfl.ts	Updated as part of this pull request.
website/examples/configuration/storage-mixing.ts	Updated as part of this pull request.
website/examples/configuration/storage-entity-meta.ts	Updated as part of this pull request.
website/examples/configuration/storage-directory.ts	Updated as part of this pull request.
website/examples/configuration/search-postgres.ts	Updated as part of this pull request.
website/examples/configuration/search-fields.ts	Updated as part of this pull request.
website/examples/configuration/handlers-ro-crate.ts	Updated as part of this pull request.
website/examples/configuration/handlers-file.ts	Updated as part of this pull request.
test/consumer/src/search.ts	Updated as part of this pull request.
src/types/fileHandlers.ts	Updated as part of this pull request.
src/test/integration.setup.ts	Updated as part of this pull request.
src/test/helpers/storedItems.ts	Updated as part of this pull request.
src/test/helpers/storageApp.ts	Updated as part of this pull request.
src/test/helpers/s3.ts	Updated as part of this pull request.
src/test/helpers/rows.ts	Updated as part of this pull request.
src/test/helpers/getAndHead.ts	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/ocfl_layout.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/unparseable/__object__/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/unparseable/__object__/v1/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/unparseable/__object__/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/unparseable/__object__/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/unparseable/__object__/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/unparseable/__object__/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/unparseable/__object__/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/unparseable/__object__/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/item/__object__/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/item/__object__/v1/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/item/__object__/v1/content/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/item/__object__/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/item/__object__/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/item/__object__/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/escaping-manifest/__object__/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/escaping-manifest/__object__/v1/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/escaping-manifest/__object__/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/escaping-manifest/__object__/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/escaping-manifest/__object__/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/escaping-manifest/__object__/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/escaping-manifest/__object__/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/escaping-manifest/__object__/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/directory-manifest/__object__/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/directory-manifest/__object__/v1/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/directory-manifest/__object__/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/directory-manifest/__object__/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/directory-manifest/__object__/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/directory-manifest/__object__/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/directory-manifest/__object__/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/directory-manifest/__object__/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/v2/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/v2/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/v2/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/v1/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/v1/content/collection/item/removed.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/extensions/0005-mutable-head/head/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/extensions/0005-mutable-head/head/content/r1/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/extensions/0005-mutable-head/config.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/contract/__object__/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/bad-sidecar/__object__/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/bad-sidecar/__object__/v1/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/bad-sidecar/__object__/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/bad-sidecar/__object__/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/bad-sidecar/__object__/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/bad-sidecar/__object__/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/bad-sidecar/__object__/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/https_example.com/storage/bad-sidecar/__object__/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/path-direct/0=ocfl_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/ocfl_layout.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/ff/75/53/44/92/48/5e/ab/b3/9f/86/35/67/28/88/4e/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/ff/75/53/44/92/48/5e/ab/b3/9f/86/35/67/28/88/4e/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/ff/75/53/44/92/48/5e/ab/b3/9f/86/35/67/28/88/4e/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/ff/75/53/44/92/48/5e/ab/b3/9f/86/35/67/28/88/4e/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/ff/75/53/44/92/48/5e/ab/b3/9f/86/35/67/28/88/4e/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/ff/75/53/44/92/48/5e/ab/b3/9f/86/35/67/28/88/4e/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/ff/75/53/44/92/48/5e/ab/b3/9f/86/35/67/28/88/4e/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/extensions/0004-hashed-n-tuple-storage-layout/config.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/71/dd/e7/a0/dc/2d/bd/5c/74/4b/8b/7e/e4/7c/83/f0/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/71/dd/e7/a0/dc/2d/bd/5c/74/4b/8b/7e/e4/7c/83/f0/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/71/dd/e7/a0/dc/2d/bd/5c/74/4b/8b/7e/e4/7c/83/f0/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/71/dd/e7/a0/dc/2d/bd/5c/74/4b/8b/7e/e4/7c/83/f0/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/71/dd/e7/a0/dc/2d/bd/5c/74/4b/8b/7e/e4/7c/83/f0/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/71/dd/e7/a0/dc/2d/bd/5c/74/4b/8b/7e/e4/7c/83/f0/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/20/1a/a2/24/55/33/b3/9a/49/72/70/cc/51/92/88/e8/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/20/1a/a2/24/55/33/b3/9a/49/72/70/cc/51/92/88/e8/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/20/1a/a2/24/55/33/b3/9a/49/72/70/cc/51/92/88/e8/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/20/1a/a2/24/55/33/b3/9a/49/72/70/cc/51/92/88/e8/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/20/1a/a2/24/55/33/b3/9a/49/72/70/cc/51/92/88/e8/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/20/1a/a2/24/55/33/b3/9a/49/72/70/cc/51/92/88/e8/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/20/1a/a2/24/55/33/b3/9a/49/72/70/cc/51/92/88/e8/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/08/31/97/66/fb/6c/29/35/dd/17/5b/94/26/77/17/e0/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/08/31/97/66/fb/6c/29/35/dd/17/5b/94/26/77/17/e0/v1/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/08/31/97/66/fb/6c/29/35/dd/17/5b/94/26/77/17/e0/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/08/31/97/66/fb/6c/29/35/dd/17/5b/94/26/77/17/e0/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/08/31/97/66/fb/6c/29/35/dd/17/5b/94/26/77/17/e0/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/08/31/97/66/fb/6c/29/35/dd/17/5b/94/26/77/17/e0/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/08/31/97/66/fb/6c/29/35/dd/17/5b/94/26/77/17/e0/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/08/31/97/66/fb/6c/29/35/dd/17/5b/94/26/77/17/e0/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/hashed-n-tuple/0=ocfl_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/ocfl_layout.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/f3/45/18/94/abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij-f3451894fc18f49bb7cb434c3d7a687bc6e2b82afc978649034abecd8f9ef19a27fee98f4b835469daef8b3f80d0437776caeee792e19d828c9ebcddc5ec9046/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/f3/45/18/94/abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij-f3451894fc18f49bb7cb434c3d7a687bc6e2b82afc978649034abecd8f9ef19a27fee98f4b835469daef8b3f80d0437776caeee792e19d828c9ebcddc5ec9046/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/f3/45/18/94/abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij-f3451894fc18f49bb7cb434c3d7a687bc6e2b82afc978649034abecd8f9ef19a27fee98f4b835469daef8b3f80d0437776caeee792e19d828c9ebcddc5ec9046/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/f3/45/18/94/abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij-f3451894fc18f49bb7cb434c3d7a687bc6e2b82afc978649034abecd8f9ef19a27fee98f4b835469daef8b3f80d0437776caeee792e19d828c9ebcddc5ec9046/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/f3/45/18/94/abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij-f3451894fc18f49bb7cb434c3d7a687bc6e2b82afc978649034abecd8f9ef19a27fee98f4b835469daef8b3f80d0437776caeee792e19d828c9ebcddc5ec9046/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/f3/45/18/94/abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij-f3451894fc18f49bb7cb434c3d7a687bc6e2b82afc978649034abecd8f9ef19a27fee98f4b835469daef8b3f80d0437776caeee792e19d828c9ebcddc5ec9046/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/extensions/0003-hash-and-id-n-tuple-storage-layout/config.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/d3/60/1f/87/object-01/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/d3/60/1f/87/object-01/v1/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/d3/60/1f/87/object-01/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/d3/60/1f/87/object-01/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/d3/60/1f/87/object-01/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/d3/60/1f/87/object-01/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/d3/60/1f/87/object-01/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/d3/60/1f/87/object-01/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/ab/a9/bf/5d/info%3afedora%2fObject%7e01%21%27%28%29%2a/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/ab/a9/bf/5d/info%3afedora%2fObject%7e01%21%27%28%29%2a/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/ab/a9/bf/5d/info%3afedora%2fObject%7e01%21%27%28%29%2a/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/ab/a9/bf/5d/info%3afedora%2fObject%7e01%21%27%28%29%2a/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/ab/a9/bf/5d/info%3afedora%2fObject%7e01%21%27%28%29%2a/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/ab/a9/bf/5d/info%3afedora%2fObject%7e01%21%27%28%29%2a/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/ab/a9/bf/5d/info%3afedora%2fObject%7e01%21%27%28%29%2a/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/3a/9f/56/a7/%2e%2eHor%2frib%3al%c3%a8-%24id/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/3a/9f/56/a7/%2e%2eHor%2frib%3al%c3%a8-%24id/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/3a/9f/56/a7/%2e%2eHor%2frib%3al%c3%a8-%24id/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/3a/9f/56/a7/%2e%2eHor%2frib%3al%c3%a8-%24id/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/3a/9f/56/a7/%2e%2eHor%2frib%3al%c3%a8-%24id/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/3a/9f/56/a7/%2e%2eHor%2frib%3al%c3%a8-%24id/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/3a/9f/56/a7/%2e%2eHor%2frib%3al%c3%a8-%24id/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/hash-and-id-n-tuple/0=ocfl_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/ocfl_layout.json	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/object-01/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/object-01/v1/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/object-01/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/object-01/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/object-01/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/object-01/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/object-01/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/object-01/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/0=ocfl_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/..hor_rib:lé-$id/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/..hor_rib:lé-$id/v1/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/..hor_rib:lé-$id/v1/content/collection/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/..hor_rib:lé-$id/v1/content/collection/item/notes.bin	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/..hor_rib:lé-$id/v1/content/collection/item/greeting.wav	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/..hor_rib:lé-$id/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/..hor_rib:lé-$id/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/flat-direct/..hor_rib:lé-$id/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/custom-layout/ocfl_layout.json	Updated as part of this pull request.
src/test/fixtures/ocfl/custom-layout/objects/custom/v1/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/custom-layout/objects/custom/v1/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/custom-layout/objects/custom/v1/content/ro-crate-metadata.json	Updated as part of this pull request.
src/test/fixtures/ocfl/custom-layout/objects/custom/inventory.json.sha512	Updated as part of this pull request.
src/test/fixtures/ocfl/custom-layout/objects/custom/inventory.json	Updated as part of this pull request.
src/test/fixtures/ocfl/custom-layout/objects/custom/0=ocfl_object_1.1	Updated as part of this pull request.
src/test/fixtures/ocfl/custom-layout/0=ocfl_1.1	Updated as part of this pull request.
src/storage/s3/s3.ts	Updated as part of this pull request.
src/storage/results.ts	Updated as part of this pull request.
src/storage/ocfl/ocflFs.ts	Updated as part of this pull request.
src/storage/location.ts	Updated as part of this pull request.
src/storage/entity-meta/entity-meta.ts	Updated as part of this pull request.
src/storage/entity-meta/entity-meta.test.ts	Updated as part of this pull request.
src/storage/directory/directory.ts	Updated as part of this pull request.
src/storage/directory/directory.test.ts	Updated as part of this pull request.
src/storage/deliver.ts	Updated as part of this pull request.
src/storage/backend.ts	Updated as part of this pull request.
src/routes/capabilities.test.ts	Updated as part of this pull request.
src/errorHandling.test.ts	Updated as part of this pull request.
src/app.test.ts	Updated as part of this pull request.
package.json	Updated as part of this pull request.
GLOSSARY.md	Updated as part of this pull request.
example/data/seed.ts	Updated as part of this pull request.
example/data/README.md	Updated as part of this pull request.
docs/adr/0003-pluggable-search-backends.md	Updated as part of this pull request.
docs/adr/0002-library-enforces-access.md	Updated as part of this pull request.
docker-compose.test.yml	Updated as part of this pull request.
CLAUDE.md	Updated as part of this pull request.
biome.jsonc	Updated as part of this pull request.
.github/workflows/test.yml	Updated as part of this pull request.

_{Review effort: Balanced}

…e followed

github-actions · 2026-10-21T23:25:33Z

🎉 This PR is included in version 5.0.0-beta.7 🎉

The release is available on:

Your semantic-release bot 📦🚀

johnf added 9 commits October 21, 2026 17:52

docs: record pluggable storage backends in ADR 0004 and the glossary

46c568f

Adds ADR 0004 (storage backends replace fileHandler and roCrateHandler) and a "How it is stored" glossary section (Storage backend, Storage key, Delivery), and rewords ADR 0002 to refer to storage backends. Refs #121

docs: rewrite Strategies → Storage around the reference backends

d158136

Compare DirectoryStorage, S3Storage and OcflStorage, set out where Metadata comes from, and the trade-offs of stream, path and redirect, including HEAD on a presigned redirect. Drop the nginx X-Accel-Redirect pattern.

johnf requested review from moisbo and a balanced review from Copilot October 21, 2026 22:49

Copilot started reviewing on behalf of johnf October 21, 2026 22:49

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread src/storage/directory/directory.ts Resolved

Comment thread src/storage/location.ts Resolved

docs: state that storage roots are trusted and symlinks below them ar…

48f6d23

…e followed

johnf merged commit d5431c6 into main Oct 21, 2026
3 checks passed

johnf deleted the spec/123-pluggable-storage branch October 21, 2026 23:18

github-actions Bot added the released on @beta label Oct 21, 2026

Navigation Menu

Repository navigation

feat!: pluggable storage backends for 5.0 - #132

feat!: pluggable storage backends for 5.0#132
johnf merged 10 commits into
mainfrom
spec/123-pluggable-storage

johnf commented Oct 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 21, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

johnf commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Evidence

Merge Danger

Uh oh!

github-actions Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 21, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

johnf commented Oct 21, 2026 •

edited

Loading

github-actions Bot commented Oct 21, 2026 •

edited

Loading