Repository navigation
feat!: pluggable search backends for 5.0 - #108
Merged
Merged
Conversation
Switch the Prisma datasource to postgresql, regenerate the migrations from scratch with plain text columns, and replace @prisma/adapter-mariadb with @prisma/adapter-pg. Test Compose and CI run postgres:18, and the integration setup and CI fail if the migrations drift from the schema. Fixtures and docs use Postgres JSON paths. BREAKING CHANGE: arocapi requires PostgreSQL 18 or later; MySQL and MariaDB are no longer supported. Consumers must switch their Prisma datasource to postgresql and use @prisma/adapter-pg. MySQL users stay on 4.x.
A consumer declares search fields once and indexes search documents with
the exported OpenSearchBackend({ client, fields, indexName }). The factory
throws on a bad declaration, createIndex() builds the mapping from the
declarations and errors on an existing index whose mapping differs, and
index/remove upsert and remove idempotently. Declaration and document
validation, and capabilitiesFromFields, live in src/search/fields.ts for
the Postgres backend to share. Not wired into /search yet.
EntitySearch ships in the shared prisma/models, cascading from Entity. It
holds the declared values as jsonb, a weighted tsvector over basic fields, a
tsvector of every text field's lexemes prefixed with the field name, a point
behind GiST and a full-length geohash, all GIN/GiST through @@index with no
extensions.
PostgresBackend({ prisma, fields }) shares declaration checks and document
validation with OpenSearchBackend. index(docs, { tx }?) upserts a batch in
one parameterised statement and, when entities are missing, writes nothing
and throws naming them, leaving any transaction usable. remove(ids) is
idempotent.
/search now goes through the required `search` option, a SearchBackend
with capabilities, search, index and remove. arocapi validates each
request against the backend's capabilities (400 for undeclared filter
keys, sorts and search types), passes the backend a normalised
SearchRequest with the search visibility filter's result as
`visibility`, maps SearchQueryError to 422, then loads the hits under
entityVisibilityFilter, attaches access, runs the transformers and
times searchTime around the backend call. Every declared facet is in
every response.
OpenSearchBackend gains search: multi_match boosted by weight, exact
term filters and facets keyed by field name, declared sorts tie-broken
by id, and exact totals. OpenSearch 400s become SearchQueryError.
/capabilities reports RO-Crate API v0.5.0 with search taken from the
backend's capabilities, including searchTypes, geo and sorts.
Route tests use a fake SearchBackend, and the integration tests run a
shared search contract suite through OpenSearchBackend, seeded with
createIndex() and index(). The dev loaders and docs use createIndex()
and index(), and the query-builder docs are gone.
BREAKING CHANGE: the `opensearch`, `queryBuilderClass` and
`queryBuilderOptions` options are removed; pass
`search: OpenSearchBackend({ client, fields })` instead, declaring your
search fields and creating the index with `createIndex()`. The
`OpensearchQueryBuilder` export and its `QueryBuilderOptions`,
`FilterCapability`, `FacetCapability` and `SearchCapability` types are
removed; use field declarations, `BackendSearchCapability` and
`capabilitiesFromFields`, or write your own `SearchBackend`.
BREAKING CHANGE: `POST /search` only accepts the filter keys and sorts
the backend declares; the fixed `id`, `name`, `createdAt` and
`updatedAt` sorts are replaced by declared `sort` fields. Facets are
keyed by field name and present for every declared facet, and
`searchExtra` is whatever the backend returns.
BREAKING CHANGE: `GET /capabilities` reports `apiVersion` 0.5.0, adding
`search.sorts`, `search.searchTypes` and `search.geo`.
POST /search accepts, per date or number filter, exact values, an
inclusive { gte, lte } range or an array of ranges, any of which
matches. arocapi splits filters into exact terms (string and boolean
filters) and ranges (date and number filters, exact values included),
with calendar-date bounds covering their whole UTC day and date-times
read as UTC without an offset. A partial date, a wrong bound type,
gte above lte, or a range on a string or boolean filter is a 400
VALIDATION_ERROR naming the filter.
boundingBox and geohashPrecision are a 400 unless the backend supports
geo; geohashPrecision needs a boundingBox, and an upside-down box is a
400. OpenSearchBackend filters by the box (including across the
antimeridian), returns a geohash grid, buckets date facets by UTC year
named "YYYY" and returns highlights in searchExtra.
SearchRequest.filters.ranges is now Record<string, RangeFilter[]>.
arocapi parses advanced queries itself (field: value, phrases, AND, OR, unary NOT, brackets, field groups) against the declared advanced fields, and enforces fixed caps of 1024 code points, 10 levels and 64 clauses. Syntax errors, undeclared fields and cap breaches are a 422 on query. OpenSearchBackend translates the tree to bool/match/match_phrase/term. Closes #102
searchVisibilityFilter returns a SearchVisibilityWhere over id and the
declared search fields, typed from them, instead of an OpenSearch query.
Search backends check it on every search, throwing an Error naming the
field on a fault, and OpenSearchBackend translates it to a bool filter.
AllVisibleSearchFilter returns {}. AND, OR and NOT are reserved field
names, and every text field gets an exact form in the OpenSearch mapping.
BREAKING CHANGE: `searchVisibilityFilter` OpenSearch query → the shared
search visibility filter language (`SearchVisibilityWhere`).
PostgresBackend.search runs parameterised SQL over EntitySearch: basic websearch_to_tsquery with ts_rank relevance tie-broken by id, exact-term and number/date range filters, value facets, declared sorts with missing values last, exact totals and the search visibility filter translated from the shared checker's tree. The integration tests, including the shared search contract, now run once with OpenSearchBackend and once with PostgresBackend. The contract's advanced-query and dates/geo blocks are skipped on Postgres until those are implemented, and PostgresBackend doesn't publish advanced search yet. Strategies -> Search compares the backends and covers switching to Postgres and writing your own; Configuration -> Search documents the PostgresBackend options.
Yearly UTC date facets, bounding box filters on the stored point (GiST, no PostGIS) and a geohash grid by prefixes of the stored geohash. The shared contract suite's dates, numbers and geo block now runs on Postgres too. Closes #107
Advanced query trees run as a SQL boolean over the field-prefixed text tsvector, with phraseto_tsquery for phrases and exact matches for other types. Values a field can't hold and Postgres program limits give a SearchQueryError (422), matching OpenSearch.
… backends - Request filters become one VisibilityCondition (filtersCondition), translated by each backend's visibility translator; drops the hand-built terms/range DSL and SQL - fields.ts resolves declarations (basic weights, facet sizes, geo field), exports storeScalar (reused by the visibility checker), toWire and withId; stored documents carry the geo point separately - BackendSearchCapability and SearchQuery live in backend.ts - Postgres: geohash moves to postgresGeo, the text search config to postgresValues; isolated test schemas apply each migration in one execute perf: Postgres search plans - Equality and presence use jsonb @? so the values GIN index (now jsonb_path_ops) serves them; id conditions use the primary key - Hits page is a top-N sort; rank is only computed for relevance sorts - Facets dedupe values per row and count(*) instead of count(DISTINCT) - The search route starts the backend search without waiting on the entity visibility filter, whose error still takes precedence
- Drop highlights from a hit whose Metadata or Content access is denied - Log the backend error when the entity visibility filter fails too - Reject boolean filter values other than true/false with a 400 - A basic query with no words left (e.g. `*`) matches every visible entity - An advanced date match covers the whole period the value names, via a shared dateSpan helper - PostgresBackend.index names missing entities on a foreign key violation from a concurrent delete - Contract cases for the advanced query caps; OpenSearch clause-limit test
… backends A custom search backend can check the search visibility filter against its field declarations and apply the normalised condition tree. The custom backend example now does, and PostgresBackend is checked against the full SearchBackend contract.
index and remove no longer force a refresh on every bulk write. Pass `refresh: true` or `'wait_for'` when a write must be searchable at once. The dev loader indexes each fetched page in one call.
Coverage Report
File Coverage
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved visibility validation, aggregation accuracy, schema handling, and example compatibility issues need correction.
6 open issues
1 file not reviewed
pnpm-lock.yaml: Generated file
What changed in this PR
Introduces pluggable search for arocapi 5.0, supporting OpenSearch, PostgreSQL, and custom backends through a shared public contract.
Changes:
- Adds field-driven search, validation, visibility filters, advanced queries, facets, and geo support.
- Replaces MySQL with PostgreSQL 18 and adds search storage.
- Updates integration tests, consumer examples, documentation, and development tooling.
| File | Description |
|---|---|
| website/src/content/docs/strategies/search.mdx | Explains backend selection and search flow. |
| website/src/content/docs/reference/search-queries.mdx | Documents advanced grammar and limits. |
| website/src/content/docs/reference/search-backend.mdx | Defines the custom-backend contract. |
| website/src/content/docs/reference/routes.mdx | Updates search and capabilities references. |
| website/src/content/docs/reference/errors.mdx | Documents search validation errors. |
| website/src/content/docs/reference/entity-meta.mdx | Updates visibility-field guidance. |
| website/src/content/docs/introduction/what-is-arocapi.mdx | Introduces backend-based search. |
| website/src/content/docs/index.mdx | Updates the search summary. |
| website/src/content/docs/getting-started/opensearch.mdx | Demonstrates field-driven indexing. |
| website/src/content/docs/getting-started/install.mdx | Requires PostgreSQL and its adapter. |
| website/src/content/docs/getting-started/first-api.mdx | Updates required plugin options. |
| website/src/content/docs/getting-started/database.mdx | Uses PostgreSQL connection settings. |
| website/src/content/docs/configuration/visibility.mdx | Documents shared visibility conditions. |
| website/src/content/docs/configuration/http.mdx | Generalizes backend-error guidance. |
| website/src/content/docs/configuration/data-sources.mdx | Updates Prisma configuration guidance. |
| website/src/content/docs/concepts/visibility-vs-access.mdx | Updates search visibility terminology. |
| website/src/content/docs/concepts/request-lifecycle.mdx | Describes backend request processing. |
| website/package.json | Replaces the database adapter. |
| website/examples/strategies/search-query-builder.ts | Removes the obsolete subclass example. |
| website/examples/strategies/search-drafts.ts | Uses typed draft visibility conditions. |
| website/examples/strategies/search-aggregations.ts | Removes query-builder aggregation configuration. |
| website/examples/strategies/access-hidden.ts | Uses shared visibility expressions. |
| website/examples/reference/search-backend-custom.ts | Adds a custom read-only backend. |
| website/examples/reference/entity-meta-draft.ts | Updates database and search filters. |
| website/examples/introduction/what-is-arocapi-options.ts | Demonstrates new plugin configuration. |
| website/examples/getting-started/src/search.ts | Defines a reusable search backend. |
| website/examples/getting-started/src/first-api-login.ts | Registers the search backend. |
| website/examples/getting-started/src/first-api-index.ts | Uses PostgreSQL and shared search setup. |
| website/examples/getting-started/scripts/first-api-seed.ts | Indexes through the backend. |
| website/examples/getting-started/scripts/create-index.ts | Creates the declared index. |
| website/examples/configuration/visibility-collections.ts | Converts collection visibility conditions. |
| website/examples/configuration/search-query-builder.ts | Removes obsolete capability customization. |
| website/examples/configuration/search-postgres.ts | Demonstrates transactional search indexing. |
| website/examples/configuration/search-fields.ts | Demonstrates declarations and indexing. |
| website/examples/configuration/search-aggregations.ts | Removes legacy aggregation options. |
| website/examples/configuration/data-sources-clients.ts | Configures the PostgreSQL client. |
| website/examples/concepts/visibility-vs-access-guards.ts | Updates the search visibility guard. |
| TESTING.md | Describes dual-backend integration testing. |
| test/consumer/src/search.ts | Checks published search API types. |
| test/consumer/src/index.ts | Updates consumer plugin registration. |
| test/consumer/prisma/schema.prisma | Selects PostgreSQL. |
| test/consumer/prisma/note.prisma | Removes MySQL column annotations. |
| src/utils/visibility.ts | Returns the shared all-visible filter. |
| src/utils/queryBuilder.ts | Removes the legacy query builder. |
| src/types/visibility.ts | Adds field-typed search visibility. |
| src/test/integration.test.ts | Runs integration cases across both backends. |
| src/test/integration.setup.ts | Configures PostgreSQL and backend fixtures. |
| src/test/helpers/postgresSchema.ts | Creates isolated database test schemas. |
| src/test/helpers/fastify.ts | Removes the OpenSearch mock. |
| src/test/helpers/fakeSearchBackend.ts | Adds a request-recording test backend. |
| src/test/__snapshots__/integration.test.ts.snap | Separates backend-specific snapshots. |
| src/search/visibility.ts | Validates shared visibility conditions. |
| src/search/requestFilters.ts | Normalizes filters and validates geo parameters. |
| src/search/postgresVisibility.ts | Translates visibility into SQL. |
| src/search/postgresValues.ts | Provides typed SQL value helpers. |
| src/search/postgresGeo.ts | Implements bounding boxes and geohash grids. |
| src/search/postgresFacets.ts | Computes value and yearly facets. |
| src/search/postgresAdvanced.ts | Translates advanced queries into SQL. |
| src/search/postgres.ts | Implements PostgreSQL search and indexing. |
| src/search/postgres.test.ts | Tests PostgreSQL backend behavior. |
| src/search/opensearchVisibility.ts | Translates visibility into OpenSearch filters. |
| src/search/opensearchAdvanced.ts | Translates advanced query trees. |
| src/search/opensearch.ts | Implements the OpenSearch backend. |
| src/search/opensearch.test.ts | Tests OpenSearch backend behavior. |
| src/search/fields.test.ts | Tests field-derived capabilities. |
| src/search/backend.ts | Defines the public search contract. |
| src/search/advanced.ts | Parses and limits advanced queries. |
| src/routes/search.ts | Validates and dispatches backend searches. |
| src/routes/capabilities.ts | Publishes backend capabilities for API v0.5.0. |
| src/routes/capabilities.test.ts | Tests backend capability publication. |
| src/routes/__snapshots__/search.test.ts.snap | Removes legacy search snapshots. |
| src/routes/__snapshots__/capabilities.test.ts.snap | Removes legacy capability snapshots. |
| src/index.dev.ts | Updates development backend configuration. |
| src/errorHandling.test.ts | Uses the fake search backend. |
| src/app.ts | Exports and registers the new search API. |
| src/app.test.ts | Tests required backend configuration. |
| scripts/setup-integration.sh | Migrates PostgreSQL and checks schema drift. |
| scripts/loadEntities.ts | Loads documents through the backend. |
| scripts/check-consumer-types.sh | Generates PostgreSQL consumer types. |
| README.md | Updates database development commands. |
| prisma/schema.prisma | Switches the datasource to PostgreSQL. |
| prisma/models/file.prisma | Removes MySQL string limits. |
| prisma/models/entitySearch.prisma | Adds indexed search storage. |
| prisma/models/entity.prisma | Adds the search relation and removes limits. |
| prisma/migrations/migration_lock.toml | Selects PostgreSQL migrations. |
| prisma/migrations/20261020043449_entity_search/migration.sql | Creates search storage and indexes. |
| prisma/migrations/20261020000000_init/migration.sql | Creates the PostgreSQL baseline schema. |
| prisma/migrations/20260330234355_init/migration.sql | Removes the MySQL baseline migration. |
| package.json | Updates the adapter and development commands. |
| knip.jsonc | Includes the loader in dependency analysis. |
| example/data/seed.ts | Uses field-driven OpenSearch indexing. |
| example/data/README.md | Updates storage and indexing descriptions. |
| docs/adr/0003-pluggable-search-backends.md | Clarifies backend storage and refresh behavior. |
| docker-compose.yaml | Runs PostgreSQL for development. |
| docker-compose.test.yml | Runs PostgreSQL for tests. |
| CLAUDE.md | Updates architecture guidance. |
| .github/workflows/test.yml | Uses PostgreSQL and checks migration drift. |
| .env | Updates the local database URL. |
Review effort: Balanced
| ]; | ||
| const geohashGrid: Search_RequestBody['aggs'] = | ||
| geoField && geohashPrecision !== undefined | ||
| ? { [GEOHASH_GRID]: { geohash_grid: { field: geoField, precision: geohashPrecision } } } |
Collaborator
Author
There was a problem hiding this comment.
Addressed in d050c9e the other way round: an unbounded grid is a large response on any backend, so both now return at most the 10,000 cells with the most matches (size set explicitly on OpenSearch, ORDER BY count DESC … LIMIT on Postgres). Documented under Configuration → Search → Geo.
| const RANGE = ['lt', 'lte', 'gt', 'gte'] as const; | ||
|
|
||
| const isPlainObject = (value: unknown): value is Record<string, unknown> => | ||
| typeof value === 'object' && value !== null && !Array.isArray(value) && !(value instanceof Date); |
Collaborator
Author
There was a problem hiding this comment.
Fixed in d050c9e: only objects with Object.prototype or a null prototype are accepted, so a Map or class instance fails closed with a logged 500 naming the position. The shared contract suite covers a top-level Map, a nested instance and a Map as a field condition on both backends.
Reject non-plain objects (Maps, class instances) in search visibility
filters instead of reading them as {}. Both reference backends return at
most 10,000 geohash cells, those with the most hits.
Undeclared filter keys are now rejected by splitFilters alongside bad filter values, and include the rejected value.
…tgres @opensearch-project/opensearch is now an optional peer dependency, so consumers using PostgresBackend or their own backend don't install it. BREAKING CHANGE: import OpenSearchBackend from 'arocapi/opensearch' and PostgresBackend from 'arocapi/postgres' instead of 'arocapi'. Consumers of OpenSearchBackend must install @opensearch-project/opensearch.
Tests, test helpers, snapshots, src/reset.d.ts and prisma.config.ts no longer ship. dotenv and @prisma/config were only needed by prisma.config.ts, so they are now dev dependencies.
arocapi only uses fastify's types and registers into the consumer's instance, so a second copy could make the plugin fail to type-check. BREAKING CHANGE: consumers must install fastify ^5.12.7 themselves.
This was referenced Oct 21, 2026
Closed
Closed
|
🎉 This PR is included in version 5.0.0-beta.6 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
arocapi 5.0 makes search pluggable: a consumer passes a Search backend as
search, and its Field declarations drive the index,/capabilities(v0.5.0), request validation, and the typed Search documents and Search visibility filter. Two backends ship,OpenSearchBackendandPostgresBackend, for a Raspberry Pi without OpenSearch.sequenceDiagram participant Client participant Route as /search participant Backend as Search backend participant Prisma Client->>Route: POST /search Route->>Route: validate against capabilities (400)<br/>parse advanced query, caps (422) Route->>Backend: search(request, visibility) Backend->>Backend: check visibility (500 if invalid)<br/>hits, exact total, facets, geohash grid Backend-->>Route: ordered ids Route->>Prisma: load Records under entityVisibilityFilter Route->>Route: access, drop highlights if Denied, transformers Route-->>Client: hits, total, facetsCalls not in the spec:
filters.rangesis an array per field (Oni sends year filters as arrays); exact date/number filter values become ranges, a calendar date covering its UTC day; partial dates aren't valid filter bounds.checkSearchVisibilityandVisibilityConditionare exported for custom backends.OpenSearchBackendtakesrefresh(defaultfalse).number, roles ongeo,basicon non-text;AND/OR/NOTreserved.Not addressed:
SearchVisibilityFilter<typeof fields>(Fastify'sregisterdrops the generics).*matches everything on Postgres and nothing on OpenSearch.Evidence
One contract suite runs over
app.injectagainst both backends in the same run. Themainlines are the faults listed in #97, not runs of this suite.After:
pnpm test— 829 tests pass on both backends, 100% coverage;pnpm lintand the website build pass.Merge Danger
Door: one-way
Merging publishes
5.0.0-beta.1and deploys the 5.0 docs; the Postgres migrations replace the MySQL ones.Blast Radius: consumers
Every consumer moves to Postgres 18 and rewrites
searchandsearchVisibilityFilter(eachBREAKING CHANGE:footer names the replacement). Oni, nabu and ldacapi follow-ups live outside this repo.Closes #97, closes #98, closes #99, closes #100, closes #101, closes #102, closes #103, closes #104, closes #105, closes #106, closes #107