Migrate docsite embed/search to Postgres + pgvector (WIP)

[IZO-Ong]

Short Description

Migrates embed_docsite/search_docsite from Pinecone to Postgres + pgvector (0.8.1 extension), keeping the legacy Pinecone path available. Both indexing and searching support either backend, selected independently — the write target and read backend are separate settings. WIP — comments are welcome!

Addresses the first half of #305

Implementation Details

Using both backends

Indexing — embed_docsite's payload takes a target param:

{ "target": "postgres" }   // or "pinecone" (default)

Searching — search_docsite reads DOCSITE_SEARCH_BACKEND (env, default pinecone), overridable per-request via a backend payload field:

{ "query": "...", "backend": "postgres" }

job_chat's retrieve_docs.py and the search_documentation tool both resolve the backend the same way, through a shared resolve_backend() in search_docsite.py.

No new required env vars for existing setups, Two new optional vars in .env.example:

• DOCSITE_SEARCH_BACKEND — pinecone | postgres (Defaults to pinecone if not provided)
• POSTGRES_TEST_URL — separate DB for the integration suite only (see Tests below)

Search strategies

Postgres adds three strategies: semantic (pgvector cosine), keyword (Postgres full-text search), and hybrid (Reciprocal Rank Fusion of both).

Production consumers (job_chat, search_documentation) use strategy='semantic' on both backends, at the same 0.8 cosine threshold Pinecone already used. hybrid/keyword are available via the search_docsite service directly and via run_eval, but aren't wired into any production consumer yet.

Migrations

Schema lives in services/migrations/0001_docsite_batches_and_chunks.sql — docsite_batches (versioned batch lifecycle: building → complete/failed) and docsite_chunks (chunk text + vector(1536) embedding + generated tsvector for FTS), plus the vector extension and four indexes.

Applied by services/db_migrations.py, run explicitly from embed_docsite's Postgres write path. Migrations are idempotent (tracked in _migrations_docs) and lexically ordered.

No manual step needed, the first embed_docsite run with target=postgres sets up the schema.

Running run_eval

Compares both backends (strategy='semantic' on each) over a golden query set: recall@5, p50/p95 latency, and per-query doc-title agreement.

Setup:

docker run -d --name apollo-postgres \
  -e POSTGRES_USER=apollo -e POSTGRES_PASSWORD=apollo -e POSTGRES_DB=apollo_dev \
  -p 5432:5432 pgvector/pgvector:pg16

Set OPENAI_API_KEY, PINECONE_API_KEY, POSTGRES_URL in services/.env (this is the file that's actually loaded — see note below).

Populate both backends:

bun py embed_docsite --input tmp/payload_postgres.json   # {"target": "postgres"}
bun py embed_docsite --input tmp/payload_pinecone.json    # {"target": "pinecone"}

Run the eval (must run from services/, the Python path root):

cd services
poetry run python -m search_docsite.tests.eval.run_eval

Note: services/.env is the file every Python service actually loads — python-dotenv resolves relative to the importing module's location, so anything under services/ finds services/.env before it would reach a repo-root .env. Keep credentials there, not just at the root.

Results

Postgres (semantic): recall@5=1.0 (scored=9, skipped=1) p50=0.479s p95=1.311s
Pinecone (semantic): recall@5=1.0 (scored=9, skipped=1) p50=0.925s p95=1.872s
Backend agreement:   mean doc-title Jaccard=0.950 across 10 queries

Postgres matches Pinecone's recall@5 and is faster on both p50 and p95. All 9 scored queries hit on both backends — the two queries at 0.75 doc-title agreement (what is a run in OpenFn, how do I write a data transform function) differ on which extra titles round out the rest of the top 5.

One query (what adaptors are available for HTTP requests) is unscored — run_eval filters to docs_type='general_docs', but that query's answer lives in adaptor_docs.

golden_queries.yaml's labels were derived from the general_docs corpus via full-text search on each query's key terms.

Latency varies somewhat run-to-run (p95 has ranged 1.3–3.3s across runs on a 10-query sample, likely OpenAI embedding latency). p50 has stayed consistently lower on Postgres across runs.

Tests

Unit — 108 passing, fully mocked:

• embed_docsite: test_docsite_processor.py, test_docsite_indexer.py, test_embed_docsite.py, test_db_migrations.py
• search_docsite: test_docsite_search.py (new Postgres backend), test_pinecone_legacy_search.py (legacy, preserved), test_search_docsite_main.py, test_run_eval.py
• Plus updated consumer tests in job_chat and search_documentation for the new resolve_backend() delegation

Integration — 6 passing, against a real ephemeral Postgres + pgvector:

• services/embed_docsite/tests/integration/test_postgres_docsite_roundtrip.py — fresh-DB migrate→index→promote, search across all three strategies, reindex prunes the previous batch, reader gets a clear 503 without a schema

docker run -d --name apollo-pgvector-test -e POSTGRES_PASSWORD=postgres -p 5433:5432 pgvector/pgvector:pg16
export POSTGRES_TEST_URL=postgresql://postgres:postgres@127.0.0.1:5433/postgres
poetry run pytest services/embed_docsite/tests/integration -q

Caution: Do not use POSTGRES_URL for POSTGRES_TEST_URL — the fixtures DROP the docsite tables and the vector extension between tests, so pointing this at a database would destroy its data.

AI Usage

Please disclose whether you've used AI in this work (it's cool, we just want to
know!):

• Yes, I have used AI
• No, I have not used AI

You can read more details in our
Responsible AI Policy

[hanna-paasivirta]

Amazing, thanks so much Isaac! I will review this on Monday.