feat: build embedding replacements beside active vectors with Ptah - #51
denisvmedia wants to merge 1 commit into
Conversation
|
PR author is not in the allowed authors list. |
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: ZenSystemAI/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The implementation is well tested, but its external migration tooling, table rewrite, and coordinated production cutover warrant final human operational review.
Review effort: Balanced
Findings: None
What changed in this PR
Adds an opt-in Ptah workflow for building and validating replacement embeddings without disrupting active vector search.
Changes:
- Makes pgvector reads and writes select a validated embedding column.
- Adds Ptah migration configuration, documentation, and integration testing.
- Prevents legacy re-embedding against candidate columns.
| File | Description |
|---|---|
README.md |
Links the migration guide. |
docs/embedding-generations.md |
Documents preparation, cutover, and retirement. |
api/src/services/pgvector.js |
Routes vector operations through the selected column. |
api/scripts/reembed.js |
Rejects candidate-column configurations. |
api/tests/storage-pgvector-helpers.test.js |
Tests column validation and SQL generation. |
api/scripts/ptah/verify.js |
Exercises the end-to-end migration workflow. |
api/scripts/ptah/source.sql |
Adds Ptah’s generated text source. |
api/scripts/ptah/embedding-v2.json |
Provides an example Ptah specification. |
.github/workflows/ci.yml |
Adds PostgreSQL/Ptah integration CI. |
.env.example |
Documents PGVECTOR_COLUMN. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
When
reembed.jschanges dimensions, it clears the active vector column before rebuilding the corpus. This adds an optional Ptah workflow that builds a replacement beside it, so the current encoder and search can keep serving during backfill.Hi, I'm Denis, a maintainer of Ptah. We want to make embedding model changes easier to verify on real application workloads. Ptah is open source; the site provides documentation and a browser playground for trying its schema commands without installation. Related proposal: #50.
The application change is confined to the pgvector storage layer:
PGVECTOR_COLUMNselects the column for search, ordinary upserts, and atomic fact/status supersession. It defaults to the existingvectorcolumn. Startup validates the selected column and dimension.reembed.jsrefuses a selected candidate column, preventing it from accidentally operating on the legacy column with the new encoder configuration.The guide includes the diagram, specification, commands, and handover procedure. Pause writers for final catch-up and deploy the encoder and column together. Zengram does not automatically follow Ptah's pointer; old vectors alone do not provide rollback after writes resume.
Validation passed on the submitted commit in fork CI: existing tests and API security audit on Node 20/22, plus the new PostgreSQL/pgvector integration job with released Ptah 0.11.4. The test checks a 1536-to-384 dimension change, old search while candidate embedding is in flight, provider failure/retry, preservation of old vectors/payloads/index, insert/update/delete catch-up, explicit approval, both new-column write paths, and tenant/collection filters. Its deterministic local provider tests migration mechanics, not model quality. The existing typecheck step remains non-blocking as configured upstream.