[p3t3r67x0]

Summary

Adds the first operator-only AI/Search retrieval pilot for Kulturbytes Research: read-only public event extraction, normalized documents, tokenizer-bounded semantic chunks, exact content hashes, incremental embedding reuse, reconciliation and a reproducible 30-query DE/DA/EN benchmark. Chunk hits are deduplicated to stable event IDs and checked against current public eligibility.

Infrastructure

Server A runs an isolated indexer with its existing Uranus reader. Server B runs persistent, digest-pinned Qdrant and a CPU embedding service with cached pinned models. Raw ports bind only to loopback; remote access requires verified TLS, API authentication and an A-only firewall/nginx allowlist. Qdrant is capped at 2 GiB / 2 CPUs; encoder at 5 GiB / 2 CPUs with batch 2 and one request/model at a time. Existing admin, Nominatim and OneUptime services remain in place.

No database migration or new grants are required. Server A's active admin deployment remains at migration 0015, without research_area; live area metadata is explicitly marked unavailable. The local PostGIS ST_Covers assignment path is implemented and integration-tested.

Security

Uranus remains read-only: effective role checks, READ ONLY / REPEATABLE READ snapshots, no source DML/DDL/triggers/outbox. Source/admin connections close before network work. No DB credentials are sent to B. Public-field allowlists exclude internal notes, auth/session data and private contact fields; source HTML becomes plain text. Fixed model IDs, pinned safetensors/adapters and trust_remote_code=False prevent arbitrary provider code or URL execution. No public search/chat endpoint, LLM dependency or browser secrets are added.

Models

• intfloat/multilingual-e5-small: MIT, 384 dimensions.
• intfloat/multilingual-e5-base: MIT, 768 dimensions.
• jinaai/jina-embeddings-v3: CC-BY-NC-4.0, 1024 dimensions. Uses Jina's official native jina-embeddings-v3-hf conversion with pinned query/passage adapters; operator confirmed noncommercial benchmark use.

No production default or quality winner is declared.

Dataset and benchmark

The observed source has 619 events; 576 pass the existing public Research gate. All three final runs share the same normalized-document fingerprint. Each model has a separate collection, passed the 10 → 100 → full staging protocol, and exported 30 × Top10 distinct events. Repeated reconciliation performs zero embedding/payload/deletion work.

Model	Events	Final chunks	Mean query latency	P95 query latency
e5-small	576	1128	44.5 ms	49.4 ms
e5-base	576	1128	117.8 ms	134.0 ms
jina-v3	576	1124	460.6 ms	528.8 ms

See measured results for all CSV/JSON exports, CPU/RAM/disk metrics, stage timings, source/model provenance and the documented E5 incremental pricing update. Latency includes encoding and TLS retrieval; these are shared-server measurements, not a load test. Unknown source languages stay unknown. Manual relevance remains blank: the evaluator computes P@5/10, MRR@10 and pooled nDCG@10 only with real judgments.

Synchronization and limitations

plan, sync, reconcile, benchmark and evaluate are explicit operator commands. Nullable event timestamps and linked-record changes make an event-only watermark unsafe, so this pilot compares complete bounded public snapshots while embedding only changed/missing/version-incompatible chunks. Metadata-only changes reuse embeddings. Partial stages never delete other events; full reconciliation removes stale/private/deleted events after successful replacement writes. No extra state table is needed.

Not included: production chat/planner, saved searches, watches, notifications, browser push, Venue/Organization indexing or source writes. A future public retrieval endpoint still needs publication/eligibility guarantees and factual rehydration from Postgres.

Verification

Functional code e0b256b1789606b93f23782d9b6cfc5908dff8e9 passed CI; later commits only document results.

• uv sync --locked, uv run ruff check ., uv run ruff format --check .: passed.
• uv run mypy: passed, 186 source files.
• TEST_DATABASE_URL=<disposable PostGIS> TEST_QDRANT_URL=<isolated Qdrant> uv run pytest -q: 2,132 passed, 3 existing Graphviz skips. Those three rendering tests passed locally.
• uv run pytest -q tests/test_vector_index.py with explicit disposable PostGIS/Qdrant: 39 passed, no skips.
• pnpm lint, pnpm typecheck, pnpm build: passed.
• pnpm test: 881 passed across 64 files.
• TEST_PRODUCTION=1 pnpm test:e2e: 553 passed, 41 explicit viewport-specific skips.
• Deployment checks: all three PostgreSQL/PostGIS matrices passed. CodeQL and Dependency review passed.
• Live staged runs, unchanged reconciliation, TLS 401 without credentials, outside-host port blocking and healthchecks passed.
• Documentation Prettier, relative links and git diff --check: passed.

No normal unit test downloads model weights. No application deployment or database upgrade on A.