[p3t3r67x0]

Summary

Promotes the current dev branch to main with municipality-based Research areas, a shared administrative area selector, an event vector retrieval pilot, and explicitly selected experimental semantic event search for authenticated Research users.

main was fetched and merged into dev without conflicts, rebase, or rewritten history. The two main-only commits (3242152, PR #130; 8a11d2d, PR #133) are previous promotion merges: each has exactly the same tree as its merged dev parent, so their content was already present. Integration merge: 1a6c60d0840f3884a9a911ce93f8086fa831e54b. A separate 3d4db23 commit removes seven trailing-whitespace occurrences; no functional release-preparation changes were needed.

Comparison before integration: main 8a11d2deedb5f6f3751200fb21877efbc94960bb, dev 7a85a608a788b36bbd283d6f50a8ba0e72cadf8a; 2 main-only and 21 dev-only commits. During validation, remote dev advanced to 183097e33cf0d1399a80cdff3e4f18f336e87031 (PR #137, commits af2a724 and 1ef00ee). Those three additional commits were preserved via merge a03dc01aec4ffd3d12bbdb479770fc26eb7be270 in an isolated release worktree. Final comparison: 0 main-only and 27 dev-only commits. Backend/frontend checks were rerun; the Ansible tree is identical to the fully tested tree.

Highlights

• Research areas and municipalities: operator-owned OSM boundaries for the supported northern German regions and Region Syddanmark; regional municipality discovery/import, explicit municipality/district classification, and separately imported BKG population metadata with provenance.
• Research UI and API: municipality selector, URL-based geographic filtering, boundary dossier and map, events/venues/organizations, activity and usage views, population display, and mobile coverage. Pydantic, Zod, Nitro route/query allowlists, OpenAPI and fixtures are updated together.
• Shared GeoScope: Operations and Research reuse the administrative area selector and imported canonical areas. Legacy municipal scope bookmarks resolve to canonical Research UUIDs while existing noncanonical cached scopes retain their fallback.
• Research AI and semantic search: bounded public event extraction, deterministic document/chunk identities, vector indexing/reconciliation, three-model retrieval benchmarks and a blind pooled manual relevance-review tool. Adds an explicit experimental semantic mode to the existing Research landing/results search, an authenticated event-only API, and an optional configured Jina search gateway. Current PostgreSQL records and eligibility are rechecked before display.
• Deployment and documentation: research-area runtime/operator privilege contracts, optional private CPU encoder/Qdrant deployment files, resource and network restrictions, operational documentation and deliberately versioned benchmark exports.
• Tests: expanded Research area/catalog/population/scope, authorization/proxy, vector reconciliation, manual review, deployment and frontend coverage; CI adds a pinned Qdrant fixture.

Database

• New revision: backend/migrations/versions/0016_research_area.py, following 0015.
• Current single head: 0016; alembic heads and alembic history confirm a linear chain. No migration conflict or merge migration.
• Adds only admin.research_area, geometry/centroid and population provenance constraints, a GiST geometry index and a country/type index. No Uranus DDL or writes.
• Migrate with the explicitly selected admin_migrator connection before starting the new backend/workers, then provision the documented grants: runtime admin_user gets SELECT only; admin_auth_operator gets SELECT/INSERT/UPDATE. Readiness enforces the current head and grants.
• Municipality boundaries and population data require separate operator imports; migration/runtime startup does not import them automatically.
• Live migration execution, role-boundary integration and alembic check were not run locally because no disposable DB was used. CI is required for those checks.

AI / Search

This diff provides both operator-only indexing/benchmarking and authenticated experimental semantic event search. Classic search remains the default; users explicitly select semantic mode. The Research API requires journalist or system administrator access, and PostgreSQL remains authoritative for current public eligibility, displayed records and structured filters. Vector payload prose and scores are not displayed.

Online retrieval fixes the model to Jina v3. The direct encoder/Qdrant path fetches at most 50 chunks and returns at most 20 rehydrated current events. The optional exact HTTPS /search gateway supplies at most 10 candidates before filtering. Counts describe the bounded selection, not all source matches; semantic mode does not offer misleading full-result export, pagination or classic sorting. Network calls finish before acquiring the source snapshot, with an 8-second overall deadline under Nitro's 10-second proxy limit. Errors fail safely without automatic fallback.

SEMANTIC_SEARCH_NONCOMMERCIAL_JINA defaults false and requires explicit operator acknowledgment for the noncommercial pilot. Optional SEMANTIC_SEARCH_URL selects the configured HTTPS gateway; otherwise protected encoder/Qdrant credentials are required. Query text is sent to the selected trusted service. The documented gateway currently accepts requests without authentication, but no admin cookies, vector keys or other admin credentials are forwarded. Redirects and environment proxies are disabled; query/body logging must remain off. No production configuration, new service deployment or source write is performed by this PR preparation.

The optional CPU encoder/Qdrant pilot supports pinned E5-small, E5-base and Jina-v3 models, private authenticated services, single-job indexing, resource ceilings and offline model loading. No model weights, Qdrant storage/snapshots, private keys, runtime/operator env files or production secrets are committed. Checked-in benchmark JSON/CSV and snapshot metadata under docs/research-ai/results/2026-09-27/ are intentional documentation, not binary snapshots or local test output. backend/.env.example contains only example/default settings.

Historical benchmark measurements retain provenance; no relevance judgments or production model winner are fabricated. The legacy exports lack review excerpts and must be regenerated into a new private directory for pooled manual review. No LLM, chat, generated SQL, automatic interpretation of geographic/date constraints or automatic indexing is introduced.

Validation

Check	Local result
uv sync --locked	Passed; lockfile unchanged
uv run alembic heads / history	One linear head, 0016
Backend Ruff / format	Passed, 274 files formatted
uv run mypy	Passed, 188 source files
Full uv run pytest -q -ra	1,353 passed, 897 skipped, 34 warnings, no failures; includes exact OpenAPI equality and offline DB documentation tests
Backend integration coverage	896 PostgreSQL/PostGIS skips and 1 real-Qdrant skip; no disposable services configured
pnpm install --frozen-lockfile	Passed; lockfile unchanged
Frontend lint / typecheck	Passed; ESLint has 4 existing warnings and 0 errors
pnpm test	892 passed across 64 files
pnpm build	Passed
Production Chromium, desktop and mobile	54 passed, covering Research including semantic search, GeoScope, Geo-Analytics and CSP
Development Chromium (before late PR #137)	46 passed, 2 expected production-only CSP skips in a full repeat with unchanged settings. Initial run had 45 passed / 2 skipped / 1 Geo-Analytics desktop navigation timeout; trace inspected, isolated rerun passed, then the full repeat passed. The transient first-run failure is disclosed; no timeout/assertion was relaxed.
Ansible Ruff / format	Passed, 36 files formatted
Ansible deploy/preflight/sql-console-audit syntax	All three passed
Full local Ansible unittest discovery	301 run: 233 passed, 68 database skips, no failures. Tree unchanged by late PR #137; no repeat needed.
Offline database documentation	Passed; 20 admin tables and all 21 SQL/DBML/DOT/SVG/PDF artifacts verified under /tmp
Documentation / static checks	OpenAPI Prettier passed; 15 changed JSON files parsed; relative Markdown file links resolved; AI installer bash -n passed
git diff --check origin/main...dev	Passed after the isolated whitespace correction
Secret/artifact review	Changed files, paths and commit history reviewed; no production secrets, model weights, private keys or storage/test artifacts found

The extra non-gating Prettier audit of all changed Markdown files reports ansible/README.md, backend/docs/development.md and backend/docs/geo-scope.md. The same three files already fail on origin/main, with identical formatting-change line counts (88, 106 and 87 respectively). No repository Markdown CI gate is configured; this pre-existing formatting debt is preserved and disclosed.

Local E2E covers the four relevant specs, not the entire visual suite. The first production attempt stopped before tests because an existing test-results/.last-run.json was not writable; the successful rerun used a fresh /tmp output directory. Final Production E2E also covers the newly integrated semantic-search UI. The full pinned browser/OS visual suite, database deployment matrix, CodeQL and dependency review remain GitHub Actions checks; CodeQL was not recreated locally.

Local Docker tests were explicitly excluded by the requester. This PR is intentionally not a draft, also explicitly requested; outstanding integration checks are disclosed rather than treated as passing. No production database, deployment or live AI service was touched.

Deployment notes

1. Coordinate API/worker downtime for migration, provision revision 0016 and its explicit grants, and deploy compatible backend/frontend builds. Keep migrator/operator credentials out of runtime environments.
2. Verify /ready, workers and Research/GeoScope behavior separately; import reviewed municipality boundaries/population data with the separate operator commands when needed.
3. Semantic search requires the explicit noncommercial Jina acknowledgment plus either the configured exact HTTPS search gateway or protected direct encoder/Qdrant settings. Classic search works without them; missing configuration returns a safe semantic-unavailable response. Direct vector services remain optional and require separately reviewed installation, private ingress/firewall rules and explicitly prefetched pinned model data. The normal backend dependency installation does not install the optional CPU embedding group.
4. The checked-in AI installer targets the documented A/B topology; it is not a generic host bootstrap. A merge does not authorize or perform that deployment. Benchmark area context requires the migrated/imported area table.

Risk / rollback

The largest changes affect spatial selection across Research/Operations, area schema/privileges, the optional vector extraction/index lifecycle, and external semantic retrieval. Candidate limits can omit relevant matches, index ranking can lag edits, and upstream rate limits/timeouts can make semantic search unavailable. PostgreSQL/PostGIS/Qdrant integration and the CI deployment matrix remain release gates; local browser checks do not establish live backend integration.

An older backend expects its exact migration head, so application-only rollback against revision 0016 can fail readiness. Coordinate rollback with a reviewed admin backup/schema plan; downgrading 0016 drops imported area metadata and requires a backup/reimport plan. Never change Uranus data for rollback. The optional AI stack can be stopped separately; preserve protected credentials, model cache and Qdrant storage/snapshots for recovery or rebuild from the read-only source.

No force push, direct main change, auto-merge or branch-protection bypass is part of this promotion.