[p3t3r67x0]

Summary

Adds municipality-level spatial research for Northern Germany and Southern Denmark. Journalists can select a municipality, share its filters, export matching records and open a dossier with occurrence-based activity, venues, active organizations, categories and a boundary map.

• self-hosted Nominatim discovery/import with bounded plan/apply CLI
• PostGIS research-area polygons and stable UUIDs
• explicit DE/DK OSM semantics and verified city exceptions
• spatial venue/event assignment, compatible city filters and CSV
• optional offline BKG municipality catalog and dated population import, requested during implementation

Nominatim

Production source: https://nominatim.oklabflensburg.de. No public fallback. Existing timeout, byte/point limits, fixed-origin HTTP policy and pacing are reused. All boundaries are fetched before the short write transaction; provider failure leaves existing data intact. Normal Research requests read stored PostGIS data and never call Nominatim.

Area model

Migration 0016, following verified head 0015, adds only admin.research_area. Unique (osm_type, osm_id) identifies OSM relations; UUID and created_at survive updates. Polygon input becomes valid MultiPolygon/4326; the label point uses ST_PointOnSurface. GiST indexes the boundary. Invalid polygons and positive-area overlaps are rejected; shared edges remain valid. Missing discovery results never delete/deactivate areas. Retrieval time is distinct from unknown OSM modification time.

Optional BKG population columns retain count, reference date, official name, source, input SHA-256 and import timestamp. An exact OSM AGS match is required; older data cannot overwrite newer data. Geometry stays OSM. No population density or invented demographics.

OSM semantics

Regular German municipalities use admin_level 8; Danish kommuner use admin_level 7. Scope requires the exact administrative ISO hierarchy (ISO3166-2-lvl4) and country: Schleswig-Holstein, Hamburg, Mecklenburg-Vorpommern, Niedersachsen, Bremen, Region Syddanmark.

Approved city exceptions are individually pinned to relation, level, region and AGS: Flensburg R27020/6, Kiel R27021/6, Hamburg R62782/4, Bremen city R62559/6, Bremerhaven R62658/6. Bremen Land R62718 is not a municipality. The remaining 12 kreisfreie cities from the supplied BKG scope were also individually verified (Lübeck, Neumünster, eight cities in Niedersachsen, Rostock, Schwerin) and pinned; all 17 identities are documented. Further exceptions require verification.

Nominatim search is ranked discovery, not a complete catalog. An offline BKG converter reads explicit municipality names/AGS; bounded catalog batches require the same exact AGS in discovery and lookup. The supplied 01.01.2026 workbook contains 2770 municipalities in scope; 25 unincorporated areas are excluded. These are catalog counts, not imported-area counts.

Spatial semantics

ST_Covers includes boundary points. Existing EFFECTIVE_VENUE_SQL and EFFECTIVE_SPACE_SQL determine occurrence locations, including date overrides and interrupted space inheritance. Area and date filters match the same occurrence; event results retain the earliest matching public date. Organizations are filtered by event activity, never inferred headquarters.

Separate source/admin connections are preserved: resolve one EWKB boundary from admin and bind it into the read-only source snapshot. No cross-schema grants or per-record network queries. Query-time filtering can use the existing source point GiST; no assignment cache or source DDL is added.

Research API

GET /api/v1/research/areas supports country/type/name and bounded pagination. GET /api/v1/research/areas/{id} returns the dossier and boundary. Additive area_id applies to search, collections, entity dossiers and existing export. Lists omit polygons. Pydantic, Zod, explicit Nitro allowlists, API client, generated OpenAPI and fixtures agree.

UI

Search-as-you-type “Gemeinde / Kommune” shows country/region, keeps the selection in URL state and links to /research/areas/{id}. Existing Leaflet/PointMap supports the boundary and server bbox. The responsive dossier includes real filtered counts, monthly activity, categories, frequently used venues, organizations, event/venue lists, permalinks and CSV navigation. Existing Kulturbytes category colors and OSM attribution remain.

Optional population values show their historical reference date and BKG attribution/license separately from OSM. Supplied VG250-EW data is dated 31.12.2024, older than the municipality catalog.

Security and deployment

Research is read-only for journalists. Import remains operator-only CLI, with no HTTP mutation route. Runtime gains only SELECT on admin.research_area; the existing operator gains SELECT/INSERT/UPDATE. Migrator owns DDL. Explicit provisioning, readiness/boundary checks and reviewed Ansible upgrade contracts are updated; no runtime self-granting and no Uranus writes.

Migrate and provision grants before starting the compatible release. Downgrade removes imported area/population metadata only. No production deployment or import was performed.

Scope

Not included: preferences, saved searches, watches, notifications, browser push. Stable area UUIDs support those later. Discovery may reject missing ISO/AGS tags, unsupported city exceptions, ambiguity or geometries exceeding existing bounds; it does not silently widen scope.

Verification

Local verification (Python 3.13.15, Node 22.22.3, pnpm 12.3.4):

• cd backend && .venv/bin/ruff check . && .venv/bin/ruff format --check . && .venv/bin/mypy: pass (178 source files).
• .venv/bin/alembic heads: single head 0016.
• TEST_DATABASE_URL=<disposable PostGIS 17/3.5 database> .venv/bin/pytest -q: 2061 passed, no skips; 110 existing deprecation warnings. Includes real migrations, runtime/operator role boundaries, spatial index eligibility, ST_Covers, location inheritance, imports, population metadata and Research/API/CSV tests.
• .venv/bin/pytest -q tests/test_research_areas.py -k explicit_city_exceptions: 17 passed, verifying the final expanded city allowlist after the full suite was collected.
• cd frontend && pnpm lint && pnpm typecheck && pnpm test && pnpm build: pass; 869 unit tests, four pre-existing lint warnings.
• TEST_PRODUCTION=1 pnpm test:e2e tests/e2e/{research,geocoding,geo-scope,geo-analytics}.spec.ts --workers=2: 63 passed initially; one browser asset request failed with ERR_NETWORK_CHANGED while a local test container started. Targeted repeat of that case: 1 passed. All 64 production cases covered, including CSP and municipality mobile layout.
• Development runs of those four specs: 56 passed / 2 production-only skips initially; targeted repetition of the interrupted cases and corrected date-display assertion: 10 passed. No application behavior was bypassed or assertions weakened.
• Real Ansible upgrade 0015 → 0016: pass, including runtime SELECT-only and operator INSERT/UPDATE without DELETE; source snapshot unchanged.
• ANSIBLE_LOCAL_TEMP=/tmp/... backend/.venv/bin/ansible-playbook -i ansible/inventory.example.yml ansible/deploy.yml --syntax-check: pass.
• The additional full local Ansible suite completed: 299 passed, 1 failure in the already-loaded console fixture. That fixture initially lacked the existing controller filter; the two-line fixture correction passes the separate targeted real check/apply/idempotence test (1 passed). No deployment code was loosened.
• backend/.venv/bin/ruff check --config backend/pyproject.toml ansible and matching ruff format --check: pass. The first CI run found one overlong test assertion; corrected without behavior changes.
• git diff --check: pass.

Live smoke was explicitly authorized and confined to the own Nominatim instance. Full geometries verified: Flensburg Polygon (507 points), Sønderborg R1928254 MultiPolygon (10619 points). Offline workbook checks confirmed Flensburg 96326 and Kiel 252668 at 31.12.2024. After the user explicitly requested the full original regional scope, a local oklab dry run is in progress (2770 DE + 22 DK inventory entries). Completed regional plans were applied locally: Hamburg 1, Bremen/Bremerhaven 2, Syddanmark 21. Aabenraa R1928466 is absent from the own provider and was rejected, not substituted. No production deployment/import was performed. The remaining German dry run is still in progress; final import results will be recorded separately.

Complete regional operator command

python -m app.research.scope plan|apply --german-catalog ... iterates the bounded single-area importer, reports regional counts and supports --region, --offset, --limit. The 22 DK municipalities follow Statistics Denmark’s region 083 hierarchy and SOP_KOMKOD; nonmunicipal administrative codes are excluded. Requests still use only the existing own Nominatim configuration. DE city identities bypass redundant name searches but lookup rechecks exact AGS. Other DE discovery gets one bounded exact-AGS retry without the region in the query.

Additional verification after this extension: Ruff, formatting and mypy pass; 70 focused scope/import/catalog tests pass against disposable PostGIS, including outage stop, missing DK areas, no scope expansion and AGS revalidation.

Discovery now filters candidate relations by country, region and administrative level before counting exact-AGS matches. The live Weitenhagen response contains both a level-8 municipality and a level-9 subdivision with the same AGS; the regression test accepts the municipality while still rejecting two eligible municipality identities. Mecklenburg-Vorpommern’s full local dry run resolved 724/724 after this fix. A separate operator scope apply process was observed running; duplicate assistant import/check processes were stopped to avoid competing work. The operator’s import remains running, so no completed full-scope import is claimed.