Skip to content

feat(research): support nationwide DE/DK operator imports without dotenv - #141

Merged
p3t3r67x0 merged 5 commits into
devfrom
fix/research-national-scope
Sep 28, 2026
Merged

p3t3r67x0 merged 5 commits into
devfrom
fix/research-national-scope

Conversation

@p3t3r67x0

Copy link
Copy Markdown
Collaborator

Research operator commands currently try to read the runtime .env even when root has already exported operator.env before switching to oklab. The three operator entry points now explicitly disable dotenv loading; normal runtime Settings remain unchanged.

The municipality scope now covers all 16 German states from the supplied BKG catalog and an explicit 98-municipality Danish catalog across all five regions. German AGS checks remain exact, Danish imports require catalog-pinned relation IDs, and 90 additional German city exceptions were individually verified through the own Nominatim instance. The 107 exceptions still require their exact relation/region/level/AGS tuple. Scope ordering and resume remain deterministic, with the bound adjusted from 20,022 to 20,098.

Migration 0017 follows unchanged 0016 and only replaces the region constraint. Its downgrade locks and refuses to proceed while out-of-old-scope rows exist; it never deletes them. SQLAlchemy metadata and the deployment upgrade target/fingerprint are aligned. No new grants, API contract changes, automatic imports, or runtime dotenv changes.

Validation:

  • uv run ruff check ., uv run ruff format --check ., and uv run mypy: pass (283 formatted files / 194 typed source files).
  • Full uv run pytest -q completed: 2,525 passed, one optional Qdrant integration skipped (TEST_QDRANT_URL unset), 26 SQL-console fixture setup errors. These were caused by the extra national_plan_operator role created for live plans in the same disposable PostgreSQL cluster; the console deliberately rejects unknown cluster-wide EXECUTE consumers. The completed plan database and role were removed without changing application code or relaxing any guard. The affected console tests and all research/migration tests were rerun: 372 passed, no failures or skips.
  • Separate deployment packager suite: 24 passed. Actual packaged-release 0016 → 0017 deployment test: passed, including table OID/owner/grant preservation and unchanged synthetic source schema.
  • Alembic heads/history: single linear head 0017; test upgrade/current/check succeeded. Migration tests cover guarded and successful downgrade, all 21 regions and invalid country/region combinations.
  • git diff --check and documentation Prettier/link checks: pass.

Live evidence (2026-09-28): BKG workbook conversion yields 10,747 German municipalities, excluding 192 unincorporated areas. The Danish inventory has 98 entries. Read-only plans used the own Nominatim instance and an empty disposable PostGIS database: DK 96 new / 2 rejected (Thisted 787/R2095303 and Aabenraa 580/R1928466 absent in lookup); German 107 verified city exceptions plus Saarbrücken: 108 new / 0 rejected; an additional plan passed for Aachen, Göttingen, Hannover, Saarbrücken and Eisenach (Thuringia), totaling 112 distinct verified German municipalities. A bounded default-scope plan reports all 10,845 input entries. This is not a full provider plan of all German municipalities or a classification against existing production areas.

Deploy 0017 before the compatible backend, coordinating import jobs for the table lock. Existing UUIDs, geometries, grants and source data are preserved. Re-start scope imports from offset 0 after changing from the old geographic scope, since the region ordering expands; subsequent resume requires the same catalog and selection. No production apply or merge performed.

@p3t3r67x0 p3t3r67x0 self-assigned this Sep 28, 2026
@p3t3r67x0
p3t3r67x0 changed the base branch from main to dev September 28, 2026 09:32
@p3t3r67x0
p3t3r67x0 merged commit 4041a5e into dev Sep 28, 2026
24 checks passed
@p3t3r67x0
p3t3r67x0 deleted the fix/research-national-scope branch September 28, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant