Skip to content

Latest commit

 

History

History
104 lines (60 loc) · 21.6 KB

data.md

File metadata and controls

104 lines (60 loc) · 21.6 KB

Bundled data preparation

Use this workflow when changing the bundled place package, its source selection or corrections, or the GTT timetable package. Python 3 prepares and validates the packages; DuckDB runs the public Overture extraction. Places are delivered only in a reviewed APK. The GTT timetable ships in the APK and can also be replaced on device by a newer hashed package published by CI (see below); no account or paid service is involved.

Place inputs and preservation rules

src/data/places.source.json pins the Overture release, extraction rectangle, input/schema/transform versions, raw hash and counts, contributor licences, and hashes of the companion policy files. scripts/data-places.sql is a template rendered from those pins, not a runnable moving-latest query. Raw extracts stay outside Git; places.compact.json and manifest.json identify the generated runtime package.

The released 0.1.5 package contains 23,274 records: 23,238 named food/shop records selected from release 2026-09-23.1, minus one evidence-backed exclusion, plus 37 compatibility supplements. The raw selection has 25,347 records; 2,109 explicitly permanently closed records are excluded. The bounds [7.60, 45.00, 7.77, 45.14] are an extraction rectangle, not a municipal boundary or a complete business inventory.

  • places.supplements.json and its metadata retain the earlier package's 37 records outside the main category selection, with their original source dates. A main record wins an identical ID; explicit raw permanent closure removes that ID from both sources.
  • places.baseline.json pins the earlier 1,252 IDs. Preparation retains 1,251; the documented exclusion accounts for the remaining ID. Any other disappearance requires an explicit permanent-closure record and review.
  • places.exclusions.json records evidence URL/date, reason and expected original fields. Its one locality conflict is excluded without invented replacement coordinates. Changed source fields require reviewing that exclusion again.
  • places.phone-overrides.json records one first-party branch phone correction, its original value, evidence and checked date. The original phone must still match before applying the correction. Source objects and dates remain intact; correction provenance is separately visible in the place details.

Compact dictionaries/tuples round-trip every normalized application field, full source objects, coordinates and the distinction between absent and explicit null values. Normalization selects the primary name and first address/website/phone entries; it does not preserve every raw Overture property or select a later contact entry. Changing that selection requires a reviewed transform change. Invalid references, schema, dates, hashes, counts, coordinates, unsupported contributors/licences and unexplained baseline loss fail preparation. All supplied inputs validate before the first output file is replaced.

manifest.json is the authoritative generated size/hash/count record. Keep licenses.json consistent with every actual contributor, including DAC, and preserve CDLA-Permissive-2.0, Apache-2.0 and CC0 notices. Source update timestamps, retrieval date and package preparation date describe different events; preparation is not business verification or source freshness.

Reproduce or update places

Use your own external working paths in these examples:

python3 -B scripts/data-prepare.py --places-sql /path/to/overture-places.raw.json > /path/to/overture-places.sql
duckdb < /path/to/overture-places.sql
python3 -B scripts/data-prepare.py --places /path/to/overture-places.raw.json --validate-only
python3 -B scripts/data-prepare.py --places /path/to/overture-places.raw.json

The rendered SQL downloads only the pinned release, bounds and food/shop hierarchy. The exact raw bytes must match the pinned SHA-256. A later release or different extraction serialization is an explicit source-metadata change: record its retrieval/preparation dates, schema, raw hash/counts, expected union count and complete contributor set, then review companion files and update their hashes. A hash mismatch is a failed reproduction; changing the pin merely to suppress the error is not validation. Keep evidence for the source change and explain any changed IDs or fields in the PR.

--places-metadata /path/to/source.json selects a separately reviewed source configuration. Companion file paths resolve beside that configuration. --output /path/to/data selects a generated directory. A place-only refresh preserves existing GTT files, metadata and preparation timestamp; it needs no GTFS download. Repeating preparation from the same pinned inputs must produce identical compact and place-manifest bytes.

A fresh output directory contains only the requested components and their manifest sections; this is an isolated preparation artifact, not a complete app dataset. To deliver it in the app, prepare both components or update an existing complete output, then verify that the compact places, GTT stops/trips and both manifest sections are present and consistent. Independent refreshes of the checked-in complete package preserve the other component.

After the runtime catalog switches to the compact file, remove the obsolete expanded places.json and check for stale imports. Keep the separate policy/supplement inputs; they are not runtime duplicate dumps.

GTT timetable package

GTT rotates trip IDs between daily exports while stop IDs stay stable, and most GTT predictions are delay-only (arrival.delay with stop_sequence). Boards therefore need the current timetable, not just a trip lookup. scripts/gtt_package.py compacts the official ZIP without expanding stop_times on device:

  • Format avoid-maps.gtt-timetable, schema 2, deterministic ASCII JSON. stops, routes, agencies, services (weekday mask, range, added/removed dates), patterns (stop indexes, real stop_sequence values, pickup×4+drop-off codes), vectors (arrival increments and dwell seconds), headsigns, shapes, column-oriented trips (route, service, pattern, vector, start seconds, headsign, direction, shape) and frequencies. Times above 24:00 and repeated visits are kept; missing times fail the build instead of being interpolated.
  • Validation (Python and src/transit-data/package-format.ts) checks every reference, monotonic times, unique IDs, Europe/Rome, coverage dates and size floors, so a truncated export cannot replace a usable package.
  • Manifest (manifest.json): schema_version, package_hash (SHA-256 of the package bytes), package_bytes, package_path (packages/gtt-<hash>.json), source_zip_hash, feed_version, source_url, source_last_modified, source_etag, source_crc32 (CRC-32 of trips.txt, stop_times.txt, calendar*.txt, routes.txt, stops.txt from GTT's ZIP directory), upstream_checked_at (a real request to GTT; null when built from a local file without retrieval evidence), built_at, valid_from, valid_to, attribution, attribution_url, licence_url, counts and up to two previous packages. Packages are hashed, not signed: the hash detects corruption or truncation; trust comes from HTTPS and the repository's workflow. Packages are ordered by GTT source generation (feed_version, then source_last_modified), not by build time: an older source rebuilt later never replaces a newer one, from a download or from an app update.

Publication

Status (2026-11): no mirror is configured and the GTT timetable workflow is disabled manually in the repository's Actions settings. GTT's static-export host refuses connections from GitHub-hosted runners (two runs failed with a TCP connect timeout while the same URL answered from an Italian connection), so the pipeline cannot build in GitHub Actions. TIMETABLE_MIRRORS is empty, so devices make no mirror requests: a check returns unsupported without failure or backoff, and boards omit the 48-hour "unconfirmed" warning (the provenance line still shows the bundled feed version); timetable freshness comes with app releases, and rotated trip IDs are handled by the direct GTT identity remap. Re-enabling requires running the pipeline from a host that can reach GTT and listing the mirrors again.

Mirrors are an optional freshness layer: boards work fully without them (bundled package, direct GTT identity remap below, GTT realtime). The app reads an ordered list of mirrors from TIMETABLE_MIRRORS in src/transit-data/config.ts and uses the next one only when one fails: GitHub Pages (manifest.json, packages/gtt-<sha256>.json), then the assets of one rolling GitHub prerelease tagged gtt-timetable (manifest.json, gtt-<sha256>.json; release assets cannot contain /). The data prerelease is created with --prerelease --latest=false, re-asserted on every run, and its assets are replaced in place (packages first, then the manifest, then stale assets removed), so release trackers such as Obtainium keep following the app's own latest release. Its notes say it is not an app release.

.github/workflows/gtt-timetable.yml runs at 03:15, 06:15, 12:15 and 18:15 UTC and on manual dispatch. scripts/gtt-pipeline.py downloads the ZIP with If-None-Match/If-Modified-Since from the published manifest, builds and validates the package, and publishes to GitHub Pages only when the package hash changed, or when the published upstream confirmation is older than 20 hours (a manifest-only heartbeat, so devices can tell an unchanged upstream from a stalled pipeline). Previous packages are re-downloaded, re-verified and kept beside the current one. The workflow uses no secrets and never receives signing credentials; the release-mirror job alone has contents: write (needed for release assets), checks out no code and only uploads the verified files. The Pages deployment is opt-in: set Settings → Pages → Build and deployment → Source: GitHub Actions, then the repository variable GTT_PAGES_ENABLED to true. Without it only the Releases mirror is updated, and because the pipeline reads the previous manifest from Pages, each run then publishes a fresh package set without carried rollback packages.

Local reproduction, writing the APK bundle (gtt-timetable.gttpkg (JSON bytes shipped as an Expo asset), gtt-timetable.manifest.json, gtt-stops.json and the gtt section of manifest.json) from the same build:

python3 -B scripts/gtt-pipeline.py --site-url https://avoid-quest.github.io/maps/gtt/ --output /path/to/empty-site --no-previous --bundle-dir src/data
python3 -B scripts/data-prepare.py --gtfs /path/to/gtt_gtfs.zip   # offline ZIP: no upstream confirmation recorded

A local ZIP records upstream_checked_at: null unless the retrieval evidence is passed explicitly (--gtfs-checked-at/--gtfs-last-modified, or --zip-checked-at/--zip-last-modified for the pipeline), taken from the original download's time and Last-Modified header. Boards then say the last check against GTT is unknown.

The package ships as a file asset (.gttpkg, registered in metro.config.js), not as a JavaScript module: it is read, hash-verified and parsed only to import it, then released, so it never stays in the JS heap. gtt-stops.json is the synchronous stop list for map and search, derived from the same package and tagged with its hash. Place-only refreshes leave the GTT files untouched.

Re-joining rotated trip IDs directly from GTT

GTT re-issues trip IDs in bulk for unchanged trips (calendar re-cuts clone them under new trip and service IDs). Realtime descriptors carry only trip_id, start_time, start_date and the schedule relationship, and start_time equals the static first departure. Boards therefore join a live update by trip_id first, and otherwise by schedule identity: route, direction, shape and first departure, on the descriptor's start_date.

The identity of a live trip_id comes from GTT's own export, read with HTTP ranges (src/transit-data/zip-range.ts, remap.ts): the ZIP directory from the last 8 KB, then trips.txt, calendar.txt, calendar_dates.txt and feed_info.txt in two grouped ranges pinned with If-Range. Members are inflated with fflate (pure JavaScript) in 1 KB input steps (at most ~1 MB of output per step), yielding to the event loop every few milliseconds, verified by CRC-32 and size, and stored in their own SQLite database (gtt-remap.db). Every layer is bounded: bytes per request (Content-Length at the headers and while receiving), exact Content-Range coordinates and body length, central directory size (256 KB), entry count, offsets within the archive, declared and actual inflated bytes per member (16 MB) and per read (24 MB), and CSV record counts and line length; inflation stops as soon as a limit is passed. A server that ignores Range, or an archive that changes between requests, is refused at the response headers, so the 20 MB file is never downloaded. A damaged remap database is discarded and recreated; a failing remap never affects exact trip-ID joins. Measured live on 2026-11-01: 3 requests, 422,670 bytes, about 0.8 s on a desktop; a later conditional check is one If-None-Match request answered 304. Persisting the 61,212-trip remap took 0.2 s and created a 2.7 MB database.

The remap runs only for a visible board — when the live feed contains trip IDs known neither to the package nor to the current remap, when the remap's calendar does not run a live trip's service (GTT may extend calendars without new IDs), or when a remap older than a day is in use — at most once a day (Aggiorna orari may retry after 10 minutes), and follows the Wi-Fi-only preference. Requests go only to www.gtt.to.it, without cookies, stop or location. When the directory's CRC-32 values for trips, stop times and calendars equal the active package's source_crc32, the package is confirmed current directly by GTT (it counts as upstream confirmation) and nothing else is downloaded.

An identity join requires that GTT's current calendar runs the remapped service on start_date and that every reported stop_sequence (and stop_id, when present) exists in the matched local pattern. All local variants sharing the identity are compared, whether or not they run locally that day. Variants with different stops or times are separated only by evidence: the single variant that runs locally on every day the live service runs in the coming fortnight (as far as both calendars define it), or, when GTT defines the service for too few days, on every weekday it runs. Anything else is not joined. Identity-joined rows say so under the row, and rows outside the installed calendar explain why they appear. Replaying the 2026-11-01 feed (157 trip updates) against the 20261011 export: 0 matches by trip ID, 156 by identity, 1 refused because its schedule changed; against the current export, 150 of the 156 had identical reported times and 6 differed by 1–3 seconds. A live trip whose local schedule exists but does not run that day in the installed calendar is shown only while its live update matches, with a note that its times come from the installed package. Scheduled-only rows are never extended beyond the installed calendar: trips.txt has no times.

On device

src/transit-data/ imports a package into a new candidate SQLite database (expo-sqlite), checks references and compares representative board queries against the parsed package, then switches the active package in one registry transaction. The previous package is retained for rollback; a failed download, hash mismatch, invalid package, full storage or process death leaves the active database in place, and leftovers are removed on the next start. At startup the active database must contain every table and index, the row counts recorded at import and pass PRAGMA quick_check; otherwise it rolls back to the previous package, then to the bundled one. Indexes: pattern_stop(stop_id, pattern_id) and trip(pattern_id, service_id, start_seconds); only the vectors needed by a board are decoded.

The manifest is checked when a board opens or resumes and the last successful check is older than 6 hours, on Aggiorna orari, and once per hour at most when a nonempty realtime feed joins zero trips. Failures back off from 5 minutes to 6 hours. Any connection is used by default; an optional preference restricts checks to Wi-Fi, and when the network type cannot be determined that preference defers the check. There is no background task. Validators (ETags per mirror) and upstream confirmations are stored per package hash, so a rollback or switch never reuses another package's evidence and restarts the 6-hour interval. Boards warn when the active package has not been confirmed by a manifest for 48 hours (or the confirmation is unknown). Coverage runs from the first service day to the last event of the final service day, overnight trips included; outside it boards say schedules are unavailable instead of extrapolating. Requests carry no identifiers, stop, location or query parameters; the mirror host sees the device IP and request time.

Realtime rows: candidates are looked up three hours back, matching the largest accepted delay, so heavily delayed vehicles stay until their predicted time. Predictions, cancellations and skipped stops share one freshness rule: definite up to 2 minutes after the older of feed and entity timestamps, shown with their age up to 30 minutes, then no longer current (a cancellation report stays visible with its age). Optional, off by default: Tempo reale esteso dalla fermata (GTT) asks GTT's per-stop endpoint (palina.getTransitiOld, public stop code, bacino=U) for the open board only, at most once per 30 s with a 25 s cache, never for favourites or in the background. GTT then learns which stop is viewed. Its minute-precision passages are added as declared-realtime-no-source-time rows ("Tempo reale · GTT (fermata)") with retrieval time only; they are not linked to trips, never replace a trip-joined prediction of the same route within 90 s, skip metro and extraurban lines, and still arrive when the citywide feed host is down. All GTT and mirror requests use credentials: 'omit', which in React Native selects OkHttp's no-cookie jar: GTT's session cookies are neither stored nor sent.

StopTimeEvent.uncertainty is kept; above two minutes the time is shown as approximate with ±N min instead of a countdown. Inexact frequency service keeps its headway row; realtime instances named by start_time are shown only with absolute predicted times, never delays against the template.

On a desktop (Bun's SQLite) the 20261101 package imported, validated and switched in about 0.7 s, produced a 9.3 MB database, and a Castello (472) board took about 3.5 ms with or without the realtime join. Android timing and memory remain device checks.

Evidence and acceptance

Run bun run verify; its Bun wrapper also runs the Python regressions. For an isolated pipeline check, run:

python3 -B -m unittest discover -s scripts -p 'test_*_package.py' -v

Check deterministic regeneration and that an independent place refresh leaves GTT bytes/metadata unchanged. Inspect exact-name, category, typo, numeric-token, address, cap and coverage flows with online search enabled and with persisted Solo ricerca locale. The runtime catalog shares taxonomy membership and materializes full provenance/details after limiting results. Native map layers continue to render geographic results.

The desktop prototype compared 23,238 normalized main records: expanded JSON was 24,867,733 bytes versus 5,653,387 bytes compact; fresh Bun-process index/import took 543.5 ms versus 150.2 ms and whole-process RSS was 163.8 MB versus 105.9 MB. Compatibility records and reviewed corrections change the shipped size. These measurements support trying the compact path; they do not establish Android performance or justify a native SQLite dependency by themselves. Measure release startup, repeated typing/category queries and memory on supported Android configurations before accepting native performance.

A separate final-runtime desktop probe measured catalog loading at 154.2 ms and search-module loading/index creation at 110.4 ms, with 24.33 MB heap after catalog loading. The preceding mutable-surface version measured 132.6 ms, 109.4 ms and 23.42 MB respectively. This single fresh-process comparison includes module loading and records the cost of freezing exposed entries, coordinates, shared arrays and metadata while retaining compact dictionary sharing. It is distinct from the earlier prototype comparison and is not native startup, typing or memory acceptance. The catalog takes ownership of its retained compact inputs; materialized details remain fresh mutable copies.

Twenty source-selected search cases and 40 exact/category comparisons matched the expanded encoding; that checks encoding/ranking, not real-world coverage. A separate 14-business source-selected spot check found contact/locality conflicts and incomplete evidence. It established no businesses as currently operating and is not a representative accuracy estimate. Field/first-party acceptance must distinguish source presence, correct branch/contact, storefront/entrance, current operation and usable opening hours. Missing locator results alone do not prove closure. Strict same-name duplicate handling preserves branches and tenants but intentionally leaves potential differently named duplicates for later evidence-based review.

Existing favourites save full place snapshots and do not automatically adopt new package contacts or exclusions. Test persistence/upgrades separately; reconciliation belongs to a future dataset-refresh feature. Follow Android acceptance, the roadmap and review delivery before publishing a signed release. Report browser evidence, unresolved provider results and device-only checks separately. Route previews remain planning.