Live performance groups: relay Mirror → Python sidecar

.gitignore

@@ -55,6 +55,11 @@ inference/data/pilot_clusters.csv
 inference/data/group_centres.csv
 # Collected telemetry (one file per collection session, written by data_logger.py)
 inference/data/session_flight_data.csv
+# Written by live_groups.py for one flight, and the files the README's replay demo makes
+inference/data/live_flight_data.csv
+inference/data/live_flight_data_instructions.csv
+inference/data/replay_pilots.csv
+inference/briefs/replay.csv
 
 # Local build archives
 /xplane_plugin.zip

GLOSSARY.md

@@ -64,6 +64,10 @@ _Avoid_: Core metrics, KPIs
 A set of pilots whose **Deviation features** resemble each other. Groups are not ranked.
 _Avoid_: Tier (a **Synthetic pilot**'s known level is its **Hidden tier**, a different thing), skill level, class
 
+**Reference cohort**:
+The pilots whose **Deviation features** form the **Performance groups**, held fixed so a pilot outside them can be placed in a group.
+_Avoid_: Training set, baseline, reference group
+
 **Segment**:
 One of the seven phases of a **Flight**, from takeoff through landing.
 _Avoid_: Phase, section, leg (the five-value **Flight phase** in the out-of-scope section and a **Leg** are different things)
@@ -73,9 +77,17 @@ A stretch of the route flown on one heading, such as the 190° climb or the 220
 _Avoid_: Segment (the client's seven phases), phase
 
 **Instruction**:
-The timestamped moment a pilot is told a new **Target**, such as "change heading to 20". Instructions are standardized across pilots.
+A directive to the pilot that sets new **Targets** from the moment it is given, such as "change heading to 20". Targets it does not mention carry over from the one before. The client's instructions are standardized across pilots.
 _Avoid_: Command, prompt, target change
 
+**Brief**:
+The ordered list of **Instructions** for a **Flight**, each with the **Targets** it sets. The client's scenario briefing is one; so is one we write for a flight we fly ourselves.
+_Avoid_: Script, scenario, flight plan, briefing
+
+**Settle window**:
+The time after an **Instruction** during which the **Targets** it changes are not scored, so the turn or level-off it causes does not count as **Deviation**. Targets it leaves alone, or restates at the same value, are scored throughout.
+_Avoid_: Grace period, transition, buffer
+
 ### Pilots and flights
 
 **Pilot**:
@@ -93,7 +105,7 @@ _Avoid_: Pilot profile, demographics
 ### Synthetic pilots
 
 **Synthetic pilot**:
-A generated pilot with generated flights, built to deviate by a known amount, used to check that **Performance groups** find real differences before real pilots are available.
+A generated pilot with generated flights, built to deviate by a known amount, used to check that **Performance groups** find real differences, and as the **Reference cohort**, before real pilots are available.
 _Avoid_: Test pilot, fake pilot, mock pilot
 
 **Hidden tier**:
@@ -120,14 +132,16 @@ _Avoid_: Fake data, dummy data, mock data
 
 - A **Pilot** flies many **Flights**; a **Flight** produces many **Samples**.
 - A **Sample** has one value per telemetry field, and one **Deviation** per **Flight dynamics factor** once a **Target** exists.
-- A **Flight** is divided into **Segments**, and is also flown as a sequence of **Legs**. An **Instruction** sets a new **Target** from that moment on.
-- A **Pilot** falls into exactly one **Performance group**, decided by their **Deviation features** alone.
+- A **Flight** is divided into **Segments**, and is also flown as a sequence of **Legs**. A **Brief** lists its **Instructions** in order.
+- An **Instruction** sets new **Targets** from the moment it is given. The targets it changes are not scored during its **Settle window**.
+- A **Pilot** falls into exactly one **Performance group**, decided by their **Deviation features** alone. A pilot outside the **Reference cohort** is placed in the nearest group without changing the groups.
 - **Pilot metadata** is compared against **Performance groups** after they are formed; it does not form them.
 - Out of scope: a **Sample** receives exactly one **Flight event label**, and a **Flight phase** is a kind of **Flight event label**.
 
 ## Flagged ambiguities
 
 - **Pilot rating** is the type of certification, not a performance score (client, see `docs/client_meeting_notes.md`). Whether it means the same as certification level, or an extra rating such as instrument, is still unclear.
-- **Target** source is the **Instruction**: the client gives standardized, timestamped instructions during the flight, and iMotions holds the timestamps. None have been shared and no targets are captured yet, and the heading reference (true vs. magnetic) is unresolved.
+- **Target** source is the **Instruction**: the client gives standardized, timestamped instructions during the flight, and iMotions holds the timestamps. Her **Brief** and timestamps have not been shared, so targets exist only for flights we brief ourselves. The heading reference for her routes (true vs. magnetic) is unresolved.
+- Climbs are not settled: whether altitude **Deviation** counts while the pilot climbs to a new altitude **Target**, or only after level-off. A climb outlasts a **Settle window**. Our level-hold **Briefs** avoid the question; the client's scenario does not.
 - **Leg** and **Segment** may or may not line up. The client's seven **Segments** are unknown until she shares them, so how they map to our **Legs** is open.
 - **Segment** names and boundaries are unknown until the client shares her seven-phase segmentation, which she marks by hand from the iMotions audio and video. Detecting **Instructions** automatically, for example from the eye tracker's audio, is later work; today the client segments by hand.

docs/adr/0004-live-clustering-runs-in-a-python-sidecar.md

@@ -0,0 +1,17 @@
+# Live performance groups run in a Python sidecar, not in the relay
+
+To update **Performance groups** while a **Flight** is under way, the relay forwards each **Sample** over TCP to a separate Python program, the sidecar. The sidecar keeps the live pilot's running **Deviation features** and assigns the pilot to **Performance groups** fitted once on a reference cohort, using the existing functions in `inference/pilot_inference/performance/features.py` and `clustering.py`. The relay embeds no Python or ML and only shows the result. We chose this so there is one implementation of the clustering, the one the evidence in docs/adr/0002 was measured on.
+
+## Considered Options
+
+- **Port clustering to Rust inside the relay.** Rejected. It would duplicate KMeans, the log/floor scaling, the standardization over the cohort and the k-sweep stability check. Two copies could drift apart, and the ADR 0002 evidence would only describe one of them. The relay would also need an ML dependency for a 10 to 20 pilot cohort.
+- **Embed Python in the relay** (for example through PyO3). Rejected. It couples the relay's build and release to a Python runtime and scikit-learn, and it puts a slow refit on the process that must keep forwarding to iMotions.
+- **Python sidecar (chosen).** The sidecar is a separate process started like the other inference tools. A slow update, or a sidecar that is down, cannot block the relay's path to iMotions.
+
+## Consequences
+
+- The sidecar reads the relay's **Mirror** (docs/adr/0003), so the relay already forwards to iMotions unchanged when the sidecar is absent or slow. The copy is best effort, so the sidecar must tolerate dropped samples and must keep reading its socket on a thread separate from the slower work, or it loses the mirror.
+- The return channel (how the sidecar tells the relay a pilot's group) is not decided here.
+- The groups are fitted once and the live pilot is assigned to the nearest one, rather than refitting with the live pilot included. A refit would renumber the groups between updates, and one live pilot cannot form groups alone (`candidate_ks` needs 6). Until real pilots have **Targets**, the reference cohort is **Synthetic pilots**, so the live group is a demonstration rather than a measurement.
+- The client's **Targets** are not available (`docs/telemetry_schema.md` section 3). For our own X-Plane flights we are the instructor: we write a **Brief** of **Instructions**, and the operator marks each one live as it is spoken, so the sidecar knows when every target starts. The targets an instruction changes are not scored during its **Settle window**, so the turn or level-off is not scored, since **Synthetic pilots** fly only steady holds. Targets it leaves alone are scored throughout.
+- If the sidecar is later judged too heavy to run beside the relay, this decision should be revisited, since porting to Rust then means re-validating the clustering against ADR 0002's evidence.

docs/telemetry_schema.md

@@ -65,8 +65,11 @@ outliers, but they are not part of the deviation metric set.
 - The plugin only reads and sends **actual** instrument values. There is no
   DataRef read, UI control, or CSV column for a *designated/target* value for
   altitude, heading, vertical speed, or airspeed.
-- Nothing in `relay/`, `data_logger.py`, or the `inference/` scripts defines
-  or stores a target value either.
+- Nothing in `relay/` or `data_logger.py` defines or stores a target value
+  either. The one exception is a flight we fly ourselves: there we are the
+  instructor, and `live_groups.py` takes the targets from a **Brief** we wrote
+  and writes them into its CSV (docs/adr/0004, `inference/README.md`). The
+  client's own targets are still missing.
 
 To satisfy the client's requirement, the pipeline needs a source of target
 values per flight (or per flight segment), one of:
@@ -78,9 +81,9 @@ values per flight (or per flight segment), one of:
    some parameters (e.g. autopilot altitude/heading bugs) that could stand in
    for "designated" values when the autopilot is being used as the reference.
 
-Until one of these is implemented, `target_*` columns in section 4 will be
-unpopulated (`NaN`) and deviation cannot be computed from real data — only
-from synthetic data where targets are generated. **Flag this to the client
+Until one of these is implemented for the client's flights, `target_*`
+columns in section 4 will be unpopulated (`NaN`) for her data, and deviation
+can only be computed from synthetic data or from flights we brief ourselves. **Flag this to the client
 directly:** ask whether "designated" means an instructor-assigned briefing
 target, an autopilot bug setting, or something else, since the schema and
 ingestion path differ depending on the answer.
@@ -91,20 +94,20 @@ ingestion path differ depending on the answer.
 
 | Field | Type | Units | Source | Status |
 |---|---|---|---|---|
-| `timestamp` | ISO-8601 string, UTC | — | data_logger.py | exists |
-| `pilot_id` | string | — | `data_logger.py` | exists, see capture-point note below |
-| `flight_id` | string | — | `data_logger.py` | exists, see capture-point note below |
+| `timestamp` | ISO-8601 string, UTC | — | `data_logger.py` or `live_groups.py` | exists |
+| `pilot_id` | string | — | `data_logger.py` or `live_groups.py` | exists, see capture-point note below |
+| `flight_id` | string | — | `data_logger.py` or `live_groups.py` | exists, see capture-point note below |
 | `altitude` | float | ft MSL | plugin | exists |
-| `target_altitude` | float | ft MSL | scenario/autopilot (TBD, see §3) | **missing** |
+| `target_altitude` | float | ft MSL | our **Brief**, via `live_groups.py`; client's source TBD (§3) | ours only; client's **missing** |
 | `altitude_deviation` | float | ft | computed | derived, see §5 |
 | `heading` | float | deg magnetic | plugin | exists |
-| `target_heading` | float | deg magnetic | scenario/autopilot (TBD) | **missing** |
+| `target_heading` | float | deg magnetic | our **Brief**, via `live_groups.py`; client's source TBD | ours only; client's **missing** |
 | `heading_deviation` | float | deg | computed | derived, see §5 |
 | `vertical_speed` | float | ft/min | plugin | exists |
-| `target_vertical_speed` | float | ft/min | scenario/autopilot (TBD) | **missing** |
+| `target_vertical_speed` | float | ft/min | our **Brief**, via `live_groups.py`; client's source TBD | ours only; client's **missing** |
 | `vertical_speed_deviation` | float | ft/min | computed | derived, see §5 |
 | `velocity` (airspeed) | float | kts (indicated) | plugin | exists |
-| `target_airspeed` | float | kts | scenario/autopilot (TBD) | **missing** |
+| `target_airspeed` | float | kts | our **Brief**, via `live_groups.py`; client's source TBD | ours only; client's **missing** |
 | `airspeed_deviation` | float | kts | computed | derived, see §5 |
 | `roll` | float | deg | plugin | exists (extra, not part of deviation set) |
 | `pitch` | float | deg | plugin | exists (extra) |
@@ -119,7 +122,11 @@ anyway. `data_logger.py` takes `--pilot-id` and `--flight-id` and writes them
 into every row, which leaves the plugin, baton, and relay untouched. When
 `--flight-id` is omitted, a UTC timestamp is generated for each relay
 connection. When `--pilot-id` is omitted, rows are written as `unknown` and
-cannot be grouped per pilot.
+cannot be grouped per pilot. `live_groups.py`, which replaces the logger for a
+flight we brief, takes the same two arguments and also writes the `target_*`
+columns, blank where a target is not being scored (its **Settle window**). Its
+generated flight id is one per run rather than per connection, so a mirror
+reconnect continues the same flight.
 
 ### 4b. Pilot metadata table (one row per pilot, joined on `pilot_id`)
 

inference/README.md

@@ -11,14 +11,15 @@ Forest) is kept in its own folder, **out of scope** for the client's request; se
 ```
 inference/
   pyproject.toml, uv.lock
+  briefs/                       instructions with targets, for flights we fly ourselves
   data/                         input and output CSVs (not code)
     prepared/                   train/test splits written by event_classification
   models/                       trained models (not tracked; train_model.py writes them)
   pilot_inference/              the package
     angles.py                   wrapped angular difference, shared
     paths.py                    where data/ and models/ are
-    capture/                    data_logger.py
-    performance/                features, clustering, synthetic pilots and their CLIs
+    capture/                    data_logger.py, replay.py
+    performance/                features, clustering, synthetic pilots, live groups and their CLIs
     event_classification/       OUT OF SCOPE: labeler, Random Forest, evaluation
   tests/                        mirrors pilot_inference/
 ```
@@ -73,6 +74,9 @@ uv run python -m pilot_inference.performance.generate_synthetic_pilots
 uv run python -m pilot_inference.performance.compute_features --input data/synthetic_pilot_telemetry.csv
 uv run python -m pilot_inference.performance.cluster_pilots
 uv run python -m pilot_inference.performance.cluster_pilots --k 3 --truth data/synthetic_pilot_tiers.csv
+
+# Live groups during a flight (replaces the capture logger for that flight)
+uv run python -m pilot_inference.performance.live_groups --brief briefs/example_level_holds.csv --pilot-id pilot-07
 ```
 
 `--pilot-id` labels every row so samples can be grouped per pilot and joined to pilot metadata.
@@ -138,16 +142,75 @@ the adjusted Rand index against the hidden tiers; the file must list each pilot
 synthetic tiers overlap, so this is below 1: the default seed gives 0.79, and generator seeds 0 to
 11 range from 0.37 to 1.00.
 
+### Live groups
+
+`pilot_inference.performance.live_groups` is the sidecar from `docs/adr/0004`. It listens where
+the capture logger would (port 5001, the relay's mirror suggestion) and prints, every 5 seconds,
+the pilot's deviation features so far and the nearest performance group. Use it instead of the
+logger for that flight: it writes the same columns plus the `target_*` columns to
+`data/live_flight_data.csv`, and each instruction with its time to
+`data/live_flight_data_instructions.csv`. Both files are appended to, like the logger's session
+file, so one file can hold several flights and pilots. Running `compute_features` on it gives each
+flight the features the console showed for it in `flight_features.csv`; the per-pilot features
+average a pilot's flights.
+
+The client's targets are not available yet, so live groups work on flights we fly ourselves,
+where we are the instructor:
+
+1. Write a brief: a CSV with an `instruction` column and a target per factor (`heading`,
+   `altitude`, `airspeed`, `vertical_speed`), one row per instruction in order.
+   `briefs/example_level_holds.csv` is four level holds. The first instruction sets all four
+   targets; after that a blank cell keeps the target from the instruction before, as a real
+   "turn right heading 360" keeps the altitude. Headings are magnetic, as the X-Plane heading
+   indicator and the plugin show them.
+2. Start `live_groups` with the brief and a `--pilot-id`, then press `Connect mirror` in the relay
+   (or `Connect TCP` with no iMotions). It replaces the capture logger for the flight, so stop
+   `data_logger` first; if the port is still taken, `live_groups` says so and exits.
+   On Windows and Linux keep the plugin's "Positional Flight Data" window open the whole flight:
+   the stream stops when it closes (see the root README), and the status line's scored time then
+   stops growing.
+3. Press Enter at the moment each instruction is spoken. The targets it changes are not scored
+   for 30 seconds (`--settle`), so the turn or level-off does not count; targets it leaves alone,
+   or restates at the same value, are scored throughout, so altitude lost in a turn counts.
+   Type `q` and Enter to stop. The sidecar ends the relay connection, writes every sample it
+   already received, then closes its files.
+
+The sidecar is used on Windows, the lab machine's OS. It also stops cleanly on Linux and macOS,
+where closing a listening socket alone does not wake a waiting `accept()`, so stopping before a
+relay connects would otherwise hang. That case was checked under WSL; the tests run on Windows.
+
+No group is shown until every factor has 60 seconds scored (`--min-scored`). Once one is, it
+stays shown while a later instruction's factors settle, since the others are still scored. The
+groups are fitted once at start-up on a reference cohort of 15 synthetic pilots with k = 3, and
+the live pilot is placed in the nearest one without refitting, so the group numbers do not change
+during the flight. The cohort is
+synthetic, so the group demonstrates the pipeline; it does not measure the pilot against real
+pilots.
+
+To try it without X-Plane, replay a synthetic flight the cohort does not contain (another seed),
+using its targets as a one-instruction brief:
+
+```
+uv run python -c "from pilot_inference.performance.generate_synthetic_pilots import generate_pilots; generate_pilots(3, seed=7)[0].to_csv('data/replay_pilots.csv', index=False)"
+uv run python -m pilot_inference.capture.replay --input data/replay_pilots.csv --flight P03-F1 --brief-out briefs/replay.csv
+uv run python -m pilot_inference.performance.live_groups --brief briefs/replay.csv --settle 0
+# in a second terminal, then press Enter in the first
+uv run python -m pilot_inference.capture.replay --input data/replay_pilots.csv --flight P03-F1
+```
+
+`replay` sends a telemetry CSV in the relay's packet format at 20 Hz (`--hz`), so either listener
+reads it like a live flight. It needs a single flight, so pick one with `--flight`.
+
 ## Tests
 
 ```
 uv sync              # installs pytest from the dev group
 uv run pytest
 ```
 
-`tests/` mirrors the package: `tests/capture/`, `tests/performance/` (deviation features, the
-synthetic pilot generator and performance groups, including recovering the synthetic tiers end to
-end), `tests/event_classification/` (the labeler's heading change), and `tests/test_angles.py`
+`tests/` mirrors the package: `tests/capture/` (the logger and replay), `tests/performance/`
+(deviation features, the synthetic pilot generator, performance groups including recovering the
+synthetic tiers end to end, and live groups from replayed packets to a saved CSV), `tests/event_classification/` (the labeler's heading change), and `tests/test_angles.py`
 for the angle helper.
 
 ## Out of scope: event classification