Skip to content

fix(interpolation): skip empty values in operator columns when resampling (audit D4) - #81

Merged
ramondalmau merged 2 commits into
devfrom
fix/partial-null-columns
Oct 23, 2026
Merged

ramondalmau merged 2 commits into
devfrom
fix/partial-null-columns

Conversation

@ramondalmau

Copy link
Copy Markdown
Collaborator

Problem (audit D4)

Operators can send their own fuel flow, aircraft mass, engine efficiency or true airspeed per
trajectory point, and may leave some values empty. PyNEATS resamples every trajectory to one
point per minute by drawing a straight line between neighbouring operator points.

An empty value broke the two lines touching it: every minute between the previous and the next
operator point got no value. Those minutes counted as zero fuel, with no warning.

Example (golden flight EAF2143, operator points about every 5 minutes, fuel flow empty at 19:12):

Time If the value had been sent Before this fix With this fix
19:02 0.685 kg/s 0.685 0.685
19:03 0.664 missing 0.676
... ... missing (15 min) ...
19:17 0.530 missing 0.537
19:18 0.491 0.491 0.491

Total fuel: 3,988 kg with the value; 3,505 kg before this fix (-12.1 %); 4,051 kg with this fix
(+1.6 %: a straight line cannot know the dip at 19:12). An empty mass value: about -15 % before.
With dense operator data (a point every ~25 s) one empty value usually changed nothing.

Also: a column empty at every point, in a DataFrame given directly to FlightRunner or the
parser, was used as all-empty, so the flight got no fuel at all.

Fix (decision: Ramon Dalmau)

An empty value is skipped, as if that point had not been sent: each minute is interpolated
between the nearest valid operator values; before the first / after the last valid value the
nearest one is used. A column empty at every point is treated as not provided (computed by
PyNEATS). Gaps are filled by a straight line, not by BADA (no mixing within a column).

Effect

  • NM sample (4,959 flights): bit-identical. These flights carry no operator performance data.
  • Golden test data: 57 flight records change, exactly those whose inputs contain empty operator
    values (near the end of the flight). 47 regain their last minute of fuel (fuel and CO2
    +0.0002 % to +0.96 %); 10 change only efficiency- or speed-dependent results. Every other
    record is byte-identical.

Checks

  • 6 new tests (tests/test_interpolation_partial_nulls.py); 5 fail on the old code.
  • Full test suite: 337 passed. Golden pipeline tests rerun after a final cleanup: 25 passed.
  • Independent review: no code defects; its wording corrections are applied.
  • Docs: input-format.md explains empty values; audit D4 closed with the numbers above.

Known limit: an empty mass value at the very start or end is held constant, so no fuel is
derived there (same total as before).

ramondalmau and others added 2 commits October 22, 2026 15:10
…audit D4)

An AO column (fuel flow, mass, efficiency, TAS) with an empty value at one
point made every 1-min sample between its two neighbours NaN, so those
minutes had no value and dropped out of the totals. Measured on dev
(EAF2143, AO points about every 5 min, one empty fuel flow): 15 minutes
lost, fuel -12.1 % (mass: -14.9 %). On the DataFrame path a column empty
at every point was kept as all-NaN: the flight got no fuel at all.

Maintainer decision (Ramon Dalmau): skip empty values, as if the point
had not been sent. Each optional column is interpolated on its valid
points only; the nearest valid value is used before the first and after
the last one; a column with no valid value is dropped (computed as when
not provided). Gaps are a straight line, not BADA. Example above: +1.6 %
against the complete data instead of -12.1 %.

Effect: NM sample (4,959 flights) bit-identical (no AO columns). Golden
data: 57 records change (empty AO values near the end of the flight):
fuel/CO2 +0.03 % to +0.96 %; only those records replaced.

Tests: interior and edge gaps, per-column validity, complete column
unchanged, fully empty column dropped (5 fail on the previous code).
Docs: input-format.md (empty values), audit D4.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review follow-ups: the all-empty column bug concerned a DataFrame given
directly to FlightRunner or the parser (JSON and split_df_into_flights
already dropped it); 47 of the 57 golden records regain their last
minute of fuel (+0.0002 % to +0.96 %), 10 change only efficiency/TAS
results; empty mass value about -15 %; fuel flow may come from the mass;
a held edge mass now gives 0 instead of empty (same total). The
pop() branch could not run (resample_and_fill keeps only time and
position), so it is removed; behaviour unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ramondalmau
ramondalmau merged commit 5aba2f9 into dev Oct 23, 2026
6 checks passed
@ramondalmau
ramondalmau deleted the fix/partial-null-columns branch October 23, 2026 04:43
ramondalmau added a commit that referenced this pull request Oct 28, 2026
…utput (D-a)

Text only; no code or behaviour change.

- Empty AO values (#81): mrv-specification.md and input-prioritization.md
  said an empty value empties its neighbours and drops out of the fuel
  totals. It is interpolated in time between the nearest points with a
  value, as input-format.md already says.
- Fuel flow derived from the AO mass (#79): describe the current formula
  (mass decrease to the next point / time to the next point).
- output-format.md rewritten around runner.results: success entries,
  low-altitude zero results and error records (order, keys, examples that
  run), fleet_meta_data keys from report.py, bada_version / bada_code absent
  on the AO fuel flow + efficiency + mass path, multi-engine entries. The
  per-segment field list moves to a "Golden dump" section.
- flight_schema.json: description strings only (hydrogen content wt.%,
  calorific value J/kg, departure/arrival times as supplied, EDB engine UID,
  load factor basis, whole-aircraft fuel flow, propulsion efficiency, flight
  level, TAS in m/s).
- usage.md: working ZarrPaths and DWDFactory snippets; examples/README.md and
  example-scripts.md: units and the efficiency definition; EAGWP units
  W m-2 yr in the MRV formulas; hydrogen content formula in wt.%.

Checked by independent reviewers over 5 rounds against the code on dev;
every code example in output-format.md and the changed usage.md snippets
were run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ramondalmau added a commit that referenced this pull request Oct 28, 2026
…utput (D-a) (#88)

Text only; no code or behaviour change.

- Empty AO values (#81): mrv-specification.md and input-prioritization.md
  said an empty value empties its neighbours and drops out of the fuel
  totals. It is interpolated in time between the nearest points with a
  value, as input-format.md already says.
- Fuel flow derived from the AO mass (#79): describe the current formula
  (mass decrease to the next point / time to the next point).
- output-format.md rewritten around runner.results: success entries,
  low-altitude zero results and error records (order, keys, examples that
  run), fleet_meta_data keys from report.py, bada_version / bada_code absent
  on the AO fuel flow + efficiency + mass path, multi-engine entries. The
  per-segment field list moves to a "Golden dump" section.
- flight_schema.json: description strings only (hydrogen content wt.%,
  calorific value J/kg, departure/arrival times as supplied, EDB engine UID,
  load factor basis, whole-aircraft fuel flow, propulsion efficiency, flight
  level, TAS in m/s).
- usage.md: working ZarrPaths and DWDFactory snippets; examples/README.md and
  example-scripts.md: units and the efficiency definition; EAGWP units
  W m-2 yr in the MRV formulas; hydrogen content formula in wt.%.

Checked by independent reviewers over 5 rounds against the code on dev;
every code example in output-format.md and the changed usage.md snippets
were run.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant