Repository navigation
fix(interpolation): skip empty values in operator columns when resampling (audit D4) - #81
Merged
Merged
Conversation
…audit D4) An AO column (fuel flow, mass, efficiency, TAS) with an empty value at one point made every 1-min sample between its two neighbours NaN, so those minutes had no value and dropped out of the totals. Measured on dev (EAF2143, AO points about every 5 min, one empty fuel flow): 15 minutes lost, fuel -12.1 % (mass: -14.9 %). On the DataFrame path a column empty at every point was kept as all-NaN: the flight got no fuel at all. Maintainer decision (Ramon Dalmau): skip empty values, as if the point had not been sent. Each optional column is interpolated on its valid points only; the nearest valid value is used before the first and after the last one; a column with no valid value is dropped (computed as when not provided). Gaps are a straight line, not BADA. Example above: +1.6 % against the complete data instead of -12.1 %. Effect: NM sample (4,959 flights) bit-identical (no AO columns). Golden data: 57 records change (empty AO values near the end of the flight): fuel/CO2 +0.03 % to +0.96 %; only those records replaced. Tests: interior and edge gaps, per-column validity, complete column unchanged, fully empty column dropped (5 fail on the previous code). Docs: input-format.md (empty values), audit D4. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review follow-ups: the all-empty column bug concerned a DataFrame given directly to FlightRunner or the parser (JSON and split_df_into_flights already dropped it); 47 of the 57 golden records regain their last minute of fuel (+0.0002 % to +0.96 %), 10 change only efficiency/TAS results; empty mass value about -15 %; fuel flow may come from the mass; a held edge mass now gives 0 instead of empty (same total). The pop() branch could not run (resample_and_fill keeps only time and position), so it is removed; behaviour unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ramondalmau
added a commit
that referenced
this pull request
Oct 28, 2026
…utput (D-a) Text only; no code or behaviour change. - Empty AO values (#81): mrv-specification.md and input-prioritization.md said an empty value empties its neighbours and drops out of the fuel totals. It is interpolated in time between the nearest points with a value, as input-format.md already says. - Fuel flow derived from the AO mass (#79): describe the current formula (mass decrease to the next point / time to the next point). - output-format.md rewritten around runner.results: success entries, low-altitude zero results and error records (order, keys, examples that run), fleet_meta_data keys from report.py, bada_version / bada_code absent on the AO fuel flow + efficiency + mass path, multi-engine entries. The per-segment field list moves to a "Golden dump" section. - flight_schema.json: description strings only (hydrogen content wt.%, calorific value J/kg, departure/arrival times as supplied, EDB engine UID, load factor basis, whole-aircraft fuel flow, propulsion efficiency, flight level, TAS in m/s). - usage.md: working ZarrPaths and DWDFactory snippets; examples/README.md and example-scripts.md: units and the efficiency definition; EAGWP units W m-2 yr in the MRV formulas; hydrogen content formula in wt.%. Checked by independent reviewers over 5 rounds against the code on dev; every code example in output-format.md and the changed usage.md snippets were run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ramondalmau
added a commit
that referenced
this pull request
Oct 28, 2026
…utput (D-a) (#88) Text only; no code or behaviour change. - Empty AO values (#81): mrv-specification.md and input-prioritization.md said an empty value empties its neighbours and drops out of the fuel totals. It is interpolated in time between the nearest points with a value, as input-format.md already says. - Fuel flow derived from the AO mass (#79): describe the current formula (mass decrease to the next point / time to the next point). - output-format.md rewritten around runner.results: success entries, low-altitude zero results and error records (order, keys, examples that run), fleet_meta_data keys from report.py, bada_version / bada_code absent on the AO fuel flow + efficiency + mass path, multi-engine entries. The per-segment field list moves to a "Golden dump" section. - flight_schema.json: description strings only (hydrogen content wt.%, calorific value J/kg, departure/arrival times as supplied, EDB engine UID, load factor basis, whole-aircraft fuel flow, propulsion efficiency, flight level, TAS in m/s). - usage.md: working ZarrPaths and DWDFactory snippets; examples/README.md and example-scripts.md: units and the efficiency definition; EAGWP units W m-2 yr in the MRV formulas; hydrogen content formula in wt.%. Checked by independent reviewers over 5 rounds against the code on dev; every code example in output-format.md and the changed usage.md snippets were run. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (audit D4)
Operators can send their own fuel flow, aircraft mass, engine efficiency or true airspeed per
trajectory point, and may leave some values empty. PyNEATS resamples every trajectory to one
point per minute by drawing a straight line between neighbouring operator points.
An empty value broke the two lines touching it: every minute between the previous and the next
operator point got no value. Those minutes counted as zero fuel, with no warning.
Example (golden flight EAF2143, operator points about every 5 minutes, fuel flow empty at 19:12):
Total fuel: 3,988 kg with the value; 3,505 kg before this fix (-12.1 %); 4,051 kg with this fix
(+1.6 %: a straight line cannot know the dip at 19:12). An empty mass value: about -15 % before.
With dense operator data (a point every ~25 s) one empty value usually changed nothing.
Also: a column empty at every point, in a DataFrame given directly to FlightRunner or the
parser, was used as all-empty, so the flight got no fuel at all.
Fix (decision: Ramon Dalmau)
An empty value is skipped, as if that point had not been sent: each minute is interpolated
between the nearest valid operator values; before the first / after the last valid value the
nearest one is used. A column empty at every point is treated as not provided (computed by
PyNEATS). Gaps are filled by a straight line, not by BADA (no mixing within a column).
Effect
values (near the end of the flight). 47 regain their last minute of fuel (fuel and CO2
+0.0002 % to +0.96 %); 10 change only efficiency- or speed-dependent results. Every other
record is byte-identical.
Checks
Known limit: an empty mass value at the very start or end is held constant, so no fuel is
derived there (same total as before).