Skip to content

fix(runners): a runner no longer reuses another runner's cached steps - #78

Merged
ramondalmau merged 2 commits into
devfrom
fix/step-cache-per-runner
Oct 22, 2026
Merged

ramondalmau merged 2 commits into
devfrom
fix/step-cache-per-runner

Conversation

@ramondalmau

Copy link
Copy Markdown
Collaborator

Problem

A FleetRunner does not build its steps (parser, BADA performance model, emissions, climate)
once at the start. Each worker process builds them the first time it needs them and keeps
them in a per-process cache, so the next batch of flights reuses them.

That cache was looked up by step name only ("neats" parser, "bada" performance, ...), not by
the settings the step was built with. So if a second runner ran in the same Python process,
it got the steps built for the first runner, with the first runner's settings.

Example (both runners in one Python session, njobs=1):

r1 = FleetRunnerLargeEmitter(FleetRunnerParams(..., njobs=1))                       # no airport fuel file
r1.eval()
r2 = FleetRunnerLargeEmitter(FleetRunnerParams(..., airport_fuel_path=AF, njobs=1))
r2.eval()

Before this fix, r2 used r1's parser, built without the airport fuel file, so AF was silently
ignored. In the opposite order, the second runner applied a file it was not given. The same
holds for any step setting (bada_path, emission or climate parameters).

Who is affected

Only when one Python process runs two or more runners with different settings, AND the steps
run inside that process or in reused workers:

  • njobs=1;
  • prefer="threads";
  • shutdown_executor_between_steps=False;
  • njobs=-1 on a machine or job with only one usable CPU (joblib then runs everything in the
    calling process).

Typical cases: a notebook, a script comparing two configurations, the test suite.
A normal production run (one runner per process, default settings, several CPUs) was not
affected, so past results are unchanged.

Fix

Each runner's step gets its own token when the runner is created. The cache keeps the token
next to the step: a runner with another token builds its own step. Within one runner the
token is the same in every worker, so each worker still builds each step only once
(no slowdown). At most one step per type is kept in memory, as before.

Checks

  • 4 new tests (tests/test_step_cache.py), including two real FleetRunners in one process
    with and without the airport fuel file, in both orders. The 3 bug tests fail on the old code.
  • Full test suite on this branch (rebased on dev with fix(fuel): reject a q_fuel outside 35-50 MJ/kg (unit error guard) #77): 325 passed, 0 failed.
  • Independent review: no defects.
  • Audit: new item I11 in docs/internal/pyneats_units_audit.md.

ramondalmau and others added 2 commits October 22, 2026 09:04
FleetRunner builds its steps lazily and caches them per process. The
cache was keyed by (interface, name) only, so with njobs=1, threads or a
reused worker pool, a second runner in the same process reused the first
runner's steps and their parameters (e.g. its parser with or without the
airport fuel file, its bada_path). The default settings were not
affected.

Each make_step_func callable now gets a unique token stored with the
cached step; another token rebuilds and replaces the entry. Within one
runner the token travels with the pickled callable, so each worker still
builds each step once. At most one step per (interface, name) is kept.

Tests: two callables keep their own params; one callable (and its
pickled copies) builds once per process; two FleetRunners in one process
(njobs=1) each apply their own airport fuel file, in both orders. The
bug tests fail on the previous code. Audit: new item I11.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joblib runs every task in the calling process when only one CPU is
usable, so a 1-core job or container with the default settings was also
affected. Code comment and audit I11 corrected (review finding); note
that concurrent runners in one process rebuild their steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ramondalmau
ramondalmau merged commit 0ef52ce into dev Oct 22, 2026
6 checks passed
@ramondalmau
ramondalmau deleted the fix/step-cache-per-runner branch October 22, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant