perf: cut ~23% off planning by removing redundant scans and copies by springfall2008 · Pull Request #4549 · springfall2008/batpred

springfall2008 · 2026-08-16T17:48:05Z

Four independent changes to the Python side of planning, which the previous round
(#4540 / #4546) left as ~88% of plan time. Each was found by profiling, each is
semantics-preserving, and every one is gated on the 20-scenario byte-identical comparison.

Result: optimisation time over the 20-scenario suite goes from 26.13s to 20.15s, 22.9% faster, with every scenario producing an identical plan.

The measurement

main (2eb3466) against this branch, interleaved rather than grouped, best of three, on a
16-core machine with the kernel confirmed active and batch-capable on both sides of every run.

20-scenario suite	`main`	this branch	change
Optimisation time (sum of per-scenario)	26.130s	20.149s	22.9% faster
id 6 (heaviest)	5.741s	4.036s	29.7% faster
id 15	2.311s	1.481s	35.9% faster
id 2	1.530s	0.889s	41.9% faster

960 fields compared across three rounds of all 20 scenarios — metric, cost, both PV futures,
final SoC, cycles and carbon — zero mismatches. The heaviest scenarios gain most, which is what
matters: ids 6 and 19 alone were 42% of suite runtime.

What changed

1. Scan window candidates once per price threshold

optimise_charge_limit_price_threads sweeps four nested loops of slot counts and freeze flags
against each price threshold. The charge side was already memoised; the export side was not, so it
rescanned the whole price_set_export list — a few hundred entries — for every one of the ~900
combinations per threshold, for at most a couple of dozen distinct answers.

Both sides now scan once per (threshold, freeze) and every slot count is served by slicing that
one list. That is sound because of the prefix property: capping at max_slots takes exactly the
first max_slots of the unbounded selection, since the original cap was monotonic — once reached,
nothing further was ever taken. Slot counts at or above the candidate count therefore collapse onto
one cache entry instead of one per entry in the slot-length list.

Self time 607ms → 315ms on the profiled scenario.

2. Only copy `minute_data`'s history when the glitch filter can write to it

minute_data deep-copied every history list it was not explicitly told it could modify. The only
code in it that writes to history is the glitch filter, which runs solely for backwards incrementing
data — so every other caller paid for a copy nothing could use.

Measured on realistic Home Assistant history, the copy was 79-88% of the whole call, so the
affected calls are now roughly twice as fast (20k entries: 54.8ms → 29.1ms). That lands on the
production fetch path, which the fixture-driven benchmark does not exercise.

3. Share the load-independent kernel context across the marginal cost matrix

calculate_marginal_costs builds 28 Predictions — four load levels across seven time offsets — that
differ only in their load forecast, and each rebuilt the entire kernel context: 576 steps of rate,
PV, carbon, temperature and car-slot lookups, 27 times for an answer already computed.

Profiling settled the approach: the whole cost is Python array building, and pk_context_create does
not register at all, so no ABI change was needed. create_kernel_context takes an opt-in
static_cache; only calculate_marginal_costs uses it, where "only the load differs" holds by
construction. The temperature caps are also memoised per distinct temperature, which helps every
context build including the main plan's.

calculate_marginal_costs 69ms → 25ms; find_battery_temperature_cap 17,856 calls → 3,471.

4. Stop deep-copying the export window list in `optimise_export`

optimise_export deep-copied the whole export window list on entry, on every one of its ~1000 calls
per plan — 2.5 million deepcopy calls on the heaviest scenario, the largest single block of copying
left in a plan.

It never needed one. Nothing in optimise_export writes to a window dict, and the only write on this
path — the trial start — is already applied copy-on-write by _prepare_export (since #4540). The
list is still copied, shallowly, so a caller cannot reorder it underneath a batch that has not
flushed.

Tests

Every new test was verified by deliberately breaking what it guards and confirming the right one
fails:

Caching the volatile export starts, or dropping the dedup, fails the prefix-property test — which
is checked against a reference implementation of the original capped loop.
Removing minute_data's copy fails the test that the glitch filter must not write through to the
caller. A dict subclass counts its own deep copies, so "no copy when nothing can be written" is a
behaviour assertion rather than a timing one.
Caching the load alongside the static kernel arrays — the exact bug that cache could introduce —
makes cell two answer with cell one's load and fails both new tests.
Making _prepare_export write the start in place corrupts the caller's window from 720 to 835 and
fails the export-copy guard. That is the defect the copy-on-write was introduced to fix, now pinned
rather than left to the next reader.

One test earned its keep immediately: a distinctness assertion caught the kernel-cache fixture having
no rates, so every prediction cost 0.0 and the equivalence test would have compared 0.0 against 0.0
and passed regardless of whether the cache leaked.

Alongside that: the full suite of 215 tests, pre-commit clean, and the 20-scenario byte-identical
gate re-run at every step.

Also fixed

coverage/run_random_profile pointed at random_scenarios.yaml while run_random benchmarks
cases/random_scenarios.yaml. Those were different files, so every profile was taken against
different scenarios from every benchmark — and since the former was never committed, the script could
not run at all on a fresh clone. Now points at cases/.

Relationship to #4536

#4536 overlaps at one line: it changes the same
optimise_export deepcopy to clone_windows. Its stated rationale — that
thread_run_prediction_export writes the start in place — no longer holds on main, since #4540 moved
that to copy-on-write in _prepare_export, shared by both the threaded and batch paths. With no dict
mutated, the per-dict copies are unnecessary and a plain list() suffices.

#4536's window-bounds cache does not overlap with anything here and remains the stronger answer for
the prediction cache key; a version of that was explored on perf/cache-key-window-memo and parked
in its favour.

🤖 Generated with Claude Code

…er planning) optimise_charge_limit_price_threads sweeps four nested loops of slot counts and freeze flags against each price threshold, and rebuilt its charge and export selections from scratch inside the innermost one. The charge side was already memoised per (max_charge_slots, try_charge_freeze); the export side was not, so it rescanned the whole price_set_export list - a few hundred entries - for every one of the ~900 combinations per threshold, for at most a couple of dozen distinct answers. That scan was 607ms of self-time on the heaviest benchmark scenario, and 737k hit_car_window calls. Both sides now scan once per (threshold, freeze) via select_window_candidates and every slot count is served by slicing that one list. Capping at max_slots is exactly taking the first max_slots of the unbounded selection, because the cap in the original scan was monotonic - once reached, nothing further was ever taken. Slot counts at or above the candidate count therefore collapse onto one cache entry instead of one per entry in the slot length list. Two supporting changes: the car/iboost filter is memoised per export window, which is sound for the same reason hit_charge_cache is - the optimiser turns windows on and off but never moves a start or end; and the cached export pair is copied on write, since the charge-collision pruning below mutates it and that pruning genuinely varies per charge combination. Measured interleaved against main, best of three, kernel active with batch support on both sides: per-scenario optimisation time over the 20-scenario suite 26.37s -> 21.42s, 18.8% faster. The heaviest scenarios gain most - id 6 25.4%, id 2 36.8%, id 15 26.2%. Function self-time 607ms -> 315ms. Plans are unchanged: 960 fields compared across three rounds of all 20 scenarios (metric, cost, both PV futures, SoC, cycles, carbon), zero mismatches. test_window_selection pins the prefix property against a reference implementation of the original capped loop, since that equivalence is what makes the slicing sound. Both new tests were verified by breaking the picker and watching the right one fail: marking rejected windows as seen fails the retry test, removing the dedup fails the prefix test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…an write to it minute_data deep-copied every history list it was not explicitly told it could modify. The only code in it that writes to history is the glitch filter, which runs solely for backwards incrementing data, so every other caller paid for a copy that nothing could have used. That is most of them. The two calculate_yesterday calls both pass clean_increment=False and between them accounted for ~150k deepcopy calls per plan cycle, including 17k datetime __reduce_ex__ round trips, because the copy walks every sample dict and its nested attributes. The copy is now taken only when clean_increment and backwards are both set, with can_modify_history remaining the caller's explicit opt-out on top of that. The guarantee callers actually rely on - that minute_data does not write to history they still intend to read - is unchanged, because the case where it could write is exactly the case that still copies. Measured by cProfile on the benchmark scenarios: minute_data 95ms -> 3ms on scenario 0 and 102ms -> 3ms on scenario 6, and calculate_yesterday 126ms -> 35ms. Whole-plan deepcopy calls drop from 148,881 to 45,199 on scenario 0. Plans are unchanged: 320 fields across all 20 scenarios, zero mismatches, and the full suite of 213 tests passes. All four new tests were verified by breaking what they guard. A dict subclass counts its own deep copies, so "no copy when nothing can be written" is a behaviour assertion rather than a timing one; removing the copy entirely fails the test that the filter must not write through to the caller. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…cost matrix calculate_marginal_costs builds 28 Predictions - four extra-load levels across seven time offsets - that differ only in their load forecast. Each one rebuilt the entire kernel context: 576 steps of rate, PV, alert, carbon, gas, iboost, car-slot and battery-temperature lookups, 27 times for an answer already computed. Profiling settled how to fix it. The whole cost is the Python array building - 39ms of self time plus 16ms in find_battery_temperature_cap - while pk_context_create does not register at all. So this needed no ABI change, just somewhere to keep the arrays. create_kernel_context now takes an opt-in static_cache, and the load-independent arrays move to build_static_context_arrays so they can be built once and reused. The contract is the caller's to keep and is documented there: anything other than the load differing between contexts sharing a cache would be silently taken from the first build. Only calculate_marginal_costs opts in, where it holds by construction - same base, same PV arrays, only the injected load varies. Two smaller changes alongside. The temperature rate caps are memoised per distinct temperature rather than recomputed per step, which helps every context build including the main plan's: 17,856 calls per plan become 3,471. And the load forecast is copied with dict() rather than copy.deepcopy - it is a flat minute -> kWh dict, so deepcopy only walked it more slowly. Measured by cProfile on the benchmark scenarios: calculate_marginal_costs 69ms -> 25ms, create_kernel_context 71.5ms -> 28ms across the plan, and build_static_context_arrays runs 4 times per plan rather than 31. Plans are unchanged: 320 fields across all 20 scenarios, zero mismatches, and the full suite of 214 tests passes. Both new tests were verified by breaking what they guard. Caching the load alongside the static arrays - the exact bug the cache could introduce - makes the second cell answer with the first cell's load, and both tests fail. The distinctness test earned its place immediately: it caught the first fixture having no rates, so every prediction cost 0.0 and the equivalence test would have compared 0.0 against 0.0 and passed no matter what. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…e_export call optimise_export deep-copied the whole export window list on entry - every window dict, on every one of its ~1000 calls per plan. On the heaviest benchmark scenario that was 2.5 million deepcopy calls and the largest single block of copying left in a plan. It never needed one. Nothing in optimise_export writes to a window dict; the only write anywhere on this path is the trial start, and _prepare_export already applies that copy-on-write, taking its own list and replacing the single window it changes with dict(window, start=start). The list is still copied, shallowly, so a caller cannot reorder it underneath a batch that has not flushed yet. test_optimise_export_copy pins both halves: that no deep copy is taken, and that the caller's window dicts come back untouched - the guarantee the deepcopy was providing, now provided by _prepare_export. Verified by making _prepare_export write the start in place instead, which corrupts the caller's window from 720 to 835 and fails the second test. That is the same defect the copy-on-write was introduced to fix, so it is now pinned rather than left to the next reader. Also fixes coverage/run_random_profile, which pointed at random_scenarios.yaml while run_random uses cases/random_scenarios.yaml. Those were different files, so every profile was taken against different scenarios from every benchmark - and since the former was never committed, the script could not run at all on a fresh clone. The stale local copy is deleted. Plans are unchanged: 320 fields across all 20 scenarios, zero mismatches, and the full suite of 215 tests passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Performance-focused refactor of Predbat’s Python-side planning loop to reduce redundant scanning/copying and to reuse load-independent kernel context for marginal cost simulations, while preserving plan semantics via new targeted unit tests.

Changes:

Memoise price-threshold window candidate selection in optimise_charge_limit_price_threads and introduce select_window_candidates to support slicing-based caps.
Reduce unnecessary deep copies (HA history copying in minute_data, export window copying in optimise_export, load dict copying in calculate_marginal_costs).
Add an opt-in static cache path for kernel context construction and introduce regression tests for the above invariants.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`coverage/run_random_profile`	Fix scenario file path so profiling runs against committed benchmark scenarios.
`apps/predbat/utils.py`	Avoid deep-copying history unless the glitch filter can actually mutate it.
`apps/predbat/marginal.py`	Reuse a shared kernel static-cache across marginal-cost matrix cells; replace unnecessary deepcopy of load dict.
`apps/predbat/prediction.py`	Thread `kernel_static_cache` through `Prediction` into kernel context creation.
`apps/predbat/prediction_kernel.py`	Split and cache load-independent kernel context arrays (`static_cache`) to avoid rebuilding across marginal-cost cells.
`apps/predbat/plan.py`	Add `select_window_candidates`; memoise charge/export candidate selection per (threshold, freeze); remove `deepcopy` from `optimise_export` in favour of shallow list copy.
`apps/predbat/unit_test.py`	Register new tests for window selection, export copy semantics, minute_data copying, and kernel static cache.
`apps/predbat/tests/test_window_selection.py`	New tests pinning prefix-property and selection semantics for threshold window picking.
`apps/predbat/tests/test_optimise_export_copy.py`	New tests ensuring no export-window deepcopy and caller windows remain unmodified.
`apps/predbat/tests/test_minute_data_copy.py`	New tests pinning when history copying happens and that glitch filtering doesn’t write through unless allowed.
`apps/predbat/tests/test_kernel_static_cache.py`	New tests validating kernel static-cache correctness across differing loads.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+def run_one(my_predbat, pv_step, load_step, static_cache):
+    """Build a Prediction for this load and return its prediction result"""
+    prediction = Prediction(my_predbat, pv_step, pv_step, load_step, load_step)
+    prediction.kernel_handle = create_kernel_context(prediction, static_cache=static_cache)
+    if not prediction.kernel_handle:
+        return None


…4550) * test: vary iboost and low power export across the random scenarios The 20-scenario byte-identical benchmark is the gate every planner change is held to, but three feature flags were pinned by the template for all 20 runs: low power charge always on, iboost and low power export always off. Anything guarded by the two disabled ones could be changed - or broken - without the gate noticing. That was not theoretical. The iboost arm of export_window_allowed, added in #4549, could not be reached by any scenario: it needs iboost_enable, a non-empty iboost_plan and iboost_on_export off. It is now taken 126 times across the suite. kernel_parity did cover iboost for prediction, but it never runs the optimiser, so nothing reached that branch. iboost_enable is drawn at 40% (matching kernel_parity) and set_export_low_power 50/50, from an rng salted off the scenario seed rather than the main stream - the same device the car block uses, and for the same reason: every pre-existing parameter and stored profile is bit-identical, so a plan that moves has moved because of the new flags and nothing else. Verified: 0 drift across all 20 scenarios, and with the flags present but unapplied the plans still match the old baseline exactly. The whole iboost block is written on every scenario rather than only the enabled ones. Setting it only when enabled left the previous scenario's values behind, so a plan depended on what ran before it - the same trap run_debug_cases documents, and it moved all 20 scenarios instead of the 13 the flags actually touch. With the off-case values taken from the template, the 7 scenarios neither flag touches are byte-identical to before, which is what makes the regenerated baseline reviewable. iboost_plan is built here on the same condition fetch_sensor_data uses, because the scenario runner never calls fetch and an empty plan cannot reach the optimiser path this exists to cover. Also fixes the static context cache fixture, which ran with iboost off and no battery temperature curves - so iboost_plan_load was 288 zeroes and every temperature produced the identical cap. Both are what the cache reuses, so the equivalence tests would have agreed whatever it did with them. Now 36 of 288 iboost steps are non-zero and the caps take 3 distinct values. Baseline regenerated: 13 scenarios move, 7 are unchanged, 217 tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: run the random benchmark in the suite, and randomise the load scalings Three changes to the random scenario benchmark. The 20-scenario plan comparison now runs as part of run_all rather than only by hand. It fails if any recorded field differs from the committed baseline, and reports runtime without asserting on it - the suite runs on machines of wildly different speeds, so a runtime threshold would either be so loose it caught nothing or so tight it failed for reasons unrelated to the change under test. It takes ~28s, so it runs under --quick too. Verified falsifiable: a 0.0001 metric change and a changed final SoC both fail it, while multiplying every baseline runtime by ten does not. compare_results gains a per-scenario time_diff column and a suite total, both as percentages as well as seconds. Absolute seconds only mean something against the machine that produced them; the ratio survives a comparison between machines. The load scalings for the three simulated futures are now randomised over 0.2-2.0, sorted so load_scaling90 <= load_scaling <= load_scaling10 - the order the planner requires, PV90 being the sunny light-load future and PV10 the cloudy heavy one. The template pinned load_scaling 0.5 and load_scaling10 0.6 and left load_scaling90 at its 0.7 default, which the planner detects as inverted and clamps back to 0.5. Every scenario therefore ran PV90 with exactly the central case's load, and the warning fired on all twenty. Both clamp warnings are now gone, and the mean gap between cost_pv90 and cost widens from 105.84 to 349.32 - the pv90 column was previously separated from nominal only by the PV forecast, never by load. Drawn from their own rng stream, as the car and feature blocks are, so adding them leaves every pre-existing scenario parameter bit-identical. Baseline regenerated. 217 tests pass, pre-commit clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): reset the iboost carry-over between scenarios, correct a step count Both review comments on this PR. calculate_plan writes iboost_next back onto the instance and the scenario runner never calls fetch_config_options, which is what resets it every cycle in the product. A scenario therefore inherited the previous one's iboost carry-over, which is the same order dependence this PR already fixed for the rest of the iboost block - just incompletely. The reset now mirrors fetch_config_options: iboost_next, the three running flags and iboost_energy_today. No scenario's plan moves. Verified by running the full suite with and without the reset and comparing all twenty metrics: zero differ, so this closes a real hazard rather than a live defect, and the baseline is unchanged. The static cache fixture docstring said iboost_plan_load would be 576 zeroes. The fixture sets forecast_minutes to 24 hours and the kernel steps at 5 minutes, so the arrays are 288 long - the number the comment exists to make concrete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

springfall2008 and others added 4 commits August 16, 2026 17:26

Copilot AI lite review requested due to automatic review settings August 16, 2026 17:48

Copilot started reviewing on behalf of springfall2008 August 16, 2026 17:48 View session

Copilot AI reviewed Aug 16, 2026

View reviewed changes

springfall2008 merged commit ea44f61 into main Aug 16, 2026
3 checks passed

springfall2008 deleted the perf/export-selection-memo branch August 16, 2026 17:52

This was referenced Aug 16, 2026

perf: cache window bounds and replace deepcopy on window lists (-21% planning) #4536

Merged

test: vary iboost and low power export across the random scenarios #4550

Merged

Navigation Menu

Uh oh!

perf: cut ~23% off planning by removing redundant scans and copies - #4549

perf: cut ~23% off planning by removing redundant scans and copies#4549
springfall2008 merged 4 commits into
mainfrom
perf/export-selection-memo

springfall2008 commented Aug 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

springfall2008 commented Aug 16, 2026

The measurement

What changed

1. Scan window candidates once per price threshold

2. Only copy minute_data's history when the glitch filter can write to it

3. Share the load-independent kernel context across the marginal cost matrix

4. Stop deep-copying the export window list in optimise_export

Tests

Also fixed

Relationship to #4536

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

2. Only copy `minute_data`'s history when the glitch filter can write to it

4. Stop deep-copying the export window list in `optimise_export`