Skip to content

test: vary iboost and low power export across the random scenarios - #4550

Merged
springfall2008 merged 3 commits into
mainfrom
test/widen-random-scenario-coverage
Aug 16, 2026
Merged

test: vary iboost and low power export across the random scenarios#4550
springfall2008 merged 3 commits into
mainfrom
test/widen-random-scenario-coverage

Conversation

@springfall2008

Copy link
Copy Markdown
Owner

The 20-scenario byte-identical benchmark is the gate every planner change is held to, but three feature flags were pinned by the template for all 20 runs:

flag before now
set_charge_low_power always on always on (already covered)
set_export_low_power always off 10 on / 10 off
iboost_enable always off 9 on / 11 off

Anything guarded by the two disabled flags could be changed — or broken — without the gate noticing.

This was not theoretical

The iboost arm of export_window_allowed, added in #4549, could not be reached by any scenario. It needs iboost_enable, a non-empty iboost_plan and iboost_on_export off. It is now taken 126 times across the suite.

kernel_parity does randomise iboost, but it tests prediction parity and never runs the optimiser, so nothing in the repo reached that branch.

Keeping the regenerated baseline reviewable

cases/random_results.json has to be regenerated, so the change is only auditable if it can be attributed. Two things make that possible:

The new draws come from an rng salted off the scenario seed, not the main stream — the same device the car block uses, and for the same documented reason. Verified: every pre-existing parameter and stored profile is bit-identical across all 20 scenarios, and with the flags present but unapplied the plans still match the old baseline exactly.

The whole iboost block is written on every scenario, not just the enabled ones. Setting it only when enabled left the previous scenario's values behind, so a plan depended on what ran before it — the same trap run_debug_cases documents. That moved all 20 scenarios; with the off-case values taken from the template it moves only the 13 the flags actually touch:

scenarios with a new flag on : 10 changed
scenarios with both flags off: 7 unchanged, 0 unexpectedly changed

Those 7 untouched scenarios being byte-identical is what makes the rest of the diff meaningful.

iboost_plan is built here on the same condition fetch_sensor_data uses, because the scenario runner never calls fetch and an empty plan cannot reach the optimiser path this exists to cover.

Static context cache fixture

test_kernel_static_cache ran with iboost off and no battery temperature curves, so iboost_plan_load was 288 zeroes and every temperature produced an identical cap. Both are arrays the cache reuses, so its equivalence tests would have agreed whatever it did with them:

before now
iboost_plan_load non-zero steps 0 of 288 36 of 288
distinct temp_charge_cap values 1 3

Verification

  • 217 tests pass, pre-commit clean
  • Scenario parameter drift against the previous file: 0
  • New flags present but unapplied reproduces the old baseline exactly

🤖 Generated with Claude Code

The 20-scenario byte-identical benchmark is the gate every planner change is held
to, but three feature flags were pinned by the template for all 20 runs: low power
charge always on, iboost and low power export always off. Anything guarded by the
two disabled ones could be changed - or broken - without the gate noticing.

That was not theoretical. The iboost arm of export_window_allowed, added in #4549,
could not be reached by any scenario: it needs iboost_enable, a non-empty
iboost_plan and iboost_on_export off. It is now taken 126 times across the suite.
kernel_parity did cover iboost for prediction, but it never runs the optimiser, so
nothing reached that branch.

iboost_enable is drawn at 40% (matching kernel_parity) and set_export_low_power
50/50, from an rng salted off the scenario seed rather than the main stream - the
same device the car block uses, and for the same reason: every pre-existing
parameter and stored profile is bit-identical, so a plan that moves has moved
because of the new flags and nothing else. Verified: 0 drift across all 20
scenarios, and with the flags present but unapplied the plans still match the old
baseline exactly.

The whole iboost block is written on every scenario rather than only the enabled
ones. Setting it only when enabled left the previous scenario's values behind, so
a plan depended on what ran before it - the same trap run_debug_cases documents,
and it moved all 20 scenarios instead of the 13 the flags actually touch. With the
off-case values taken from the template, the 7 scenarios neither flag touches are
byte-identical to before, which is what makes the regenerated baseline reviewable.

iboost_plan is built here on the same condition fetch_sensor_data uses, because
the scenario runner never calls fetch and an empty plan cannot reach the optimiser
path this exists to cover.

Also fixes the static context cache fixture, which ran with iboost off and no
battery temperature curves - so iboost_plan_load was 288 zeroes and every
temperature produced the identical cap. Both are what the cache reuses, so the
equivalence tests would have agreed whatever it did with them. Now 36 of 288
iboost steps are non-zero and the caps take 3 distinct values.

Baseline regenerated: 13 scenarios move, 7 are unchanged, 217 tests pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 16, 2026 18:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR strengthens Predbat’s 20-scenario “byte-identical” planning benchmark by varying previously pinned feature flags (set_export_low_power and iboost_enable) so optimizer code paths guarded by those flags are exercised, and updates the associated recorded baselines/fixtures.

Changes:

  • Add per-scenario features blocks to the random scenario suite to toggle set_export_low_power and iBoost settings deterministically.
  • Update the random scenario generator/applier to draw/apply these feature flags from a separate RNG stream (seed-salted).
  • Improve the static kernel context cache fixture to include non-trivial iBoost plan load and temperature-cap variation; regenerate random_results.json.

Reviewed changes

Copilot reviewed 2 out of 4 changed files in this pull request and generated 2 comments.

File Description
coverage/cases/random_scenarios.yaml Adds features sections to each scenario to vary iBoost/export-low-power behavior across the suite.
coverage/cases/random_results.json Updates the expected per-scenario metrics/costs/runtime baselines after enabling the new feature variability.
apps/predbat/tests/test_random_scenarios.py Generates/applies new per-scenario feature flags using a salted RNG stream; builds an iBoost plan when needed for optimizer coverage.
apps/predbat/tests/test_kernel_static_cache.py Expands fixture environment to vary temperature caps and include a real iBoost plan so static-cache equivalence checks are meaningful.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread apps/predbat/tests/test_random_scenarios.py
Comment thread apps/predbat/tests/test_kernel_static_cache.py
springfall2008 and others added 2 commits August 16, 2026 19:56
…calings

Three changes to the random scenario benchmark.

The 20-scenario plan comparison now runs as part of run_all rather than only by
hand. It fails if any recorded field differs from the committed baseline, and
reports runtime without asserting on it - the suite runs on machines of wildly
different speeds, so a runtime threshold would either be so loose it caught
nothing or so tight it failed for reasons unrelated to the change under test. It
takes ~28s, so it runs under --quick too. Verified falsifiable: a 0.0001 metric
change and a changed final SoC both fail it, while multiplying every baseline
runtime by ten does not.

compare_results gains a per-scenario time_diff column and a suite total, both as
percentages as well as seconds. Absolute seconds only mean something against the
machine that produced them; the ratio survives a comparison between machines.

The load scalings for the three simulated futures are now randomised over
0.2-2.0, sorted so load_scaling90 <= load_scaling <= load_scaling10 - the order
the planner requires, PV90 being the sunny light-load future and PV10 the cloudy
heavy one. The template pinned load_scaling 0.5 and load_scaling10 0.6 and left
load_scaling90 at its 0.7 default, which the planner detects as inverted and
clamps back to 0.5. Every scenario therefore ran PV90 with exactly the central
case's load, and the warning fired on all twenty. Both clamp warnings are now
gone, and the mean gap between cost_pv90 and cost widens from 105.84 to 349.32 -
the pv90 column was previously separated from nominal only by the PV forecast,
never by load.

Drawn from their own rng stream, as the car and feature blocks are, so adding
them leaves every pre-existing scenario parameter bit-identical.

Baseline regenerated. 217 tests pass, pre-commit clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tep count

Both review comments on this PR.

calculate_plan writes iboost_next back onto the instance and the scenario runner
never calls fetch_config_options, which is what resets it every cycle in the
product. A scenario therefore inherited the previous one's iboost carry-over,
which is the same order dependence this PR already fixed for the rest of the
iboost block - just incompletely. The reset now mirrors fetch_config_options:
iboost_next, the three running flags and iboost_energy_today.

No scenario's plan moves. Verified by running the full suite with and without the
reset and comparing all twenty metrics: zero differ, so this closes a real hazard
rather than a live defect, and the baseline is unchanged.

The static cache fixture docstring said iboost_plan_load would be 576 zeroes. The
fixture sets forecast_minutes to 24 hours and the kernel steps at 5 minutes, so
the arrays are 288 long - the number the comment exists to make concrete.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@springfall2008

Copy link
Copy Markdown
Owner Author

Both comments were right and are fixed in 136b937.

1. iBoost carry-over between scenarios — real, and my own change was incomplete. calculate_plan writes iboost_next back onto the instance (plan.py:4250) and the scenario runner never calls fetch_config_options, which is what resets it every cycle in the product. This is the same order dependence the PR already fixes for the rest of the iBoost block — I just didn't carry it far enough. The reset now mirrors fetch_config_options: iboost_next, the three running flags and iboost_energy_today.

It turns out to be latent rather than live. I ran the full suite with and without the reset and compared all twenty metrics:

scenarios whose metric differs with vs without the iboost carry-over reset: 0

So it closes a real hazard rather than a current defect, and the baseline is unchanged — the regression test still reports All 20 scenarios match the baseline across 320 compared fields.

2. 576 zeroes should be 288 — correct. The fixture sets forecast_minutes to 24 hours and the kernel steps at 5 minutes, so the arrays are 288 long. The number is the whole point of that sentence, so having it wrong defeated it. Fixed.

Since the last review this PR also gained two things you may want to look at:

  • the plan regression now runs as part of run_all (and --quick, at ~28s), failing on any recorded field changing while reporting runtime without asserting on it. Verified falsifiable both ways: a 0.0001 metric change fails it, multiplying every baseline runtime by ten does not.
  • the load scalings for the three futures are randomised over 0.2-2.0, sorted load_scaling90 <= load_scaling <= load_scaling10. The template left load_scaling90 at its 0.7 default against load_scaling 0.5, which the planner clamps — so PV90 carried the central case's load in every scenario. Mean gap between cost_pv90 and cost widens from 105.84 to 349.32.

Full suite of 217 tests and pre-commit green.

🤖 Generated with Claude Code

@springfall2008
springfall2008 merged commit 9be8f4d into main Aug 16, 2026
2 checks passed
@springfall2008
springfall2008 deleted the test/widen-random-scenario-coverage branch August 16, 2026 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants