test: vary iboost and low power export across the random scenarios - #4550
Conversation
The 20-scenario byte-identical benchmark is the gate every planner change is held to, but three feature flags were pinned by the template for all 20 runs: low power charge always on, iboost and low power export always off. Anything guarded by the two disabled ones could be changed - or broken - without the gate noticing. That was not theoretical. The iboost arm of export_window_allowed, added in #4549, could not be reached by any scenario: it needs iboost_enable, a non-empty iboost_plan and iboost_on_export off. It is now taken 126 times across the suite. kernel_parity did cover iboost for prediction, but it never runs the optimiser, so nothing reached that branch. iboost_enable is drawn at 40% (matching kernel_parity) and set_export_low_power 50/50, from an rng salted off the scenario seed rather than the main stream - the same device the car block uses, and for the same reason: every pre-existing parameter and stored profile is bit-identical, so a plan that moves has moved because of the new flags and nothing else. Verified: 0 drift across all 20 scenarios, and with the flags present but unapplied the plans still match the old baseline exactly. The whole iboost block is written on every scenario rather than only the enabled ones. Setting it only when enabled left the previous scenario's values behind, so a plan depended on what ran before it - the same trap run_debug_cases documents, and it moved all 20 scenarios instead of the 13 the flags actually touch. With the off-case values taken from the template, the 7 scenarios neither flag touches are byte-identical to before, which is what makes the regenerated baseline reviewable. iboost_plan is built here on the same condition fetch_sensor_data uses, because the scenario runner never calls fetch and an empty plan cannot reach the optimiser path this exists to cover. Also fixes the static context cache fixture, which ran with iboost off and no battery temperature curves - so iboost_plan_load was 288 zeroes and every temperature produced the identical cap. Both are what the cache reuses, so the equivalence tests would have agreed whatever it did with them. Now 36 of 288 iboost steps are non-zero and the caps take 3 distinct values. Baseline regenerated: 13 scenarios move, 7 are unchanged, 217 tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR strengthens Predbat’s 20-scenario “byte-identical” planning benchmark by varying previously pinned feature flags (set_export_low_power and iboost_enable) so optimizer code paths guarded by those flags are exercised, and updates the associated recorded baselines/fixtures.
Changes:
- Add per-scenario
featuresblocks to the random scenario suite to toggleset_export_low_powerand iBoost settings deterministically. - Update the random scenario generator/applier to draw/apply these feature flags from a separate RNG stream (seed-salted).
- Improve the static kernel context cache fixture to include non-trivial iBoost plan load and temperature-cap variation; regenerate
random_results.json.
Reviewed changes
Copilot reviewed 2 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
coverage/cases/random_scenarios.yaml |
Adds features sections to each scenario to vary iBoost/export-low-power behavior across the suite. |
coverage/cases/random_results.json |
Updates the expected per-scenario metrics/costs/runtime baselines after enabling the new feature variability. |
apps/predbat/tests/test_random_scenarios.py |
Generates/applies new per-scenario feature flags using a salted RNG stream; builds an iBoost plan when needed for optimizer coverage. |
apps/predbat/tests/test_kernel_static_cache.py |
Expands fixture environment to vary temperature caps and include a real iBoost plan so static-cache equivalence checks are meaningful. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…calings Three changes to the random scenario benchmark. The 20-scenario plan comparison now runs as part of run_all rather than only by hand. It fails if any recorded field differs from the committed baseline, and reports runtime without asserting on it - the suite runs on machines of wildly different speeds, so a runtime threshold would either be so loose it caught nothing or so tight it failed for reasons unrelated to the change under test. It takes ~28s, so it runs under --quick too. Verified falsifiable: a 0.0001 metric change and a changed final SoC both fail it, while multiplying every baseline runtime by ten does not. compare_results gains a per-scenario time_diff column and a suite total, both as percentages as well as seconds. Absolute seconds only mean something against the machine that produced them; the ratio survives a comparison between machines. The load scalings for the three simulated futures are now randomised over 0.2-2.0, sorted so load_scaling90 <= load_scaling <= load_scaling10 - the order the planner requires, PV90 being the sunny light-load future and PV10 the cloudy heavy one. The template pinned load_scaling 0.5 and load_scaling10 0.6 and left load_scaling90 at its 0.7 default, which the planner detects as inverted and clamps back to 0.5. Every scenario therefore ran PV90 with exactly the central case's load, and the warning fired on all twenty. Both clamp warnings are now gone, and the mean gap between cost_pv90 and cost widens from 105.84 to 349.32 - the pv90 column was previously separated from nominal only by the PV forecast, never by load. Drawn from their own rng stream, as the car and feature blocks are, so adding them leaves every pre-existing scenario parameter bit-identical. Baseline regenerated. 217 tests pass, pre-commit clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tep count Both review comments on this PR. calculate_plan writes iboost_next back onto the instance and the scenario runner never calls fetch_config_options, which is what resets it every cycle in the product. A scenario therefore inherited the previous one's iboost carry-over, which is the same order dependence this PR already fixed for the rest of the iboost block - just incompletely. The reset now mirrors fetch_config_options: iboost_next, the three running flags and iboost_energy_today. No scenario's plan moves. Verified by running the full suite with and without the reset and comparing all twenty metrics: zero differ, so this closes a real hazard rather than a live defect, and the baseline is unchanged. The static cache fixture docstring said iboost_plan_load would be 576 zeroes. The fixture sets forecast_minutes to 24 hours and the kernel steps at 5 minutes, so the arrays are 288 long - the number the comment exists to make concrete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Both comments were right and are fixed in 136b937. 1. iBoost carry-over between scenarios — real, and my own change was incomplete. It turns out to be latent rather than live. I ran the full suite with and without the reset and compared all twenty metrics: So it closes a real hazard rather than a current defect, and the baseline is unchanged — the regression test still reports 2. Since the last review this PR also gained two things you may want to look at:
Full suite of 217 tests and pre-commit green. 🤖 Generated with Claude Code |
The 20-scenario byte-identical benchmark is the gate every planner change is held to, but three feature flags were pinned by the template for all 20 runs:
set_charge_low_powerset_export_low_poweriboost_enableAnything guarded by the two disabled flags could be changed — or broken — without the gate noticing.
This was not theoretical
The iboost arm of
export_window_allowed, added in #4549, could not be reached by any scenario. It needsiboost_enable, a non-emptyiboost_planandiboost_on_exportoff. It is now taken 126 times across the suite.kernel_paritydoes randomise iboost, but it tests prediction parity and never runs the optimiser, so nothing in the repo reached that branch.Keeping the regenerated baseline reviewable
cases/random_results.jsonhas to be regenerated, so the change is only auditable if it can be attributed. Two things make that possible:The new draws come from an rng salted off the scenario seed, not the main stream — the same device the car block uses, and for the same documented reason. Verified: every pre-existing parameter and stored profile is bit-identical across all 20 scenarios, and with the flags present but unapplied the plans still match the old baseline exactly.
The whole iboost block is written on every scenario, not just the enabled ones. Setting it only when enabled left the previous scenario's values behind, so a plan depended on what ran before it — the same trap
run_debug_casesdocuments. That moved all 20 scenarios; with the off-case values taken from the template it moves only the 13 the flags actually touch:Those 7 untouched scenarios being byte-identical is what makes the rest of the diff meaningful.
iboost_planis built here on the same conditionfetch_sensor_datauses, because the scenario runner never calls fetch and an empty plan cannot reach the optimiser path this exists to cover.Static context cache fixture
test_kernel_static_cacheran with iboost off and no battery temperature curves, soiboost_plan_loadwas 288 zeroes and every temperature produced an identical cap. Both are arrays the cache reuses, so its equivalence tests would have agreed whatever it did with them:iboost_plan_loadnon-zero stepstemp_charge_capvaluesVerification
🤖 Generated with Claude Code