fix(octopus): don't rely on future IOG dispatch slots the car no longer needs

[chalfontchubby]

Fixes #4482.

Summary

• Octopus grants daytime Intelligent Go dispatch slots based on its own assumption of what the car needs, since it can't see the car's real SoC - only Predbat can, via car_charging_soc/car_charging_limit. load_octopus_slots() already caps car_charging_slots[car_n] at the car's real remaining requirement when octopus_intelligent_consider_full is on, zeroing the kwh of any slot beyond that - but rate_add_io_slots() (which decides which dispatch slots the house battery can rely on as a cheap rate) never looked at that, so it kept treating every granted slot as low rate regardless of whether the car's own plan still needed it.
• If Predbat had already deferred house battery charging into one of those "surplus" slots, the slot could disappear once Octopus notices the car has stopped drawing - leaving the battery undercharged with no cheap window left to make it up.
• Adds octopus_intelligent_limit_future_slots (expert mode, default Off, requires octopus_intelligent_consider_full to have any effect - warns at startup otherwise): a future out-of-window dispatch slot only counts as low rate for the house battery while it falls within a 30-min block car_charging_slots still lists a positive kwh for. A slot already underway or completed is trusted regardless (only future need is gated), and the fixed 23:30-05:30 window is never affected, since it's guaranteed cheap by the tariff itself.

Test plan

• 6 new rate_add_io_slots scenarios: partial-need-vs-full settlement period, car already full, current/completed dispatch exempted, switch off restores old behaviour, fixed window unaffected
• fetch_config_options warning test for the consider_full dependency
• ./run_all --quick
• ./run_pre_commit

[springfall2008]

Can you review this against: #4237

[chalfontchubby]

Checked against #4237 - no overlap or conflict, they're on different pathways:

• feat: strengthen Octopus Intelligent earlier-charge skew #4237 operates on self.io_adjusted, which is populated by fetch_octopus_rates() from Octopus's own rate-API is_intelligent_adjusted attribute across the whole import/export rate curve (octopus.py:2825, unrelated to any specific car). It's consumed downstream in plan.py's sort_window_by_price_combined() as a soft pence-level pricing gradient across a contiguous run of already-built charge/export windows.
• This PR (fix(octopus): don't rely on future IOG dispatch slots the car no longer needs #4483) operates on rate_add_io_slots(), which works from the explicit per-car dispatch slot list (octopus_slots/car_charging_slots) and decides, upstream, whether a future slot's low assumed_price gets written into rates[] at all. Nothing in it touches self.io_adjusted.

No file overlap either - #4237 only touches plan.py + its own new test module; this PR touches config.py/fetch.py/octopus.py/predbat.py + existing tests. Safe to merge in either order.

Functionally they're complementary rather than duplicative: this PR answers "should a future out-of-window slot be relied on for the house battery at all" (hard exclusion, gated on the car's real remaining SoC need), #4237 answers "given the slots that are being relied on, which within a run should be preferred first" (soft risk-decay ranking). A slot this PR excludes (needed=False) just carries its normal, non-discounted rate into #4237's sort stage, so #4237's few-pence gradient has negligible effect on it either way - it isn't priced as cheap in the first place.

One thing I can't verify from code alone: whether Octopus's is_intelligent_adjusted rate-curve flag (#4237's input) always lines up 1:1 with the specific per-car dispatch slots this PR reads (octopus_slots), or whether they can diverge in practice. Not something either PR depends on for correctness, but worth knowing if there's a real-world case where they disagree.

[Speshman]

One possible interaction with #4237/the Octopus rate input path may need testing. When metric_octopus_import is used, fetch_octopus_rates() can receive a dispatch period whose value_inc_vat is already the Intelligent low rate and whose is_intelligent_adjusted flag is true. By the time rate_add_io_slots() runs, rates[minute] may therefore already be 3.99p.

In that case, needed=False only prevents another low-rate assignment; it does not restore the already-adjusted rate to the normal out-of-window tariff. The block may consequently remain available to the optimiser.

I observed this shape in the live Octopus integration:

value_inc_vat: 0.03993
is_intelligent_adjusted: true

compared with the normal peak rate of 0.293222.

Could a test be added where the input rate is already low and io_adjusted is true, asserting that an unneeded future out-of-window block returns to the firm tariff? The implementation may also need to clear or ignore the corresponding io_adjusted marker so #4237 does not continue treating the rejected block as an Intelligent window.

The existing tests initialise all rates at 10p and rely on rate_add_io_slots() to reduce accepted blocks to 4p, so they do not currently cover this input path.

[chalfontchubby]

Excellent catch, thank you - traced it through and you're right.

fetch.py:978 runs rate_replicate(import_rates, self.io_adjusted, ...) before rate_add_io_slots() ever sees the data (fetch.py:981), and rate_replicate() only gap-fills minutes that are missing - it doesn't touch a minute that already has a real fetched value. So for a genuine Octopus Intelligent tariff, value_inc_vat can arrive already at the dispatch-discounted rate (your 3.99p example) for a period flagged is_intelligent_adjusted, before rate_add_io_slots() runs at all.

needed=False in this PR only skips the write (rates[minute] = assumed_price) - it was never built to undo an already-low rate, because on main today rate_add_io_slots() has no "reject" concept at all: it only ever adds a low rate for a slot it decides is valid, it never had to make one not low. This PR is the first thing that introduces that need, and I only tested it against fixtures that start every rate flat at 10p (so rate_add_io_slots() itself is always the one making a block cheap) - exactly the gap you spotted, not something the existing tests could have caught.

Proposed fix: when needed=False and the incoming rate at that slot is already below the normal tariff, restore it to self.rate_max_base (fetch.py:975 - the day's true peak rate, captured before any IO/saving-session/override distortion) rather than leaving it untouched. Also clear self.io_adjusted[slot_start] for the rejected block at the same time, so a future consumer of that flag (e.g. #4237's pricing gradient, once merged) doesn't keep treating a now-firm-tariff block as an Intelligent window.

rate_max_base is the closest existing signal I can find, but it's an approximation - the day's overall max, not necessarily that specific half-hour's true undiscounted rate. Since you've clearly got real API responses to hand: is there a better signal I should be restoring to instead (e.g. a standard/non-dispatch rate reported alongside the adjusted one), or is rate_max_base close enough in practice for how Octopus structures IOG tariffs?

Will add a test with rates pre-seeded low (mirroring your example) + io_adjusted=True, asserting an unneeded future block returns to the firm tariff, once the restore target is confirmed.

[Speshman]

I now have a live example confirming this input path and answering the restoration-target question.

For tonight, Octopus currently advertises these dispatches:

• 23:00-23:30: 1.753 kWh
• 00:30-04:30: 14.024 kWh
• 05:30-06:00: 0.5965 kWh

The car's actual SoC is 2%. With octopus_intelligent_consider_full enabled, Predbat forecasts it reaching 100% at approximately 03:15. Accordingly, Predbat assigns zero EV demand to the future 05:30-06:00 dispatch.

However, the Octopus rate event already supplies that surplus block as:

start: 2026-08-13T05:30:00+01:00
end: 2026-08-13T06:00:00+01:00
value_inc_vat: 0.03993
is_intelligent_adjusted: true
Predbat therefore still publishes 05:30-06:00 at 3.99p despite forecasting no EV demand in it. This demonstrates that needed=False must actively undo the adjusted input rate rather than merely skip writing another low rate.
Regarding the restoration value: the Octopus event does not provide an undiscounted value alongside an adjusted row. On this fixed IOG tariff, every ordinary out-of-window row is 0.293222 GBP/kWh and the event's max_rate is also 0.293222. Therefore, rate_max_base appears to be the appropriate conservative restoration target here.
One implementation detail: minute_data() populates self.io_adjusted for every minute covered by the adjusted period, not just slot_start (utils.py:598-603). The rejection path should therefore restore and clear every minute in the rejected block, for example while iterating it:
rates[minute] = self.rate_max_base
self.io_adjusted.pop(minute, None)
Clearing only self.io_adjusted[slot_start] would leave the other 29 minutes marked as Intelligent-adjusted.
This installation is currently running unpatched code, so this is evidence of the real input shape and current failure mode rather than a test of the revised PR implementation. I can also report whether Octopus later withdraws the surplus 05:30-06:00 dispatch after the car finishes charging.

[chalfontchubby]

Pushed the restoration fix discussed above.

A rejected slot (not needed, or the daily cap already reached) now restores rates[minute] to rate_max_base for every minute of the block, rather than just skipping the addition of a new low rate - previously it left whatever fetch_octopus_rates() had already put there untouched, which for a genuine Intelligent tariff can already be the dispatch-discounted rate. Also clears self.io_adjusted across the same range, not just slot_start - minute_data() sets it for every minute in an adjusted block, so a partial clear would leave the plan believing an already-restored minute was still IOG-adjusted.

New regression test simulates the exact scenario (rejected slot with rates[] pre-seeded as fetch_octopus_rates() would leave it) - confirmed it fails without this commit and passes with it. All 23 existing scenarios pass unchanged.

Full suite + pre-commit clean.

[Speshman]

Thanks, this now addresses the live failure mode I reported: restoring every minute to rate_max_base and clearing every corresponding io_adjusted marker is exactly what the pre-adjusted 3.99p input requires.

I noticed one possible side effect in the revised rejection path. The same destructive restoration also runs when the slot is rejected only because octopus_slot_max has been reached:

if needed and slots_per_day[day_offset] < octopus_slot_max:
...
else:
rates[minute] = self.rate_max_base
self.io_adjusted.pop(minute, None)

That changes the previous cap behavior even when octopus_intelligent_limit_future_slots is Off. It could also overwrite a legitimate tariff rate, including a guaranteed fixed-window rate, despite the PR intending 23:30–05:30 to remain unaffected.

Would it be safer to perform the active restore and io_adjusted clear only when needed is false? A slot rejected solely by the existing daily cap could retain the previous behavior unless its undiscounted source rate is known.

A regression test with the feature switch Off and octopus_slot_max already reached would confirm whether this behavior is intentional. A capped slot within 23:30–05:30 may also be worth covering.

I very much appreciate your work on this.

[chalfontchubby]

Pushed 3b1feda addressing both points.

The restore scope: agreed, and confirmed the exact mechanism you flagged - needed stays True even with octopus_intelligent_limit_future_slots Off, since the daily octopus_slot_max cap is a pre-existing, unrelated mechanism. The destructive restore now only fires when needed is False; a slot rejected purely because the cap is already spent is left exactly as fetch_octopus_rates() set it, same as before this PR. Added the two tests you asked for - cap-only rejection with the feature Off, and one inside the fixed 23:30-05:30 window - both fail on the pre-fix code, pass now.

Second thing, flagged separately by Rik while reviewing this: a zero-kWh dispatch entry (a plug-independent SMART grid-flex event Octopus schedules that delivers no energy to the car) was still consuming a slot of the 12-slot daily budget under the old counting, which could crowd out real charging dispatches later in the day even on a day the car never got close to 12 real slots. Added octopus_slot_count_zero_kwh (default Off) so these are now exempt from both the cap and the needed check entirely - a real tariff discount, but not something the car-dispatch cap should be modelling in the first place. On (legacy) restores the old counting.

Thanks for reading the actual diff rather than just eyeballing the PR description - the cap-rejection path especially wasn't something we'd have caught without it.

[Speshman]

If you need anything else from me please just ask. I'm still quite new to all this so on a vertical learning curve 😁

[chalfontchubby]

Thanks @Speshman - I'm pretty new here myself. Subject to review from @springfall2008 now I think