fix(load-ml): stop a training run that crosses midnight inflating the…

[mbuhansen]

… forecast

The load baseline (load_minutes_now, cumulative load since local midnight) is captured during the data fetch, while minutes_now is read live from the base at publish time. Training runs for many minutes, so when a fine-tune starts before midnight and finishes after it the two disagree: minutes_now has reset to the new day but the baseline still holds yesterday's full daily total, which then gets added on top of every published forecast point.

Seen in the wild as a single-sample spike in the ML load chart - a fetch at 23:50 followed by training that completed at 00:15 published load_today_h1 of ~25 kWh and load_today_h8 of ~30 kWh against an actual daily total of 25 kWh. The next cycle re-fetched and the values returned to normal.

Re-fetch after training whenever the data has gone stale, which re-anchors both the baseline and the lookback window feeding the prediction to the current time. As a second line of defence, detect a baseline snapshot belonging to a previous day at publish time and re-derive it from the per-step load history instead.

PR note: ML load forecast spikes when a training run crosses midnight

Working note for a later PR. Branch: fix/load-ml-midnight-stale-baseline, commit e58fd256
(rebased onto main at v8.48.3). Not intended to be committed — delete before opening the PR.

Symptom

Occasional single-sample spikes in the LoadML chart, where the Forecast (+1h) and
Forecast (+8h) series jump to a value above the household's entire daily consumption
and then return to normal on the next cycle.

Reported by a user on a ~25 kWh/day install: Forecast (+1h) spiked to ~26 kWh just
after midnight and Forecast (+8h) to ~30 kWh, against an actual daily total of 25 kWh.
The Load (Actual) series was unaffected apart from the same single sample.

Because the chart series carry a +1h / +8h plotting offset
(web.py, prune_today(..., offset_minutes=...)), the spikes appear on the chart one and
eight hours after the cycle that produced them, which makes them look unrelated to
midnight at first glance.

Root cause

load_ml_component.py mixes two different notions of "now" when publishing:

• self.load_minutes_now — cumulative load since local midnight, a snapshot taken during
  the data fetch (_fetch_load_data → _merge_fetch_data).
• self.minutes_now — read live from the base object via the ComponentBase property.

_publish_entity() uses both together:

if minute > 0 and ((minute + self.minutes_now) % (24 * 60) == 0):
    reset_amount = value + self.load_minutes_now
output_value = round(value - reset_amount + self.load_minutes_now, 4)

Normally the fetch and the publish are seconds apart and the two agree. But training runs
between them, and a fine-tune on a large history takes tens of minutes. When a training run
starts before local midnight and finishes after it:

• self.minutes_now has reset to the new day, so the midnight-reset branch does not fire
  until minute 1440 - minutes_now, which is now late in the forecast.
• self.load_minutes_now still holds the previous day's full total.

Every forecast point before that reset therefore gets a whole day of load added to it.
load_today_h1 (minute 60) and load_today_h8 (minute 480) are both in that range, so both
published stats are inflated by roughly one day of consumption.

Secondary effect, present regardless of midnight: the prediction itself is generated from the
pre-training lookback window, so after a 25-minute training run the model is fed data that is
25 minutes out of date.

Evidence

From the reporter's log (local time, Europe/Copenhagen):

2026-08-07 23:50:35  ML Component: Fetching 28 days of load history from [...]
2026-08-07 23:50:36  ML Component: Starting fine-tune training (2h interval), model age is 2.017 hours
2026-08-07 23:50:36  ML Component: Doing training...
2026-08-08 00:15:27  ML Predictor: Curriculum training complete, final val_mae=0.0059 kWh
2026-08-08 00:15:28  ML Component: Generated 576 predictions (total 48.68 kWh over 48h)
2026-08-08 00:15:28  ML Component: Prediction cycle completed        <- publishes with minutes_now = 15
2026-08-08 00:25:29  ML Component: Fetching 28 days of load history  <- next cycle, back to normal

The fetch at 23:50 captured load_minutes_now ≈ 25 kWh (minutes_now was 1430). The publish
at 00:15 used minutes_now = 15. Predicted values for that cycle were normal — the totals in
Generated 576 predictions are in the usual 43–48 kWh range throughout the log — so the error
is entirely in the published baseline, not in the model.

Scope

• Affects the sensor.<prefix>_load_ml_stats attributes and the results attribute of
  sensor.<prefix>_load_ml_forecast, i.e. the LoadML/LoadMLPower charts.
• With load_ml_source enabled, the bad cycle also feeds the planner via
  fetch_ml_load_forecast(), which reads the results attribute back. The reporter had
  load_ml_source off, so for them it was display-only.
• Every version from v8.47.3 through current main is affected identically. load_predictor.py
  is byte-identical across that range; load_ml_component.py changed once, at v8.48.0, where
  feat(plan): add a pv90 upside forecast scenario to balance the one-sided pv10 hedge #4462 (PV90) made fetch_pv_forecast() return three values — a single line at the call site,
  nowhere near the baseline handling.

Fix

Two layers:

1. Root cause. The fetch block in run() is factored out into _do_fetch(), and after
   _do_training() the data is re-fetched when it has gone stale (>= PREDICT_STEP minutes).
   This re-anchors both the baseline and the lookback window feeding the prediction to the
   current time, which also fixes the "predicting from 25-minute-old data" problem.

2. Defence in depth. load_minutes_now_time is stored alongside the baseline, and
   _load_baseline_now() detects a snapshot belonging to a previous local day and re-derives
   the value from the per-step load_data history instead, logging a warning.

New log lines to look for:

ML Component: Data is 25 minutes stale after training, re-fetching before prediction
Warn: ML Component: Load baseline of 24.9 kWh was captured on 2026-01-01 23:50 which is a
      previous day, re-derived load so far today as 0.15 kWh

Tests

New sub-test component_stale_midnight_baseline in tests/test_load_ml.py, covering:

• a stale pre-midnight baseline is re-derived from the load history;
• a same-day snapshot is used unchanged;
• a missing snapshot timestamp falls back to the previous behaviour;
• a full run() cycle where training moves the clock from 23:50 to 00:15 triggers a second
  fetch and publishes the corrected values.

Without the fix the test fails with
Expected load_today 0.15 re-derived since midnight, got 24.9.

Verification performed:

• --test load_ml: 29/29 pass (28/29 without the fix).
• pre-commit (ruff, black, cspell) on both changed files: clean.
• All other non-slow tests pass.

Field verification

The fix has run on a live install since 2026-08-08. The bug scenario occurred once in that
window, on the night of 12-13 August, and was handled correctly:

2026-08-12 23:40:42  ML Component: Doing training...
2026-08-13 00:06:09  ML Component: Training successful, val_mae=0.0065 kWh
2026-08-13 00:06:09  ML Component: Data is 25 minutes stale after training, re-fetching before prediction
2026-08-13 00:06:12  ML Component: Generated 576 predictions (total 43.25 kWh over 48h)

Training started 19 minutes before local midnight and finished 6 minutes after it. The
published forecast is continuous across the boundary — 41.52 kWh at 23:28, 43.25 kWh at the
crossing, 43.26 kWh at 00:24 — where before the fix the baseline would have carried the
previous day's ~25 kWh total into every point.

Over 11-15 August: 48 training runs, 1 midnight crossing, 244 prediction cycles all within
37.6-58.86 kWh. No inflation anywhere.

The layer-2 backstop (_load_baseline_now re-deriving the baseline) has never fired on the
live install — the re-fetch in layer 1 catches the condition first, which is the intended
order. It is covered by unit tests only.

Unrelated issues noticed while verifying

Worth separate PRs, not addressed here:

• test_fox_api.py::test_run_midnight_reset is flaky. The final assertion
  fox.start_time_today > initial_start_time races against clock resolution and fails on
  roughly 4 out of 5 runs on Windows. It aborts run_all --quick before the rest of the suite.
• ge_cloud and annual_load_octopus fail on Windows with
  RuntimeError: aiodns needs a SelectorEventLoop on Windows. Environment limitation,
  reproduces on a clean tree.

Open question, not part of this fix

The reporter's Forecast (+8h) series sits systematically below actual load through the day
(~20 kWh predicted vs ~25 kWh actual at 18:00), which is why they run with load_ml_source
disabled. This is not the midnight bug — the model's own diagnostics are strong
(ar_mae ≈ 0.003–0.006 kWh per 5-min chunk, bias near zero, teacher-forced drift ~0.0002 kWh).
A plausible candidate is the historical-pattern blending in load_predictor.py:1553-1560
(blend_floor = 0.5), which pulls the forecast toward the day-of-week mean as the horizon
grows. Needs its own investigation before anything is claimed.

[predbat (20).log](https://github.com/user-attachments/files/31106771/predbat.20.log)

[Copilot]

Pull request overview

Fixes intermittent ML load forecast spikes when a training run spans local midnight by ensuring the load baseline and lookback window are re-anchored after long training, and by adding a publish-time backstop for stale baselines.

Changes:

• Factor data fetching into _do_fetch() and re-fetch after training when fetched data is stale (>= PREDICT_STEP minutes).
• Track load_minutes_now_time and add _load_baseline_now() to re-derive the baseline from per-step history when the snapshot is stale.
• Add a new unit sub-test covering midnight-crossing stale baseline behavior and post-training re-fetch.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
apps/predbat/load_ml_component.py	Adds post-training re-fetch + baseline timestamping and a stale-baseline backstop during publish.
apps/predbat/tests/test_load_ml.py	Adds a new sub-test validating midnight-crossing baseline handling and re-fetch after training.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.