fix(history): keep Home use through device dropouts and restarts by frahlg · Pull Request #1753 · srcfl/ftw

frahlg · 2026-10-19T15:44:02Z

Relates to #1750. This fixes the cause; #1751 shows a subtotal after the cause has struck. Reworked on 19 October after Fredrik's direction: Home use has a forgiving display layer and a strict training layer.

Problem

Since #1741 (first in v0.153.0-beta.1), one short disturbance makes the whole day's load_wh null:

A Core update or restart takes 18–20 s. The source writer held a snapshot for at most 15 s, and on open it deleted its cursor, so it never bridged a stop.
A driver that loses contact makes its source unknown. After about 60 s the watchdog takes it offline and its device leaves the topology, which also stopped integration (found in fix(history): count unknown PV as 0 W after sunset #1755).
aggregateDayEnergy ORs each interval's missing bits into the day and needs full coverage. One unknown interval makes the whole day unknown; an unknown PV also makes pv_wh null.

So every day with an update has unknown Home use for every beta tester. On Göran's box on 19 October, five 5-minute intervals failed: two Core updates (00:50, 13:32), SMA silent for 145 s at 02:22, and EV unknown around 12:38.

Two layers

Display is forgiving. Home use is the residual grid − PV − battery − EV, as well as the evidence allows.

An unknown PV, battery or EV source holds its last known value. There is no time limit. With fix(history): count unknown PV as 0 W after sunset #1755, PV after sunset is 0 W instead.
A held value never makes the household negative. If it would, the source changed while unknown (a charger that finished), so that interval leaves the held flows unknown.
The site meter holds its last value for at most 2 minutes. So does a snapshot across a writer pause or a Core restart.
A topology change no longer stops integration.
A day's flow is unknown only when it is unknown or unrecorded for more than 10 % of the range. Below that the total is what was recorded; unrecorded time counts as nothing.
Unrecorded time affects only Home use, as before. Import, export and PV stay as recorded unless their own source was unknown for over 10 %.

Training is strict (#1754). It uses only minutes without MissingPower, BalanceMismatch or EstimatedLoad.

Contract between them. Only two fully measured snapshots a safe tick apart, of one topology and one process, make a measured interval. Every other interval carries EstimatedLoad in history_quality. Readers never clear it, and retention now keeps it when it merges minutes into quarter hours (historyMissingAggregate had dropped bit 256).

Also:

historySources set MissingLoad for every telemetry reason except invalid_balance. Telemetry puts a source's own failure there too (offline:sma:pv, stale:easee:ev), so in a running Core a silent inverter voided the whole balance and the hold did nothing. Now only configuration and duplicate-flow reasons set MissingLoad (telemetry.SourceFailure). The state tests feed the writer only the source's own flag, so they did not see this; TestHistoryHoldsASilentSourceThroughTheBalance goes through buildHistoryPoint and fails without the fix (1010, PV and Load). Found by the fix(history): count unknown PV as 0 W after sunset #1755 session.
On open, Core marks the cursor as stopped instead of deleting it. A stop over 2 minutes starts a new period, as before. Older Cores still delete the cursor on open, so a rollback never bridges.
Closed source-era days cached under the old rule are recomputed once (source-day-coverage-v1). With the new day rule alone, rows the old writer already saved become known days again; see below.

No new setting, table, API field or UI. None of #1751's files are touched.

Evidence

TestSourceLoadDayForgivesShortGaps: 2 h of a 1000 W house, 5 s snapshots, one disturbance one hour in. The first version of this PR is 604ed96.

Case	master	first version	now
No disturbance	2000 Wh	2000 Wh	2000 Wh
20 s restart	null	2000	2000
20 s restart, EV and its device back 30 s later	null	2000	2000
EV unknown 15 s	null	2000	2000
EV starts at 2 kW while unknown 15 s	null	2008.3	2008.3, import exact
PV unknown 145 s	null	null	2000
EV unknown 30 min	null	null	2000
EV at 2 kW finishes 25 min into a 30 min gap	null	null	1900 (without the guard 1830.6)
Meter replaced (topology change)	null	null	2000
Meter unknown 5 min	null	null	1933.3; import 1948.6
3 min restart	null	null	1950
15 min restart	null	null	null; import 1750
Meter unknown 20 min	null	null	null; import null
Invalid configuration 30 min	null	null	null; import 2000

Every known day is estimated. In the "EV starts" case the held charger value puts 2 kW × 15 s (8.3 Wh) into the house. "Meter unknown 5 min" loses the four whole minutes the unknown 185 s touch: Home use counts by the minute.

Old writer, new day rule (rows as v0.153/v0.154 saved them): 20 s restart 1994.4 Wh, EV 15 s 1983.3, PV 145 s 1950, meter replaced 1998.6, meter 5 min 1900; EV 30 min stays null. So after updating, a day like 19 October should show Home use, a little low, once its cache is recomputed. Not proven on Göran's data.

14-day simulation (seed 42; daily restarts of 15–40 s, PV unknown 30–300 s, EV 10–20 s, battery 60–180 s, meter 20–90 s, a lagging meter, broken values and one 12-minute outage), run with #1754 (fe42b90):

	Days with Home use	Mean error per day	Training quarters
master	0 of 14	–	–
`17bdaf7` (hold, no guard)	14 of 14	0.019 kWh	87.3 %, no estimated labels
`2d1a434` (with guard)	14 of 14	0.019 kWh	87.3 %, no estimated labels
+ EV driver gone 3 h while an 11 kW charge ends, no guard	14 of 14	that day 5.38 vs 20.17 kWh
same, with guard	14 of 14	0.135 kWh; that day 18.53 vs 20.17 kWh

The remaining 8 % on that day is the house's own use during the ~80 minutes the guard leaves unknown. Only the charger's energy counter could correct it afterwards.

Other new tests:

TestSourceLoadMarksEveryEstimatedMinute: an EV gap of 15 s marks exactly one minute, and the marker survives the quarter-hour merge.
TestSourceDayCacheRecomputedOnce: a day cached under the old rule is dropped once, and a recomputed one stays.

Existing tests changed on purpose. In TestSourceLoadKeepsUnknownDistinctFromZero, "missing EV", "write gap" and "topology" are now estimates and must carry the marker; "partial tail" and "time reversal" stay unknown. TestSourceLoadUsesSnapshotTimeAndDoesNotBridgeRestart becomes …MarksRestartBridge: a 10 s stop is bridged with the marker, a stop over 2 minutes is not.

make verify: passed: all Go suites (internal/state included), protocol, scripts, the Docker release context, vet and build. Nothing skipped.

Not verified

Not run on a box. Nothing was changed on any box.
How real drivers report a lost connection varies. The tests cover the snapshots Core builds, not every driver.
fix(history): count unknown PV as 0 W after sunset #1755 is stacked on this branch and touches history_sources.go. This rework replaces its two writer lines (its marker rule is included here), so it needs a rebase.

For Fredrik to decide

Day threshold: 10 %. That is 2.4 h of a full day. Below it, unrecorded time counts as nothing, so a day with an hour-long outage reads about 4 % low but is labelled estimated, and coverage_ms already shows the gap. Draft fix(history): keep Home use when at most 1 % of the day is unknown #1756 proposes 1 % (14.4 minutes) for the same rule, with a 70-day simulation: 69 of 70 days known, mean error 0.039 kWh with fix(history): count unknown PV as 0 W after sunset #1755. With the hold in this PR, little unknown time is left, so either works for short gaps; they differ for outages between 15 minutes and 2.4 hours. One constant, maxSourceDayUnknownShare.
Holding a source without a time limit. Your direction. A charger that goes offline mid-charge and then finishes would keep its last power, which is taken off the house. The guard stops the hold as soon as that makes the house negative (separate commit, so it can be dropped). It cannot catch a smaller error that leaves the house positive, such as a charge that drops from 11 to 7 kW. The guard is also strict at 0 W, so timing noise in a held interval can mark it unknown. A counter on the returning source could correct the energy afterwards; not built.
Meter and restart budget: 2 minutes. It reuses LoadEstimateRecoveryWindow. A longer stop is unrecorded time, not a hold.
Other flows. Here PV, battery and grid totals follow the same 10 % rule on their own unknown time. fix(history): keep Home use when at most 1 % of the day is unknown #1756 keeps them strict, because Repair historical Home use gaps from retained source measurements #1750 asks that rebuilding Home use must not make another flow known. With the hold, a held PV or battery minute is estimated, not unknown, so it counts as known either way.
Counter evidence is not built. Göran's SMA counter itself reset and jumped in the 02:20 bucket, and the Pixii battery counter steps in whole kWh. The hold covers both cases now.

🤖 Generated with Claude Code

…me use Since #1741 a Core restart (18-20 s), a source unknown for a few seconds or a writer pause over 15 s made the whole day's load_wh null, and an unknown PV also nulled pv_wh. The source writer now holds an episode that starts at the last fully known snapshot. If every source is known again with the same topology within LoadEstimateRecoveryWindow (2 min), the episode is written as an estimate marked EstimatedLoad: measured sources keep their snapshots, unknown ones hold the anchor's values. Otherwise it is written exactly as before. Pending time stays out of totals and expected coverage, and the day waits for it before closing. On open Core marks the cursor as stopped instead of deleting it, so a brief stop is bridged as an estimate and a longer one starts a new period as before. Older Cores still delete the cursor on open. Relates to #1750. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Short source gaps may become estimated history once #1753 lands, and then a training label. Keep these tests about gaps that stay unknown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Follows Fredrik's two-layer direction (19 October). Display is forgiving: an unknown PV, battery or EV source holds its last known value; the site meter, a writer pause or a restart holds for at most two minutes; a topology change no longer stops integration. A day's flow is unknown only when it is unknown or unrecorded for more than a tenth of the range. Only two fully measured snapshots a safe tick apart, of one topology and one process, make a measured interval. Every other interval keeps EstimatedLoad, and quarter-hour retention now keeps that bit, so the training layer can skip it. Replaces the pending episodes of the first version. Closed source-era days cached under the old rule are recomputed once. Relates to #1750. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A held value that turns the household residual negative cannot be right: the source changed while unknown, such as a charger that finished. That interval leaves the held flows unknown instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Rebased on the rewritten #1753, whose writer marks an interval as an estimate at an unknown source, a stop, a long gap or a topology change. A night past civil dusk therefore trains only if every snapshot is fully known and the sleeping inverter keeps its device in the topology. The devices line moves into historyDevices so the test drives the same code as persistTelemetryTick. The test runs eight hours of a sleeping inverter through it, with and without a map pin. TestDarkPVByTimeOfDay now also checks that PV reads 0 W and that Load stays unknown for any cause but PV; it replaces the two-hour series test, whose day-level outcome the new writer decides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

historySources set MissingLoad for every reason except invalid_balance. Telemetry puts a source's own failure there too, such as "offline:sma:pv" or "stale:easee:ev", so a real snapshot with a silent inverter or charger voided the whole balance and the source writer held nothing. Only configuration and duplicate-flow reasons now set MissingLoad; a failed source is unknown on its own flag. Found by the #1755 session through historySources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Rebased on the rewritten #1753, whose writer marks an interval as an estimate at an unknown source, a stop, a long gap or a topology change. A night past civil dusk therefore trains only if every snapshot is fully known and the sleeping inverter keeps its device in the topology. The devices line moves into historyDevices so the test drives the same code as persistTelemetryTick. The test runs eight hours of a sleeping inverter through it, with and without a map pin. TestDarkPVByTimeOfDay now also checks that PV reads 0 W and that Load stays unknown for any cause but PV; it replaces the two-hour series test, whose day-level outcome the new writer decides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#1753 (85100d6) no longer marks Load unknown when a single source fails, so darkPV need not clear it. It now changes only the PV flow and its marker; a configuration fault keeps Load unknown on its own. The night test without a pin checks Home use again: the writer holds PV's last value as an estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(forecast): train load models on closed history intervals Core computed household load three ways: history balanced per-source energy per interval, the Energyplan tracker kept its own quarter-hour accumulator, and the built-in load model learned from one instant reading per minute. The models learned a different quantity from the Home use shown in History. Add state.HouseholdIntervals, which returns closed, valid intervals from the same source energy as daily totals. The built-in model trains once per closed quarter on its mean, and the tracker takes its load labels from the same quarters. PV keeps its accumulator for the curtailment flag and waits until history decides the quarter, so each immutable observation is written once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style: gofmt the tracker observation test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: use gaps longer than the history bridge window Short source gaps may become estimated history once #1753 lands, and then a training label. Keep these tests about gaps that stay unknown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(loadmodel): preserve stored minute-based heating bounds * fix(forecast): train only on fully measured household intervals History may show Home use across a bridged or estimated gap, but a load model must not learn from it. Leave out intervals with EstimatedLoad or BalanceMismatch as well as a missing source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

chatgpt-codex-connector · 2026-10-20T04:34:53Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-20T04:37:44.490627Z	`85100d6`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 85100d65aa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-20T04:37:41Z

-	if reading.Reason != "" && reading.Reason != "invalid_balance" {
+	// sources. Missing configuration and duplicate electrical flows do. A
+	// failed source is unknown on its own flag, so history may hold it.
+	if reading.Reason != "" && reading.Reason != "invalid_balance" && !telemetry.SourceFailure(reading.Reason) {


Preserve healthy peers when one source fails

When multiple devices share a DER type, a failure of one device makes ForecastReading.Known[kind] false for the entire type while the aggregate wattage still contains the healthy peers. Treating that reason as holdable then causes the loop below to zero the aggregate and holdSourceIntervalTx to reuse the last aggregate value. For example, if one of two PV inverters becomes stale while the other changes output, Home use is calculated using both inverters' old combined output, potentially indefinitely, rather than holding only the failed inverter and retaining the healthy one's current reading. The persisted representation must preserve per-source values or otherwise separate the failed contribution before enabling this bridge.

Useful? React with 👍 / 👎.

Rebased on the rewritten #1753, whose writer marks an interval as an estimate at an unknown source, a stop, a long gap or a topology change. A night past civil dusk therefore trains only if every snapshot is fully known and the sleeping inverter keeps its device in the topology. The devices line moves into historyDevices so the test drives the same code as persistTelemetryTick. The test runs eight hours of a sleeping inverter through it, with and without a map pin. TestDarkPVByTimeOfDay now also checks that PV reads 0 W and that Load stays unknown for any cause but PV; it replaces the two-hour series test, whose day-level outcome the new writer decides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#1753 (85100d6) no longer marks Load unknown when a single source fails, so darkPV need not clear it. It now changes only the PV flow and its marker; a configuration fault keeps Load unknown on its own. The night test without a pin checks Home use again: the writer holds PV's last value as an estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(history): count unknown PV as 0 W after sunset A PV inverter that sleeps at night is the most common remaining reason for a day without Home use. Göran's SMA was silent for 145 s at 02:22 on 19 October with 0 W on both sides, longer than the 2-minute bridge. While the sun is below the horizon at the site's map pin, Core now records an unknown PV source as 0 W, marked EstimatedLoad, never as a measurement. An inverter the watchdog took offline keeps its device in the history topology, so the night is not cut out. The source writer carries the marker into every interval an estimated snapshot bounds. Without a pin (unset, 0,0 or the untouched default) nothing changes. By day the gap stays unknown. Grid, battery and EV are not affected. Relates to #1750. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(history): count PV past civil dusk as measured 0 W Load models train only on fully measured intervals (#1754). Marking every dark PV snapshot as an estimate would drop each night with a sleeping inverter from training. Past civil dusk (zenith over 96°) no panel produces, so an unknown PV source now reads as a known 0 W without EstimatedLoad: the night counts in Home use and in training. Between sunset and civil dusk diffuse light remains, and the 0 W stays a marked estimate. No new bit records the physical zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(history): a full night with a sleeping inverter stays measured Rebased on the rewritten #1753, whose writer marks an interval as an estimate at an unknown source, a stop, a long gap or a topology change. A night past civil dusk therefore trains only if every snapshot is fully known and the sleeping inverter keeps its device in the topology. The devices line moves into historyDevices so the test drives the same code as persistTelemetryTick. The test runs eight hours of a sleeping inverter through it, with and without a map pin. TestDarkPVByTimeOfDay now also checks that PV reads 0 W and that Load stays unknown for any cause but PV; it replaces the two-hour series test, whose day-level outcome the new writer decides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(history): let the dark rule touch only PV #1753 (85100d6) no longer marks Load unknown when a single source fails, so darkPV need not clear it. It now changes only the PV flow and its marker; a configuration fault keeps Load unknown on its own. The night test without a pin checks Home use again: the writer holds PV's last value as an estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(telemetry): name offline drivers through a health-gated accessor historyDevices read cached PV readings directly to find a sleeping inverter. Move the health check into telemetry.OfflineDrivers, which returns only driver names, so no caller handles a stale cached value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…1758) The negative-balance guard cleared only the interval's copy of a held value; the cursor kept it. While the source stayed unknown, a later rise in house use made the same rejected charge valid again, so Home use lost it and a phantom charge was recorded. The rejected sources now stay unknown in the cursor until they report again. Found by Codex's adversarial review of #1753. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

frahlg mentioned this pull request Oct 19, 2026

fix(forecast): train load models on closed history intervals #1754

Merged

This was referenced Oct 19, 2026

fix(history): count unknown PV as 0 W after sunset #1755

Merged

fix(history): keep Home use when at most 1 % of the day is unknown #1756

Closed

frahlg changed the title ~~fix(history): bridge short source gaps instead of losing the day's Home use~~ fix(history): keep Home use through device dropouts and restarts Oct 19, 2026

segran2 mentioned this pull request Oct 19, 2026

fix(history): repair Home use and Solar gaps with audited estimates #1751

Open

frahlg marked this pull request as ready for review October 20, 2026 04:34

chatgpt-codex-connector Bot reviewed Oct 20, 2026

View reviewed changes

frahlg merged commit 4d4ce53 into master Oct 20, 2026
31 of 33 checks passed

frahlg mentioned this pull request Oct 20, 2026

fix(history): keep a rejected hold unknown until its source returns #1758

Merged

Navigation Menu

Repository navigation

fix(history): keep Home use through device dropouts and restarts - #1753

fix(history): keep Home use through device dropouts and restarts#1753
frahlg merged 4 commits into
masterfrom
1750-bridge-short-history-gaps

frahlg commented Oct 19, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

frahlg commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Two layers

Evidence

Not verified

For Fredrik to decide

Uh oh!

chatgpt-codex-connector Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

frahlg commented Oct 19, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 20, 2026 •

edited

Loading