Skip to content

fix(history): keep Home use when at most 1 % of the day is unknown - #1756

Closed
frahlg wants to merge 1 commit into
1750-bridge-short-history-gapsfrom
1750-home-use-coverage
Closed

frahlg wants to merge 1 commit into
1750-bridge-short-history-gapsfrom
1750-home-use-coverage

Conversation

@frahlg

@frahlg frahlg commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Superseded. #1753 was rewritten after Fredrik's two-layer direction (17bdaf7). It now has its own day rule, where a flow becomes unknown only after more than 10 % of the range, and a one-time recount of cached source days (refreshSourceDayCache). Both parts of this PR are covered there, and the branch sits on the old #1753. Kept open only until Fredrik decides to close it.


Stacked on #1753 (base 1750-bridge-short-history-gaps), next to #1755. Relates to #1750.

Fredrik's choice in #1755: relax the day rule. Physics rules do nothing for a site without a map pin, and they cover one cause at a time.

Problem

Since #1741 (18 October), any unknown time makes the whole day's load_wh null: one unknown minute, a 3-minute restart, a battery that drops out for 3 minutes. Before #1741 a day summed what it had and never asked for full coverage, so nobody saw a gap. #1753 and #1755 handle two causes. Every other cause still nulls the day.

Change

  • A period keeps its Home use when at most 1 % of its time is unknown. For a day that is 14.4 minutes.
  • The total is the sum of the fully known source intervals, marked EstimatedLoad. Unknown time counts as nothing. Core already computed this number; only whether it is shown changes. The Home Assistant bridge, which reads the number whether it is known or not, sees no change.
  • Unknown time is the span from the first to the last record, or the writer periods if longer, minus the fully known source minutes. That covers minutes with an unknown source, writer gaps and stops.
  • A negative known total stays unknown. A day that mixes older history with source history keeps the strict rule.
  • Other flows keep their own unknowns. pv_wh, battery and grid totals stay null for their own gaps. history_quality may then read incomplete while load_wh has a number; each field already carries its own null.
  • Closed days are cached without a method version. A one-time history migration, home-use-unknown-share-v1, deletes cached days from the source cutover on, so days such as 18 and 19 October are recounted after the update. Days before source history are not touched.
  • fix(history): keep Home use through device dropouts and restarts #1753's bridge test: "meter replaced" now expects an estimate. Each meter measured its own time; only the 5 s between them is unknown.

No new setting, table, API field or UI.

Evidence

TestHomeUseSurvivesSmallUnknownShare: six hours of a 1000 W house at 10 s, one disturbance three hours in. 1 % of six hours is 3.6 minutes.

Case Before This PR
No disturbance 6000 Wh same
Battery unknown 2 min (3 minutes touched) incomplete, null estimated, 5950 Wh; battery totals stay unknown
Writer gap 3 min incomplete, null estimated, 5950 Wh
Writer gap 5 min incomplete, null unchanged
Battery unknown 10 min incomplete, null unchanged
Negative known total, 2 min gap incomplete, null unchanged

TestSourceDaysAreRecountedOnce: a cached source day is deleted on open, a day before source history stays, and a later cache entry survives the next open.

Each test fails when its part is switched off: no relaxation (both 5950 Wh cases stay null), no negative guard (the negative case becomes known), no recount (the cached day stays), recount every open (the second open fails).

70-day simulation: the soak used for #1753, #1754 and #1755, seed 42, 5 s ticks, Stockholm time. Each day has restarts, a night PV gap of 30–300 s, a day PV gap of 30–120 s, EV, meter and battery gaps; one day has a 12-minute outage. Days are 19–24 kWh.

Days with Home use Mean error on known days
#1753 + #1754 8 of 70 0.013 kWh
+ #1755 29 of 70 0.008 kWh
+ this PR 69 of 70 0.053 kWh
+ #1755 + this PR 69 of 70 0.039 kWh

The one null day has the 12-minute outage plus a battery gap, together over 1 %. The larger mean error is the unknown time counted as nothing, about 0.2 % of a day. #1755 still helps: night PV gaps then add no shortfall.

make verify: passed on this branch (on #1753). All 77 Go packages, protocol and script tests, vet and build. Nothing skipped. go test ./internal/state ./cmd/ftw also passes with #1753, #1754, #1755 and this PR merged together.

Not verified

  • Not run on a box. Nothing was changed on any box.
  • Unknown time counts as nothing, so a day at the 1 % limit reads up to about 1 % low.
  • 1 % is a choice, not derived from field data.

For Fredrik to decide

  1. Threshold. 1 % of the period. Higher shows more days, with a larger possible shortfall.
  2. Fill. Counting unknown time as nothing keeps the number Core already had, as before fix(history): derive household Load from interval source energy #1741. Filling it with the day's mean would be closer, but changes the number and can make a running Today total fall, which Home Assistant's energy statistics read as a meter reset.
  3. Other flows. The same rule could keep pv_wh and battery totals. Left out here: Repair historical Home use gaps from retained source measurements #1750 asks that rebuilding Home use must not make another flow known.
  4. Order. fix(history): keep Home use through device dropouts and restarts #1753 first. This and fix(history): count unknown PV as 0 W after sunset #1755 both sit on it and touch different files; they merge in either order. fix(history): repair Home use and Solar gaps with audited estimates #1751's subtotal would then appear only above 1 %.

🤖 Generated with Claude Code

Since #1741 any unknown time made the whole day's load_wh null. Before
source history a day summed what it had, and nobody missed a number.

A period now keeps its Home use, marked EstimatedLoad, when at most 1 %
of its time is unknown. The total is the sum of the fully known source
intervals; unknown time counts as nothing, so the number never changes,
only whether it is shown. A negative known total stays unknown. Other
flows keep their own unknowns.

A replaced meter no longer hides the day either: each meter measured its
own time and only the snapshot interval between them is unknown, so the
bridge test's meter-replaced case now expects an estimate.

Closed days are cached. A one-time history migration deletes cached
days from the source cutover on, so days closed under the stricter rule
are recounted.

Relates to #1750.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frahlg

frahlg commented Oct 20, 2026

Copy link
Copy Markdown
Member Author

Superseded by #1753. Its rewrite (17bdaf7, 85100d6) keeps a day's Home use unless a flow is unknown or unrecorded for more than 10 % of the range, and recounts cached source days once (refreshSourceDayCache). This PR's 1 % threshold and its simulation numbers (69 of 70 days) are listed as an alternative in #1753's decisions.

@frahlg frahlg closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant