fix(history): repair Home use and Solar gaps with audited estimates

.changeset/home-use-subtotal.md

@@ -0,0 +1,11 @@
+---
+"ftw": patch
+---
+
+Keep an estimated Home use subtotal visible after missing source readings or a recording gap. Publish its accepted interval coverage separately from the unknown full total and measured energy. Preserve original history and exclude gaps rather than filling them.
+
+Add an offline historical repair tool that snapshots the original databases, prepares a hashed plan from selected retained AC power series, and applies audited estimates only to a working copy. Core uses these corrections in household daily totals and history without changing source measurements or their quality. Unsupported gaps remain unknown.
+
+Keep the native launcher and update API available when a local patch identifies the installed release with SemVer build metadata. Match the current slot and clean-exit marker using the base release while keeping published trial identity checks exact.
+
+Support audited endpoint-mean repairs for both historical Home use and Solar. Prepare while Core collects data, review the exact hash, and publish additive correction metadata atomically without replacing a live database. Show per-flow estimate labels and completed-through timestamps, preserving source observations and update support.

docs/history-endpoint-repair.md

@@ -0,0 +1,74 @@
+# Repair Home use and Solar from bounding power
+
+`ftw-history-endpoints` estimates only incomplete retained minutes. It uses
+`Wh = (before_W + after_W) / 2 × duration_hours`, keeping the same mean through
+a contiguous gap. Home use endpoints are the closest valid retained minute
+source-energy means before and after the gap. Solar endpoints are valid
+retained ten-second PV means, expressed as positive generation. Two observed
+night-time zeros give 0 Wh. Missing endpoints never imply zero.
+
+The endpoint distance is bounded explicitly (default 15 minutes, maximum 30).
+This is historical estimation, not recovered measurement. Changes between
+the two readings are unknown. A corrected minute replaces that flow's whole
+incomplete minute in the read view; its originally recorded part is not added
+a second time. Complete minutes and other flows keep their original evidence.
+
+Build Core and the helper from the same source. Native local Core builds may
+retain valid SemVer build metadata while using the installed base-release
+slot; keep the normal launcher so updates remain available. Prepare the build
+and transfer it before a short stopped-Core binary swap. Do not stop Core
+while copying or reviewing the database. Never start a second controlling Core.
+
+Take an independent, integrity-checked SQLite online backup of `history.db`
+while the live Core continues collecting. Use SQLite's backup API with a read
+snapshot, not a filesystem copy of an active WAL database. Keep matching state
+configuration and the prior binary for rollback. Leave disk headroom for the
+backup and WAL growth. Then prepare a completed minute range, up to 48 hours:
+
+```sh
+ftw-history-endpoints prepare --state /var/lib/ftw/state.db \
+  --plan ./endpoint-plan.json --since 2026-10-19T00:00:00+02:00 \
+  --until 2026-10-19T23:15:00+02:00 --max-gap 20m \
+  --reason 'Owner requested endpoint mean, issue 1750'
+```
+
+Inspect `minutes` (flow, whole-minute Wh, endpoint times and powers), `skipped`,
+source generation and evidence digest. The boundary timestamps are not raw
+polls: their retained resolution is part of the method. Choose the smallest
+bound that covers the reviewed gaps. An unsupported edge remains unknown.
+Apply the actual reviewed hash and verified independent backup:
+
+```sh
+ftw-history-endpoints apply --state /var/lib/ftw/state.db \
+  --plan ./endpoint-plan.json --sha256 ACTUAL_REVIEWED_SHA256 \
+  --backup /private/backup/history-before.db
+```
+
+Apply is safe alongside the live collector. It briefly reserves the SQLite
+writer, rechecks historical evidence, inserts one audited metadata record and
+invalidates derived daily caches in one transaction. It neither replaces a
+live database nor changes raw energy/power/quality rows, device configuration,
+control or forecasts. New unrelated measurements survive. An identical retry
+is idempotent; tampered, stale or conflicting plans fail without partial writes.
+
+Plans are stored in the existing `energy_ledger_meta` namespace
+`endpoint-repair-v1:<sha256>`, with backup hash and application time. This
+metadata already participates in portable backup/restore. Original minute
+energy in affected quarters is protected from rollup so replacements retain
+their boundaries. No new database schema or original quality flags are needed.
+Home corrections take read precedence over earlier source-power repairs;
+the earlier audit remains present.
+
+Daily totals, system ledger buckets and aggregate charts use the corrections.
+Solar has `pv_quality: estimated`; inferred Solar is excluded from the
+`measured` fallback. Home use remains `load_quality: estimated`. Current totals
+end at the latest committed complete minute, with independent `load_until_ms`
+and `pv_until_ms`; the unfinished writer tail is not extrapolated. Aggregate
+charts carry estimated/not-for-training markers. Raw samples remain unchanged.
+
+Verify both totals, flow quality, ledger provenance, writer progress and native
+update availability after application. If a client has cached a closed-day
+aggregate, reload it. To undo a plan, stop Core briefly or take a SQLite writer
+transaction, remove only its exact `energy_ledger_meta` key and invalidate the
+three derived day-cache tables. Keep later measurements; do not overwrite them
+with an older whole-database backup.

docs/history-repair.md

@@ -0,0 +1,110 @@
+# Repair historical household energy
+
+`ftw-history-repair` reconstructs missing household energy from retained,
+explicitly selected power series. It writes durable, audited corrections to
+`history_load_estimates` on a **separate working copy**, not the source
+databases. Core then reads the corrections in daily household totals, chart
+aggregates and the household energy ledger. Original samples, other energy
+flows, quality masks, forecasts and control remain unchanged.
+
+These are estimates, not exact recovery of lost measurements. A correction
+does not make a missing PV measurement known. The API marks household energy
+with `load_quality: estimated`, independently of other missing flows. It keeps
+derived energy out of `measured.load_wh` and self-powered percentages. A live
+day corrected this way reports `load_until_ms` at its last completed minute.
+
+## Prepare a copy and inspect its plan
+
+Build the tool from the same patched source as Core:
+
+```sh
+cd go
+go build -o ../bin/ftw-history-repair ./cmd/ftw-history-repair
+```
+
+For a deployment, stop Core using the installation's actual service manager
+before taking the snapshot, and keep it stopped until the corrected history
+is installed or the operation is abandoned. A copy prepared while Core keeps
+collecting data can be used for analysis, but must not later replace the live
+history: that would discard newer measurements. Keep a verified full FTW
+backup and the original release for deployment rollback. This tool additionally
+preserves checked copies of the affected state/history databases.
+
+Select a completed range, at most 48 hours, with explicit time zones and
+minute boundaries. Select every electrical flow or explicitly declare it
+absent. Use site-side **AC** battery power; do not combine cabinet DC energy
+with AC grid/PV readings. Check the running box's metric catalogue and physical
+topology before using these example driver/metric names:
+
+```sh
+bin/ftw-history-repair prepare \
+  --state /var/lib/ftw/state.db \
+  --workdir ./repair-2026-10-19 \
+  --since 2026-10-19T00:00:00+02:00 \
+  --until 2026-10-19T14:00:00+02:00 \
+  --grid pixii:meter_w --pv sma:pv_w \
+  --battery pixii:battery_ac_w --ev easee-cloud:ev_w --v2x none \
+  --max-gap 15s --reason 'Reconstruct missing Home use intervals, issue 1750'
+```
+
+The work directory must be new. `backup/` contains state/history snapshots;
+`corrected/` starts as a separate copy. `plan.json` records their hashes,
+selected sources, period, reason, per-minute Wh, interpolation evidence,
+unsupported minutes and the exact plan SHA-256. Preparing does not apply
+corrections. The originals are opened read-only; both copies are checked
+through Core's database integrity/generation checks.
+
+The reconstruction uses retained power means at minute resolution or finer.
+It integrates between their observed time centres over the same minute for
+every flow, using `grid - PV - battery - EV - V2X`. It does not extrapolate
+edges, split coarse hourly evidence, clip signed residuals, or replace an
+existing estimate. It refuses duplicate source mappings and non-watt inputs.
+
+The default maximum point gap is 15 seconds. Unsupported minutes remain
+unknown. If the owner explicitly accepts bounded interpolation, prepare a
+**new** plan with a larger bound, at most 180 seconds, and inspect the changed
+minutes and `max_interpolated_ms_per_source` values. This is an estimate of
+allocation, not proof of constant power or an instrument's delay. Signed
+minute residuals are retained; each displayed aggregate must still have
+complete supported coverage and a nonnegative resulting balance.
+
+## Apply the reviewed plan to the copy
+
+After reviewing the actual plan and its source selections, supply its exact
+hash:
+
+```sh
+bin/ftw-history-repair apply --workdir ./repair-2026-10-19 \
+  --sha256 REPLACE_WITH_THE_REVIEWED_PLAN_SHA256
+```
+
+The operation verifies the backup hashes and unchanged working inputs,
+inserts all correction minutes and clears derived daily caches in one SQLite
+transaction. A conflict rolls the transaction back. Retrying the same applied
+plan does not add energy again. `apply-result.json` reports before/after
+household Wh and quality for the **selected range**, not an assumed full day.
+If another gap remains unsupported, the complete total stays unknown and the
+accepted subtotal remains available.
+
+The correction rows carry the plan hash, backup hash, timestamp, reason and
+source mapping in their method metadata. They survive portable backup/restore
+and Core restart. Minute energy needed to interpret a corrected quarter is
+retained rather than merged away by energy rollup. Original measurement
+coverage remains separate from reconstructed household coverage.
+
+## Validate and deploy separately
+
+Use the patched Core to read `corrected/state.db` and `corrected/history.db`
+in an isolated test setup without equipment control. Check the selected
+period's daily total, ledger provenance, estimate labels and remaining gaps.
+The tool never starts a second Core and has no install or restart action.
+
+For the owner's stopped installation, an operator must verify the generation,
+release layout, file ownership, backup and absence of newer source data before
+installing the corrected history and patched Core together. The same source
+state configuration must still bind that history generation. Do not replace
+the owner's configuration with a stale test copy. To abandon or undo the
+repair before deployment, discard the working copy; the source is unchanged.
+For deployment rollback, restore the verified original history snapshot and
+matching original release while Core is stopped. Never copy SQLite files
+over a running Core or leave mismatched WAL/SHM files beside a restored DB.

go/cmd/ftw-history-endpoints/main.go

@@ -0,0 +1,95 @@
+// ftw-history-endpoints repairs completed historical minutes from bounding
+// power means. Preparing and applying leave equipment control to the live Core.
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"flag"
+	"fmt"
+	"github.com/srcfl/ftw/go/internal/state"
+	"os"
+	"time"
+)
+
+func run(ctx context.Context, args []string) error {
+	if len(args) == 0 {
+		return errors.New("use prepare or apply")
+	}
+	f := flag.NewFlagSet(args[0], flag.ContinueOnError)
+	source := f.String("state", "", "existing state.db")
+	planFile := f.String("plan", "", "reviewable plan JSON")
+	switch args[0] {
+	case "prepare":
+		since := f.String("since", "", "RFC3339 start")
+		until := f.String("until", "", "RFC3339 completed end")
+		gap := f.Duration("max-gap", 15*time.Minute, "maximum distance between valid endpoints (1–30m)")
+		reason := f.String("reason", "", "audit reason")
+		if err := f.Parse(args[1:]); err != nil {
+			return err
+		}
+		if *source == "" || *planFile == "" || f.NArg() != 0 {
+			return errors.New("state and new plan path required")
+		}
+		from, err := time.Parse(time.RFC3339, *since)
+		if err != nil {
+			return err
+		}
+		to, err := time.Parse(time.RFC3339, *until)
+		if err != nil {
+			return err
+		}
+		p, err := state.PrepareEndpointRepair(ctx, *source, state.EndpointRepairRequest{SinceMS: from.UnixMilli(), UntilMS: to.UnixMilli(), MaxGapMS: gap.Milliseconds(), Reason: *reason})
+		if err != nil {
+			return err
+		}
+		b, err := json.MarshalIndent(p, "", "  ")
+		if err != nil {
+			return err
+		}
+		file, err := os.OpenFile(*planFile, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
+		if err != nil {
+			return err
+		}
+		_, err = file.Write(append(b, '\n'))
+		closeErr := file.Close()
+		if err != nil {
+			return err
+		}
+		if closeErr != nil {
+			return closeErr
+		}
+		return json.NewEncoder(os.Stdout).Encode(map[string]any{"plan_sha256": p.SHA256, "correction_minutes": len(p.Minutes), "skipped": p.Skipped})
+	case "apply":
+		sha := f.String("sha256", "", "exact reviewed plan digest")
+		backup := f.String("backup", "", "independent verified history.db backup")
+		if err := f.Parse(args[1:]); err != nil {
+			return err
+		}
+		if *source == "" || *planFile == "" || *backup == "" || f.NArg() != 0 {
+			return errors.New("state, plan and backup required")
+		}
+		b, err := os.ReadFile(*planFile)
+		if err != nil {
+			return err
+		}
+		var p state.EndpointRepairPlan
+		if err = json.Unmarshal(b, &p); err != nil {
+			return err
+		}
+		n, err := state.ApplyEndpointRepair(ctx, *source, *backup, p, *sha)
+		if err != nil {
+			return err
+		}
+		return json.NewEncoder(os.Stdout).Encode(map[string]any{"plan_sha256": p.SHA256, "applied_minutes": n, "method": "endpoint_mean_v1"})
+	default:
+		return errors.New("unknown operation")
+	}
+}
+func main() {
+	if err := run(context.Background(), os.Args[1:]); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+}

go/cmd/ftw-history-repair/main.go

@@ -0,0 +1,92 @@
+// ftw-history-repair prepares and applies historical household corrections on
+// a separate database copy. It never writes the owner's source databases.
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"flag"
+	"fmt"
+	"io"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/srcfl/ftw/go/internal/state"
+)
+
+func run(ctx context.Context, args []string, out io.Writer) error {
+	if len(args) == 0 {
+		return errors.New("use prepare --state PATH --workdir NEW_DIR --since RFC3339 --until RFC3339 --grid DRIVER:METRIC --pv DRIVER:METRIC|none --battery DRIVER:AC_METRIC|none --ev DRIVER:METRIC|none --v2x DRIVER:METRIC|none --reason TEXT [--max-gap 15s], then apply --workdir DIR --sha256 REVIEWED_PLAN_HASH")
+	}
+	f := flag.NewFlagSet(args[0], flag.ContinueOnError)
+	workdir := f.String("workdir", "", "new private working directory for prepare; same directory for apply")
+	if args[0] == "apply" {
+		sha := f.String("sha256", "", "SHA-256 of the exact reviewed plan")
+		if err := f.Parse(args[1:]); err != nil {
+			return err
+		}
+		if *workdir == "" || f.NArg() != 0 {
+			return errors.New("workdir is required; no positional arguments")
+		}
+		result, err := state.ApplyHistoryRepair(ctx, *workdir, *sha)
+		if err != nil {
+			return err
+		}
+		return json.NewEncoder(out).Encode(result)
+	}
+	if args[0] != "prepare" {
+		return errors.New("unknown repair operation")
+	}
+	source := f.String("state", "", "source state.db; opened read-only")
+	since := f.String("since", "", "start RFC3339 with timezone, on a minute boundary")
+	until := f.String("until", "", "end RFC3339 with timezone, on a completed minute boundary")
+	reason := f.String("reason", "", "reason for this historical estimate")
+	gap := f.Duration("max-gap", 15*time.Second, "largest source-point gap to interpolate; up to 180s, explicitly reviewed")
+	flows := map[string]*string{}
+	for _, name := range []string{"grid", "pv", "battery", "ev", "v2x"} {
+		flows[name] = f.String(name, "", "explicit driver:power_metric or none; battery must be site-side AC")
+	}
+	if err := f.Parse(args[1:]); err != nil {
+		return err
+	}
+	if *source == "" || *workdir == "" || f.NArg() != 0 {
+		return errors.New("state and workdir required; no positional arguments")
+	}
+	start, err := time.Parse(time.RFC3339, *since)
+	if err != nil {
+		return err
+	}
+	end, err := time.Parse(time.RFC3339, *until)
+	if err != nil {
+		return err
+	}
+	r := state.HistoryRepairRequest{SinceMS: start.UnixMilli(), UntilMS: end.UnixMilli(), MaxGapMS: gap.Milliseconds(), Reason: *reason}
+	for name, value := range flows {
+		v := state.RepairPowerSource{Flow: name}
+		if *value == "none" {
+			v.Absent = true
+		} else {
+			parts := strings.SplitN(*value, ":", 2)
+			if len(parts) != 2 {
+				return fmt.Errorf("%s needs driver:power_metric or explicit none", name)
+			}
+			v.Driver, v.Metric = parts[0], parts[1]
+		}
+		r.Sources = append(r.Sources, v)
+	}
+	state.SortRepairSources(r.Sources)
+	plan, err := state.PrepareHistoryRepair(ctx, *source, *workdir, r)
+	if err != nil {
+		return err
+	}
+	return json.NewEncoder(out).Encode(plan)
+}
+
+func main() {
+	if err := run(context.Background(), os.Args[1:], os.Stdout); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+}