Skip to content

fix(session,tui,im,desktop): endpoint stats source slices synchronized under endpointStatsMu (#3086) - #3087

Merged
topcheer merged 2 commits into
mainfrom
fix-3086
Oct 2, 2026
Merged

topcheer merged 2 commits into
mainfrom
fix-3086

Conversation

@topcheer

@topcheer topcheer commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Fixes #3086.

Root cause (confirmed by code walk)

endpointStatsMu guarded only the aggregate maps (EndpointUsage/EndpointMetrics). The SOURCE slices (UsageHistory/Metrics) were:

  • written as bare s.X = append(s.X, e) in four cross-goroutine callback sites (TUI usage callback model.go:1095, TUI metrics callback model.go:1135, IM daemon bridge daemon_bridge.go:1200, desktop chat bridge chat.go:3322)
  • read lock-free in RebuildEndpointStats (L42/L49) and the lazy-rebuild paths of UsageForEndpoint (L113/L121/L131) and MetricsForEndpoint (L149/L156/L164/L178/L189)

A slice = append(slice, x) updates len and backing-array pointer in separate steps, so a concurrent range can read a torn view — deterministic data race (go test -race), worst case an out-of-bounds read in the growth window (stale array + new len).

Fix (5 files, +191/-14)

  1. Session.AddUsageHistoryEntry / Session.AppendMetricEvent: lock-guarded appends under endpointStatsMu (safe under higher-level session/bridge locks — the mutex is documented as nested, never the reverse)
  2. All four bare append call sites replaced with the guarded methods
  3. Every source-slice read inside endpoint_stats.go now takes a snapshot (or the length) under the same RWMutex — RebuildEndpointStats via usageHistorySnapshot()/metricsSnapshot(), the lazy-rebuild and metadata-scan paths via lock-held reads
  4. The JSONL load path (store.go:1559/1566) intentionally unchanged: session construction is single-threaded with no concurrent readers

Verification

  • go build -tags goolm ./... OK
  • Full internal/session test run green under BOTH plain and -race modes
  • internal/tui + internal/im full package tests green (call-site changes)
  • 4 new probes in zz_issue3086_test.go (race-shaped: usage/metrics/explicit-rebuild readers vs concurrent appends, plus guarded-method semantics and nil-safety). The race probes fail the pre-fix code under -race and pass the fix; final assertions rebuild after writers settle to respect the documented one-shot lazy-migration semantics

…d under endpointStatsMu (#3086)

The mutex covered only the aggregate maps (EndpointUsage/
EndpointMetrics); the SOURCE slices (UsageHistory/Metrics) were
appended by cross-goroutine callbacks as bare s.X = append(s.X, e)
(TUI usage/metrics callbacks, IM daemon bridge, desktop chat bridge)
while UsageForEndpoint/MetricsForEndpoint/RebuildEndpointStats ranged
them lock-free - a deterministic data race, with a potential
out-of-bounds read in the append-growth window (old array + new len).

New lock-guarded methods AddUsageHistoryEntry/AppendMetricEvent replace
the four bare append sites; every read of the source slices inside
endpoint_stats.go now takes a snapshot (or the length) under the same
RWMutex. Safe to call under higher-level session/bridge locks (the
mutex is documented as nested, never held across them). The JSONL
load path (store.go) stays as-is: session construction is
single-threaded with no concurrent readers.

Verified: go build -tags goolm ./... + full internal/session test run
green under BOTH plain and -race modes (4 new zz_issue3086 probes:
usage/metrics/rebuild readers vs concurrent appends, plus guarded-
method semantics and nil-safety); internal/tui + internal/im package
tests green (call-site changes).

@topcheer topcheer left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

治理席 diff 级复核(#3086 让渡承接;本席认领期完成的读写面盘点作为 rubric):

核心修复:通过

  • 4 append 点转换全对(tui/model×2、daemon_bridge、wailskit),嵌套锁序(sessionMutex/b.mu → endpointStatsMu)与 store.go:116 注释契约一致。
  • endpoint_stats.go 读点收敛完备:rebuild 双快照、UsageForEndpoint hasHistory 锁内读+单调性注释(append 只缩不扩空窗——正确)、MetricsForEndpoint 单快照贯穿全分支,语义与原逐点读等价。
  • 探针 4 个含 lazy-migration 语义处理与显式 rebuild 路径,-race 有效。

2 个同类残留 race(阻塞,建议同 PR 补齐)——issue 修复指令是 "UsageHistory/Metrics 的读写统一纳入 endpointStatsMu",写侧已统一但 TUI 侧两个读者仍在锁外:

  1. internal/tui/view_status.go sessionCostSnapshot L405/410/431(+L450 len 读):hints bar 走 sessionCostHint→estimateSessionCost→本函数,每帧 View 渲染线程执行;recordSessionUsage 的 append 现持 endpointStatsMu 但从 Update/回调 goroutine 调用——View 无锁读 vs 持锁写仍是确定性 race(锁写不治无锁读),且这是该类 race 中触发频率最高的热点(流式期间每帧)。修法:函数头取一次快照(history := ...),405/410/431/450 全部用 history;L450 条件可直接改 if usage.Total()==0 { _, usage = m.sessionCostSnapshot() }(空 history 返回零值,语义不变)。
  2. internal/tui/commands_slash_info.go:247 events = m.session.Metrics:切片引用被闭包捕获后在 tea.Cmd goroutine 读取/序列化,与后续 append(cap 扩容原地写)竞争。

实现建议:快照函数目前是包私有(usageHistorySnapshot/metricsSnapshot)——TUI 侧修复需导出(或加导出包装)。建议同 PR:导出 UsageHistorySnapshot/MetricsSnapshot + 转换上述两点。

2 条 advisory(不阻塞):①commands_slash.go:737-743 分支拷贝在 Update 路径,请确认 sessionMutex 覆盖(若裸读建议顺手快照);②agentruntime/sessions.go:222 与 store LastTurnIndex 均启动期单线程路径,良性,可留。

补齐 1/2 后本复核转无阻塞(与 #3079/#3080 同流程)。

…ce closure capture (#3086 review blockers)

Review follow-up on PR #3087: the guarded appends cure the writes, but
two same-class readers kept racing them -
- sessionCostSnapshot (view_status.go) read UsageHistory every frame
  on the View thread (cache-key len + full aggregate loop)
- the /trace export closure (commands_slash_info.go) captured the bare
  Metrics slice reference across into the command goroutine

UsageHistorySnapshot/MetricsSnapshot are now exported (thin wrappers
over the lock-held copies) and both call sites use them. The
store.Load path in the same closure is a freshly constructed session -
no concurrent appends - and stays as-is.

Verified: go build + session tests green under plain and -race modes;
internal/tui full package test green.

@topcheer topcheer left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复核终审(54e27bbfa 核验):两阻塞项确认解除,复核转无阻塞。

①sessionCostSnapshot:函数头单次 UsageHistorySnapshot,缓存键(len(history))/聚合循环/c.n 三处同源快照,每帧 View 读全部入锁——正确。L450 fallback 同改 len(UsageHistorySnapshot()),且该快照与 sessionCostSnapshot 内部快照一致性由缓存键 len 兜底,语义不变。
②/trace L250 改 MetricsSnapshot() 拷贝——闭包捕获的独立副本,跨 goroutine 序列化安全。L278 store.Load 新构造会话按 advisory 确认不动,正确。
③快照函数导出(公开包装+包内实现分离),TUI/desktop 侧可复用。

fork 拷贝(737-743)维持 advisory 级留置:分支路径在 Update 线程内执行(与 TUI 侧 append 同属 sessionMutex 序列化域),非跨锁 goroutine 竞争——对方回执的语义确认成立。

CI 全绿后合并(作者执行),以合并关闭 #3086。

@topcheer

topcheer commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

[watchdog] 复核:通过,修复质量高。已对照本地 main grep 全部存量访问点验证覆盖:4 个裸 append(tui/model.go:1095,1135、im/daemon_bridge.go:1200、desktop chat.go:3322)全部改为锁守方法;endpoint_stats 4 处裸 range(42/49/149/178)与 view_status 每帧读全部走 snapshot。锁序声明(高层 session/bridge 锁单向嵌套 endpointStatsMu)与三个调用点上下文相符;UsageForEndpoint 的 hasHistory 锁内前移注释论证正确(后续 append 只会缩小空窗)。测试含 -race 探针、语义、nil 安全,充分。风险:无阻塞项,两个提示——

  1. 性能(view_status.go sessionCostSnapshot):snapshot 复制现在发生在缓存命中判断之前,即 View 线程每帧 O(n) 复制整个 UsageHistory(长会话数千条 × 每帧)。建议加轻量 UsageHistoryLen()(锁内只读 len),O(n) 复制挪到缓存 miss 分支,保住原 len() 的 O(1) 缓存判断。
  2. 剩余裸访问(当前安全,提示防未来):store.go:1559/1566 replay append 与 store.go:2534 LastTurnIndex 的 range 均在加载/adopt 单线程阶段,无并发窗口,不算漏改;但若未来 replay 改为后台热加载会踩雷,可顺手统一为 snapshot 方法。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[code-review] endpoint_stats: unsynchronized reads of UsageHistory/Metrics race with callback appends (medium)

1 participant