Stop the live-sessions reconciler deleting genuinely open sessions - #227
Merged
Merged
Conversation
reconcileLiveSessions() compared two independently truncated reads:
the first MAX_RECONCILE liveSessions docs by document id (a content
hash, effectively random order) against the first MAX_RECONCILE open
RTDB sessions in the caller's read order. With more than 500 sessions
open at once, a mirror document for a session that simply ranked
outside the second cut looked identical to one whose session had
ended, and got deleted -- erasing a genuine in-progress participant
from the dashboard for the rest of their session, with mirrorFixed
going non-zero every run.
Fixed by never deleting on a run whose open-session read was itself
capped: readWasTruncated(entries.length) treats entries.length hitting
the cap as proof the read may be incomplete (indistinguishable, from
inside this module, from there happening to be exactly that many open
sessions -- the conservative side of that ambiguity is to skip
deletion, not to delete). idsToDelete() is the extracted, pure
decision -- given the existing docs, the wanted ids, and whether the
read was truncated -- so the fix is a couple of unit-testable functions
rather than a change to the caller's signature. Creation and update
keep working exactly as before, including their own inherent cap
(documented on MAX_RECONCILE): a session ranking outside the cap gets
no mirror row until it ranks inside one on a later run, which is a
bounded-read tradeoff, not a soundness bug like deletion's was.
Reviewed LiveSessionsPanel.js and confirmed its copy already avoids
any completeness claim ("participants who have started and not yet
finished", plus its own "N more... not shown" note for the display-side
cap) -- no change needed there.
functions/src/__tests__/live-sessions.test.js: unit tests for
readWasTruncated and idsToDelete, including the actual bug -- more
open sessions than MAX_RECONCILE, all with mirror docs, none deleted
-- and the case it must still catch: a mirror doc for a session that
is genuinely gone is deleted once the read was not truncated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q8xf16ov88M3KojPVfQwJT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
reconcileLiveSessions()(functions/src/live-sessions.ts) decided whichliveSessionsmirror documents to delete by comparing two independentlytruncated sets:
existing, the firstMAX_RECONCILE(500)liveSessionsdocs by document id (a SHA-256 hash, effectively random but stable order),
and
wanted, the firstMAX_RECONCILEopen sessions in the caller's RTDBread order. Those orderings are unrelated, so with more than 500 concurrent
sessions a mirror document whose session was genuinely still open, but
ranked 501+ in that run's RTDB listing, was absent from
wanted, passed thestartedAt > readAt - 60screation-race guard, and got deleted. It wasn't inthe scoped set either, so it was never recreated: a real in-progress
participant vanished from the dashboard for the rest of their session, and
mirrorFixedwould go non-zero every run.The fix
Never delete on a run whose open-session read was itself capped -- a capped
read can prove a session is open, but never prove one is absent, since an
absence might just mean it ranked outside the cut.
Two pure functions now carry the logic, both unit-tested without an
emulator:
readWasTruncated(entriesCount, cap = MAX_RECONCILE)-- true once theentries this run received hit the cap. The caller already always hands in
at most
MAX_RECONCILEentries (it slices its own RTDB read beforefetching each session's meta), so
entries.length >= MAX_RECONCILEis asufficient, self-contained signal from inside this module alone. It can't
tell that case apart from "there happen to be exactly
capopensessions" -- treating both as truncated is the conservative side of that
ambiguity.
idsToDelete(existing, wanted, readAt, truncated)-- the extracteddeletion decision. Returns nothing at all when
truncatedis true;otherwise applies the pre-existing 60-second creation-race guard exactly
as before.
No signature change to
reconcileLiveSessions, and no change toscheduled-staging-sweep.ts's call site -- the fix is entirely internal tolive-sessions.ts. Creation and update behavior is unchanged; thecreation-side cap (a session ranking outside
MAX_RECONCILEgets no mirrorrow until it ranks inside one on a later run) is inherent to bounding
per-run RTDB reads, not a soundness bug like deletion's was, and is now
documented on
MAX_RECONCILEitself alongside the deletion fix.Reviewed
components/dashboard/LiveSessionsPanel.js: its copy already makesno completeness claim ("participants who have started and not yet
finished", plus its own "N more... not shown" note for the unrelated
25-row display cap), so no change was needed there.
Tests
functions/src/__tests__/live-sessions.test.js(pure, no emulator):readWasTruncated: false under the cap, true at and past it (with aninjectable cap parameter so the test doesn't need to build fixtures at
production scale), and the production default (
MAX_RECONCILE).idsToDelete:read was not truncated
MAX_RECONCILE + 1mirror docs, all for genuinely opensessions, with
wantedcovering only the firstMAX_RECONCILE(standing in for a read that couldn't fit the rest) and
truncated = true-- asserts zero deletions.Verified in this session's worktree:
cd functions && npm run buildandroot
npm run lintare clean, and the coordinator's emulator run reportsstaging-emulator,live-sessions, andsession-id-validation-emulatorall green (58 tests).
🤖 Generated with Claude Code
https://claude.ai/code/session_01Q8xf16ov88M3KojPVfQwJT