fix(agents-server): runFinished child wakes delivered only for the last child — read shape delete row from value - #4803
Open
espaciofuturoio wants to merge 2 commits into
Conversation
…, not only `old_value` Electric's `replica: full` delete message carries the deleted row in `value` (`old_value` is only set on updates). WakeRegistry read `old_value.id`, got NaN, and reset the whole in-memory registration cache on every delete. Because manifest sync unregisters+registers per child, a parent that spawned N children with `runFinished` wakes kept only the registration inserted after the last delete — only its last child ever woke it — and the shape log replayed the same deletes on every restart. Measured on two 5-child swarms against electricsql/electric 1.7.10 + agents-server 0.6.4. The existing sync test modelled deletes without a row, which cannot tell 'removed one' from 'reset everything'; the new test deletes one of two registrations with the row in `value` and asserts the other still evaluates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
A parent entity that spawns N children with
wake: { on: 'runFinished' }is woken only by the last child spawned. The other children's run completions never reach the parent, and restarting the coordinator does not help. No warning is logged.Reproduced on two 5-child swarms (
electricax/agents-server:0.6.4,electricsql/electric:1.7.10): all 5 rows are present inwake_registrations, but only the last registration's source produces awakeevent on the parent stream, before and after a coordinator restart. Sending a message to any of the first four children (so they complete a new run) never wakes the parent; the last child does.It is easy to miss because a parent that runs the LLM on every wake stays 20 s in idle afterwards, and children finishing inside that window are consumed in-process without going through the coordinator. As soon as the parent closes quickly (
sleep()or a state-only handler), fan-in silently breaks.Root cause
WakeRegistry.applyShapeMessagehandles shapedeletemessages by reading the deleted row id frommessage.old_value. Electric's shape stream withreplica: fullcarries the deleted row inmessage.value—old_valueis only set on updates. Measured againstelectricsql/electric:1.7.10:{"value":{"id":"90","manifest_key":"child:…","source_url":"/survey_worker/…","subscriber_url":"/orchestrator/kit1"}, "key":"\"public\".\"wake_registrations\"/\"90\"","headers":{"operation":"delete"}}So
Number(undefined)isNaNand the code takes its fail-closed branch,resetCachedRegistrations(), on every delete.syncManifestWakesdoes unregister + register per child manifest, so after N spawns the cache holds only the registration inserted after the last delete. The shape log replays those deletes on every restart, which is why a restart does not recover.Fix
Read the id from
valuefirst and fall back toold_value; keep the full reset only when neither carries an id.The existing sync test modelled the delete message without a row, which cannot distinguish "removed one registration" from "reset everything". The new test inserts two registrations, deletes one with the row in
value, and asserts the other still evaluates.Verified end to end with the image built from this branch: 5/5 children wake the parent (registrations 110–114 all deliver), and a child completing a run 28 s after the parent's session closed also wakes it.
Related
Changeset:
@electric-ax/agents-serverpatch.