Skip to content

fix(agents-server): runFinished child wakes delivered only for the last child — read shape delete row from value - #4803

Open
espaciofuturoio wants to merge 2 commits into
electric-sql:mainfrom
espaciofuturoio:fix/wake-registry-shape-delete-value
Open

espaciofuturoio wants to merge 2 commits into
electric-sql:mainfrom
espaciofuturoio:fix/wake-registry-shape-delete-value

Conversation

@espaciofuturoio

Copy link
Copy Markdown

Symptom

A parent entity that spawns N children with wake: { on: 'runFinished' } is woken only by the last child spawned. The other children's run completions never reach the parent, and restarting the coordinator does not help. No warning is logged.

Reproduced on two 5-child swarms (electricax/agents-server:0.6.4, electricsql/electric:1.7.10): all 5 rows are present in wake_registrations, but only the last registration's source produces a wake event on the parent stream, before and after a coordinator restart. Sending a message to any of the first four children (so they complete a new run) never wakes the parent; the last child does.

It is easy to miss because a parent that runs the LLM on every wake stays 20 s in idle afterwards, and children finishing inside that window are consumed in-process without going through the coordinator. As soon as the parent closes quickly (sleep() or a state-only handler), fan-in silently breaks.

Root cause

WakeRegistry.applyShapeMessage handles shape delete messages by reading the deleted row id from message.old_value. Electric's shape stream with replica: full carries the deleted row in message.valueold_value is only set on updates. Measured against electricsql/electric:1.7.10:

{"value":{"id":"90","manifest_key":"child:…","source_url":"/survey_worker/…","subscriber_url":"/orchestrator/kit1"},
 "key":"\"public\".\"wake_registrations\"/\"90\"","headers":{"operation":"delete"}}

So Number(undefined) is NaN and the code takes its fail-closed branch, resetCachedRegistrations(), on every delete. syncManifestWakes does unregister + register per child manifest, so after N spawns the cache holds only the registration inserted after the last delete. The shape log replays those deletes on every restart, which is why a restart does not recover.

Fix

Read the id from value first and fall back to old_value; keep the full reset only when neither carries an id.

The existing sync test modelled the delete message without a row, which cannot distinguish "removed one registration" from "reset everything". The new test inserts two registrations, deletes one with the row in value, and asserts the other still evaluates.

Verified end to end with the image built from this branch: 5/5 children wake the parent (registrations 110–114 all deliver), and a child completing a run 28 s after the parent's session closed also wakes it.

Related

Changeset: @electric-ax/agents-server patch.

…, not only `old_value`

Electric's `replica: full` delete message carries the deleted row in `value`
(`old_value` is only set on updates). WakeRegistry read `old_value.id`, got NaN,
and reset the whole in-memory registration cache on every delete. Because
manifest sync unregisters+registers per child, a parent that spawned N children
with `runFinished` wakes kept only the registration inserted after the last
delete — only its last child ever woke it — and the shape log replayed the same
deletes on every restart. Measured on two 5-child swarms against
electricsql/electric 1.7.10 + agents-server 0.6.4.

The existing sync test modelled deletes without a row, which cannot tell
'removed one' from 'reset everything'; the new test deletes one of two
registrations with the row in `value` and asserts the other still evaluates.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant