Conversation
Orchestrators driving idle agents reported re-spawn gaps every ~5 min (5 iterations × 60 s default per call) where messages queued unread between subagent cycles. The 5-iteration cap was a Pluto design choice, not a Claude Code constraint — the only platform constraint is the 600 s stream watchdog, which is satisfied by per-call timeout, not by total iteration count. Changes: - Default iterations 5 → 15 (15 × 60 s = 15 min subagent lifetime) - New --iterations CLI flag on PlutoMCPFriend so orchestrators can tune per-deployment without code edits - Threaded iterations through PlutoMCPServer and every prompt builder (build_connection_block, build_watch_prompt_body, build_role/protocol/ guide_prompt_body) - Updated tests to lock in the new default and added two tests pinning the custom-iterations behavior so regressions surface fast Trade-off: longer subagent lifetime trades a slightly larger subagent budget for a 3× reduction in respawn-gap latency. Smaller is still available via --iterations N for memory-constrained deployments. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two related correctness bugs in agent lifecycle:
1. Stale agents accumulating across restarts
pluto_persistence saved every agent (incl. long-disconnected) and
restored them all marked disconnected. With no TTL, agents from days
or weeks ago kept reappearing in pluto_msg_hub:list_agents_detailed/0
after each restart, confusing callers of GET /agents?detailed=true.
Fix: new ?DEFAULT_AGENT_STALE_AFTER_MS (7 days) + agent_stale_after_ms
pluto_config key. Filter applied at snapshot save (don't grow
unboundedly) and snapshot restore (don't resurrect ancient entries).
No runtime ETS pruning — restart is the natural cleanup cycle.
2. HTTP token leak on unregister_agent
The unregister handler released the agent's name from the registry
and deleted its session record but left the HTTP session token alive
in ?ETS_HTTP_SESSIONS. Because /agents/poll, /agents/peek, /agents/ack
resolve token -> agent_id and then drain the inbox by agent_id,
any orphaned client could keep polling with the stale token and
consume messages destined for whoever next claimed the same agent_id.
Fix: call evict_http_sessions(AgentId) in both the unregister and
grace-expired handlers so unregister atomically invalidates all
access tokens. Safe for grace-period reconnects, which use session_id
(not the token) for identity resumption — the reconnecting client
gets a fresh token issued by do_register_http_with_session.
Affected callers of the leaky path:
- pluto_session:cleanup/1 (TCP session close)
- pluto_session:execute_admin OP_ADMIN_DISCONNECT
- pluto_heartbeat.erl TCP heartbeat timeout
Tests added (TODO: run rebar3 eunit before merging):
- src_erl/test/pluto_stale_agents_tests.erl — pure-function tests of
the drop_stale_agents/3 filter (boundary, default 7-day, non-record
pass-through, empty list)
- src_erl/test/pluto_token_leak_tests.erl — integration tests proving
old token gets 404 on peek/ack after unregister, and cannot drain
messages addressed to a new owner that claimed the same agent_id
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Establishes a one-time session-wide "spec contract" the Orchestrator broadcasts at session bootstrap (lock protocol, queue rule, no-emoji, test command, release locks). Subsequent task_assigned dispatches reference it by spec_id via the new spec_ref field instead of inlining ~40 lines of boilerplate per task — saving ~1-2K tokens per dispatch. - protocol.md: new §4.12 spec_contract message, §7 walkthrough, spec_ref field on §4.1 task_assigned. - roles/orchestrator.md: bootstrap step + concrete broadcast/dispatch curl examples + clarification-request handling for unseen spec_ref. - roles/specialist.md + 7 other worker roles: cache spec_contract by spec_id, apply on task_assigned with matching spec_ref, ask for re-broadcast when missing. Also documents in docs/guide/pluto-mcp-friend.md that --agent-id is the agent's identity in Pluto's distributed system, that Pluto auto-suffixes when the requested name is taken, and that mid-session role activation is via the /pluto-role-<name> MCP slash command.
leondavi
changed the title
PlutoServer.sh --stats hits the existing TCP "stats" op (no registration required) and renders a colour dashboard from pluto_stats:get_summary/0 — messages sent/received/broadcast with avg msg/sec, msg/hour, msg/day computed against uptime; locks acquired/released/expired/renewed/waited; deadlocks, agents (cumulative + live), and a top-10 agents-by-activity table. server_stats.py is a thin TCP client mirroring server_info.py's shape; --json mode emits the raw payload.
…P port mismatches
Two bugs the wizard hit on macOS (default bash 3.2):
- `${ans,,}` is bash 4+ only. macOS default is 3.2, so the y/n prompt
in offer_to_start_server (and the parallel one in PlutoInstall.sh)
errored with "bad substitution" and dropped the user out of the
flow. Replaced with a portable `tr '[:upper:]' '[:lower:]'`.
- check_pluto_reachable said "OFFLINE" whenever the configured HTTP
port had no listener, even though the daemon was alive on its TCP
control port (and on a stale HTTP port). The typical case: the
user edits config/pluto_config.json to bump host_http_port from
9201 to 9202 but the daemon is still bound to 9201. Now the
wizard probes the TCP control port and 9201/9202; if any of them
respond it tells the user the daemon is alive but on a different
HTTP port and prints the exact restart command.
…g/min Extrapolating 2 messages over 3 minutes to "898 msg/day" is misleading, not informative. Now any rate whose window exceeds the current uptime prints "n/a (uptime <1h)" / "n/a (uptime <1d)" instead of a fabricated projection. Also adds avg msg/min, the most useful short-window rate during normal multi-agent sessions.
Previously the watcher only respawned when the in-flight Task itself ended (messages, N iterations empty, or watchdog). Drains via turn-start pluto_recv or _pluto_inbox piggyback didn't trigger a respawn, so a still-running watcher could finish its cycle and leave the agent with no listener until the next drain noticed. New rule in build_connection_block: any drain — watcher fire, pluto_recv, or piggyback — MUST be followed by an immediate watcher respawn in the same turn. /pluto-check inherits the rule too. Adds /pluto-watch-stop slash command + build_watch_stop_prompt_body() for the watcher_stop kill switch: a session-level flag the agent honours to disable auto-respawn (turn-driven pluto_recv only). /pluto-watch implicitly clears the flag on resume.
…021/v023 Adds 4 unit tests in tests/test_mcp_friend.py and one capabilities assertion: - test_watch_stop_prompt_engages_kill_switch — /pluto-watch-stop body must name watcher_stop, forbid spawning new Tasks, allow the in-flight one to drain, point at the resume slash, and emit the exact "watcher_stop engaged" ack. - test_check_prompt_respawns_watcher_after_drain — /pluto-check is a drain, so the body must include the respawn instruction with the watcher_stop exception. - test_connection_block_drain_respawn_rule — the always-on role block must spell out all three drain paths (watcher fire, pluto_recv, _pluto_inbox piggyback) and document watcher_stop + /pluto-watch-stop. - test_watch_prompt_clears_kill_switch — /pluto-watch must explicitly clear watcher_stop on resume. - test_capabilities_register — extended to require pluto-watch-stop. Also fixes test_v021_http_sessions and test_v023_features so they auto-discover ports from config/pluto_config.json (with PLUTO_HOST / PLUTO_PORT / PLUTO_HTTP_PORT env-var overrides). They hardcoded 9000/9001 and silently failed against any server using the current 9200/9202 defaults — pre-existing breakage from the v0.2.6 port bump that this surface change finally pulls into the green. All 148 unit + integration tests now pass; the 2 remaining failures in test_404_hints.py (not_found vs unknown_route response shape on two URL patterns) are an unrelated pre-existing server bug.
…tuses, coordination_requests counter - Add explicit `unregister` protocol op so agents leave cleanly (selftest agents now show in Disconnected rather than Live after test completes) - Add four agent lifecycle statuses: connected, disconnected (grace period), disconnected_timeout (tombstone after grace expires), recovered (reconnected within grace period); all routing updated to treat recovered == connected - selftest: each check sends unregister at end; check_direct_message registers B cleanly before A sends to it - Add agents_unregistered_clean counter incremented on explicit unregister op - Add coordination_requests counter (register/unregister, locks, messaging, tasks, pub/sub) vs maintenance (ping, stats, queries, admin, ack) - Stats dashboard: Registered (Total) label, Unregistered (clean) row, Requests section split into Coordination/Maintenance/Total, selftest agent note on Total agents line Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Owner
Author
|
Superseded by the v0.2.9 PR — work continues on the new branch with the snapshot/restore feature, recovered_from_file status, and the rest of the v0.2.8 series rolled forward. Closing without merge. |
Merged
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
spec_contractmessage type + optionalspec_refontask_assigned. The Orchestrator broadcasts universal constraints (lock protocol, queue rule, no-emoji, test command, release locks) once at session bootstrap; subsequent dispatches reference them byspec_idinstead of inlining ~40 lines of boilerplate per task — saves ~1–2K tokens per dispatch. Roles updated to bootstrap (orchestrator) and cache/apply (all 8 worker roles).--iterationsknob; sleep no longer blocks the inbox loop.pluto_stale_agents_testsandpluto_token_leak_testssuites.--agent-idis the agent's identity on the Pluto distributed system, that Pluto auto-suffixes when the requested name is taken, and that mid-session role activation is via the/pluto-role-<name>MCP slash command.Changes (20 files, +598 / -30)
library/protocol.md— §4.12spec_contract, §7 walkthrough,spec_refon §4.1task_assigned.library/roles/*.md— orchestrator bootstraps, all 8 worker roles cache and apply.docs/guide/pluto-mcp-friend.md— agent-name + auto-suffix + slash-command activation.src_erl/src/pluto_persistence.erl,pluto_msg_hub.erl,include/pluto.hrl— stale-agent + token-leak fixes.src_erl/test/pluto_stale_agents_tests.erl,pluto_token_leak_tests.erl— new test suites.src_py/agent_mcp_friend/{prompts,server,pluto_mcp_friend}.py,tests/test_mcp_friend.py— watcher lifetime + iterations flag.Test plan
tests/test_mcp_friend.py(37),tests/test_agent_friend.py(55, 1 skip).pluto_stale_agents_tests,pluto_token_leak_tests) — pending CI run.spec_contract, a worker caches it, then atask_assignedwithspec_refresolves correctly.PlutoMCPFriend.sh --agent-id specialist-1against the same server — second registration should land under an auto-suffixed id.