Conversation
Orchestrators driving idle agents reported re-spawn gaps every ~5 min (5 iterations × 60 s default per call) where messages queued unread between subagent cycles. The 5-iteration cap was a Pluto design choice, not a Claude Code constraint — the only platform constraint is the 600 s stream watchdog, which is satisfied by per-call timeout, not by total iteration count. Changes: - Default iterations 5 → 15 (15 × 60 s = 15 min subagent lifetime) - New --iterations CLI flag on PlutoMCPFriend so orchestrators can tune per-deployment without code edits - Threaded iterations through PlutoMCPServer and every prompt builder (build_connection_block, build_watch_prompt_body, build_role/protocol/ guide_prompt_body) - Updated tests to lock in the new default and added two tests pinning the custom-iterations behavior so regressions surface fast Trade-off: longer subagent lifetime trades a slightly larger subagent budget for a 3× reduction in respawn-gap latency. Smaller is still available via --iterations N for memory-constrained deployments. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two related correctness bugs in agent lifecycle:
1. Stale agents accumulating across restarts
pluto_persistence saved every agent (incl. long-disconnected) and
restored them all marked disconnected. With no TTL, agents from days
or weeks ago kept reappearing in pluto_msg_hub:list_agents_detailed/0
after each restart, confusing callers of GET /agents?detailed=true.
Fix: new ?DEFAULT_AGENT_STALE_AFTER_MS (7 days) + agent_stale_after_ms
pluto_config key. Filter applied at snapshot save (don't grow
unboundedly) and snapshot restore (don't resurrect ancient entries).
No runtime ETS pruning — restart is the natural cleanup cycle.
2. HTTP token leak on unregister_agent
The unregister handler released the agent's name from the registry
and deleted its session record but left the HTTP session token alive
in ?ETS_HTTP_SESSIONS. Because /agents/poll, /agents/peek, /agents/ack
resolve token -> agent_id and then drain the inbox by agent_id,
any orphaned client could keep polling with the stale token and
consume messages destined for whoever next claimed the same agent_id.
Fix: call evict_http_sessions(AgentId) in both the unregister and
grace-expired handlers so unregister atomically invalidates all
access tokens. Safe for grace-period reconnects, which use session_id
(not the token) for identity resumption — the reconnecting client
gets a fresh token issued by do_register_http_with_session.
Affected callers of the leaky path:
- pluto_session:cleanup/1 (TCP session close)
- pluto_session:execute_admin OP_ADMIN_DISCONNECT
- pluto_heartbeat.erl TCP heartbeat timeout
Tests added (TODO: run rebar3 eunit before merging):
- src_erl/test/pluto_stale_agents_tests.erl — pure-function tests of
the drop_stale_agents/3 filter (boundary, default 7-day, non-record
pass-through, empty list)
- src_erl/test/pluto_token_leak_tests.erl — integration tests proving
old token gets 404 on peek/ack after unregister, and cannot drain
messages addressed to a new owner that claimed the same agent_id
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Establishes a one-time session-wide "spec contract" the Orchestrator broadcasts at session bootstrap (lock protocol, queue rule, no-emoji, test command, release locks). Subsequent task_assigned dispatches reference it by spec_id via the new spec_ref field instead of inlining ~40 lines of boilerplate per task — saving ~1-2K tokens per dispatch. - protocol.md: new §4.12 spec_contract message, §7 walkthrough, spec_ref field on §4.1 task_assigned. - roles/orchestrator.md: bootstrap step + concrete broadcast/dispatch curl examples + clarification-request handling for unseen spec_ref. - roles/specialist.md + 7 other worker roles: cache spec_contract by spec_id, apply on task_assigned with matching spec_ref, ask for re-broadcast when missing. Also documents in docs/guide/pluto-mcp-friend.md that --agent-id is the agent's identity in Pluto's distributed system, that Pluto auto-suffixes when the requested name is taken, and that mid-session role activation is via the /pluto-role-<name> MCP slash command.
PlutoServer.sh --stats hits the existing TCP "stats" op (no registration required) and renders a colour dashboard from pluto_stats:get_summary/0 — messages sent/received/broadcast with avg msg/sec, msg/hour, msg/day computed against uptime; locks acquired/released/expired/renewed/waited; deadlocks, agents (cumulative + live), and a top-10 agents-by-activity table. server_stats.py is a thin TCP client mirroring server_info.py's shape; --json mode emits the raw payload.
…P port mismatches
Two bugs the wizard hit on macOS (default bash 3.2):
- `${ans,,}` is bash 4+ only. macOS default is 3.2, so the y/n prompt
in offer_to_start_server (and the parallel one in PlutoInstall.sh)
errored with "bad substitution" and dropped the user out of the
flow. Replaced with a portable `tr '[:upper:]' '[:lower:]'`.
- check_pluto_reachable said "OFFLINE" whenever the configured HTTP
port had no listener, even though the daemon was alive on its TCP
control port (and on a stale HTTP port). The typical case: the
user edits config/pluto_config.json to bump host_http_port from
9201 to 9202 but the daemon is still bound to 9201. Now the
wizard probes the TCP control port and 9201/9202; if any of them
respond it tells the user the daemon is alive but on a different
HTTP port and prints the exact restart command.
…g/min Extrapolating 2 messages over 3 minutes to "898 msg/day" is misleading, not informative. Now any rate whose window exceeds the current uptime prints "n/a (uptime <1h)" / "n/a (uptime <1d)" instead of a fabricated projection. Also adds avg msg/min, the most useful short-window rate during normal multi-agent sessions.
Previously the watcher only respawned when the in-flight Task itself ended (messages, N iterations empty, or watchdog). Drains via turn-start pluto_recv or _pluto_inbox piggyback didn't trigger a respawn, so a still-running watcher could finish its cycle and leave the agent with no listener until the next drain noticed. New rule in build_connection_block: any drain — watcher fire, pluto_recv, or piggyback — MUST be followed by an immediate watcher respawn in the same turn. /pluto-check inherits the rule too. Adds /pluto-watch-stop slash command + build_watch_stop_prompt_body() for the watcher_stop kill switch: a session-level flag the agent honours to disable auto-respawn (turn-driven pluto_recv only). /pluto-watch implicitly clears the flag on resume.
…021/v023 Adds 4 unit tests in tests/test_mcp_friend.py and one capabilities assertion: - test_watch_stop_prompt_engages_kill_switch — /pluto-watch-stop body must name watcher_stop, forbid spawning new Tasks, allow the in-flight one to drain, point at the resume slash, and emit the exact "watcher_stop engaged" ack. - test_check_prompt_respawns_watcher_after_drain — /pluto-check is a drain, so the body must include the respawn instruction with the watcher_stop exception. - test_connection_block_drain_respawn_rule — the always-on role block must spell out all three drain paths (watcher fire, pluto_recv, _pluto_inbox piggyback) and document watcher_stop + /pluto-watch-stop. - test_watch_prompt_clears_kill_switch — /pluto-watch must explicitly clear watcher_stop on resume. - test_capabilities_register — extended to require pluto-watch-stop. Also fixes test_v021_http_sessions and test_v023_features so they auto-discover ports from config/pluto_config.json (with PLUTO_HOST / PLUTO_PORT / PLUTO_HTTP_PORT env-var overrides). They hardcoded 9000/9001 and silently failed against any server using the current 9200/9202 defaults — pre-existing breakage from the v0.2.6 port bump that this surface change finally pulls into the green. All 148 unit + integration tests now pass; the 2 remaining failures in test_404_hints.py (not_found vs unknown_route response shape on two URL patterns) are an unrelated pre-existing server bug.
…tuses, coordination_requests counter - Add explicit `unregister` protocol op so agents leave cleanly (selftest agents now show in Disconnected rather than Live after test completes) - Add four agent lifecycle statuses: connected, disconnected (grace period), disconnected_timeout (tombstone after grace expires), recovered (reconnected within grace period); all routing updated to treat recovered == connected - selftest: each check sends unregister at end; check_direct_message registers B cleanly before A sends to it - Add agents_unregistered_clean counter incremented on explicit unregister op - Add coordination_requests counter (register/unregister, locks, messaging, tasks, pub/sub) vs maintenance (ping, stats, queries, admin, ack) - Stats dashboard: Registered (Total) label, Unregistered (clean) row, Requests section split into Coordination/Maintenance/Total, selftest agent note on Total agents line Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… status - Add `snapshot_self` and `restore_from_snapshot` protocol ops (TCP + HTTP) so an agent can persist its coordination state to a `.plut` JSON file plus a markdown recovery prompt and restore them on a fresh process - Add `recovered_from_file` agent status, propagated through routing, listing, find_agents, and broadcast/publish/inbox-delivery paths - pluto_msg_hub: `capture_snapshot/1` returns agent_id, session_id, attributes, custom_status, subscriptions, and held locks (with fencing tokens); `restore_from_snapshot/2` overlays state, splits snapshot locks into reclaimed_locks / lost_locks - Session handler builds a stepwise recovery prompt for the agent (reload CLAUDE.md, audit reclaimed locks, drain inbox, etc.) - New HTTP routes `/agents/snapshot_self` and `/agents/restore_from_snapshot` - Python client (TCP + HTTP) gains `snapshot_self`, `restore_from_snapshot`, and `save_snapshot_files` convenience methods - Two new MCP tools: `pluto_snapshot_self`, `pluto_restore_from_snapshot` - 7 new eunit integration tests in pluto_snapshot_tests.erl; full suite: 134/134 green - Docs: pluto-mcp-friend, pluto-agent-friend, tcp-connection updated with snapshot/restore sections, lock-reclaim semantics, when-not-to-use guidance, and Python + bash usage examples - Version bumped to 0.2.9 in VERSION.md, pluto.hrl, pluto.app.src, rebar.config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…toMCPFriend Both launchers now accept --restore <plut> and apply it automatically after register, so a recovered identity is in place before the wrapped LLM gets its first turn. The agent_id stored inside the .plut is authoritative — when --agent-id is omitted the launcher auto-derives it from the snapshot; when both are passed and disagree the launcher exits with a clear error rather than silently sending all snapshot locks into lost_locks. PlutoAgentFriend - bash: new --restore flag, validates the .plut, auto-derives agent_id, threads --restore through to the python wrapper - pluto_agent_friend.py: new --restore CLI arg, restore_path constructor param, _apply_restore_snapshot helper that loads the JSON and calls PlutoConnection.restore_from_snapshot post-connect - pluto_connection.py: new restore_from_snapshot(plut) method delegating to PlutoHttpClient.restore_from_snapshot PlutoMCPFriend - bash: new --restore flag, same .plut validation + auto-derive logic, threads --restore into write_mcp_json - write_mcp_json: appends --restore argv when set so .mcp.json launches the python entry with the snapshot path - pluto_mcp_friend.py: new --restore CLI arg, plumbed into PlutoMCPServer - server.py: new restore_path constructor arg + _restore_from_file_blocking helper invoked from _lifespan after register; logs reclaim/lost counts Help text and existing docs already describe the flag (added in v0.2.9). Erlang test suite: 134/134 still green. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The earlier draft overstated what the launchers do — it claimed they prepend the recovery markdown to the LLM's system prompt. That part is not yet wired; the launcher only validates the .plut, registers, calls restore_from_snapshot, and logs reclaim/lost counts. - pluto-mcp-friend.md, pluto-agent-friend.md: explicit step-by-step of what the launcher does today, plus a callout that the recovery .md is still loaded by the user / agent on its own - Both guides clarify that --agent-id is auto-derived from the .plut and that mismatches are rejected fail-fast (not silently demoted to lost_locks) - tcp-connection.md: short pointer to the launcher path so the Python API section doesn't leave the reader thinking restore is API-only Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Picks up everything from the closed v0.2.8 PR (#40 — spec contracts, watcher lifetime, stale-agent/token-leak fixes) and adds the v0.2.9 work.
New: self-snapshot + restore-from-file
Two protocol ops, end-to-end:
Wired through every layer:
Both launchers validate the `.plut` upfront, auto-derive `--agent-id` from the snapshot, and reject mismatched IDs (otherwise every snapshot lock would silently end up in `lost_locks`).
New agent status `recovered_from_file`
Distinct from `recovered` (Pluto remembered you within the grace period). Propagated through every routing/listing/broadcast/publish/inbox path so the new status is treated as "active" — messages flow normally, just visible to monitoring.
Earlier v0.2.8 work folded in
Snapshot scope (what is and isn't restored)
Recovery markdown auto-injection into the LLM's system prompt is not yet shipped — the launcher only validates, registers, and calls `restore_from_snapshot`. The agent reads the `<agent_id>-recovery.md` itself (or you paste it into turn 1).
Tests
Docs
Version
Bumped to `0.2.9` across `VERSION.md`, `pluto.hrl`, `pluto.app.src`, `rebar.config`.
Commits
Test plan
🤖 Generated with Claude Code