Repository navigation
Local first + drills on a copy of your network - #15
Merged
Merged
Conversation
…ed nodes
32 of 68 incident cases now run without the model (was 5):
- fixes: mempool type for the node's version, DB backend from the data on
disk, gRPC disabled, network detached (recreate), no/low peers (the
chain's other nodes), home permissions, NTP, EVM drift/indexer, catching
up, jail waiting period, signer state ahead, restarted recently
- reports (cometcli gathers, the operator decides): votes due/urgent, fee
balance, active set, self-delegation, disk, memory/load, upgrade
imminent, upgrade binary missing
Judgement cases (parse errors, app hash, double sign, corruption) stay
with the model.
Playbook engine: when conditions joined with &&; step arguments from
signals ({sig:…}, a missing one hands off to the model) and the chain's
other nodes ({peers}); playbook_kind report ends with the decisions left
to the operator.
New tools: node.set-config (one key in place, comments kept, parse-
checked, .cometcli-bak) and node.recreate (compose up -d --force-recreate
from the container's own compose file). Node files are read and written
through the container's bind mount, so a crashed docker node can be
fixed; net.add-peer uses that too and takes several peers.
New signals: config.mempool_want (from the running version, else the
image), data.db_backend (from the files in data/application.db),
node.host_home, proc.image_version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, route stats
- direct commands run without the model: restart/stop/start, recreate,
unjail, vote, send, delegate, withdraw, logs, add peer, set config,
build, switch, check the network, triage — the words are the tool's
arguments; approvals as usual; a word that isn't one of your nodes
('stop worrying') goes to the model
- known questions: health (from triage), fleet status, version, upgrade
readiness, recent log errors, spec conformance, lag
- learning: when the model works a known case that has no playbook, its
working steps are offered as the playbook (saved to ~/.cometcli/kb, so
the next identical incident runs locally)
- cometcli route stats: share answered locally, top intents/cases, and
what the model keeps being asked; drills report 'without the model'
- every prompt's audit record says where it went and why
- fix playbooks that involve downtime end with wait/unjail/verify when
the validator was jailed
- mempool rule corrected: an app-side mempool (max-txs >= 0) needs type
'app'; 'app' with -1 is valid — a stopped node was misdiagnosed
- node.logs level=error/info actually filters (it was ignored)
Drills (cosmos/evm v0.7.2): 5/5 fixed, 4/5 without the model (the broken
config file needs judgement). On the drill session, 15 of 17 prompts were
answered locally.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- drill testnet up --spec <network> builds the drill network from the network's spec with genesis create's own steps (keys on each node, gentxs, collect, strict validation, configs rendered for the role), under <id>-drill, with drill timing (20-block window, 60s jail, 60s votes) and gRPC/JSON-RPC on; network check runs against it; down removes the drill's spec and genesis too (only ever a -drill chain's) - genesis create's workflow is networktool.CreateNetwork, shared by both; genesis validators' profiles now get their valoper and node-keyring signer set - spec review warns on a jail under a minute (SDK v0.54 rejects it) Found by the first spec-built drill run (4/5 fixed, 2/5 local): - a crash-looping node loses its peers: the 20 crash causes now explain node-no-peers/node-low-peers, so an OOM loop runs its playbook instead of going to the model as a two-root judgement call (614k tokens -> 0) - halt-height drill: the margin is ~20s of blocks (8 blocks was gone before the restart on a 0.2s chain), and a node exiting in a loop counts as halted - drills count container restarts from injection, not docker's lifetime count (a stale count made oom look ready before it was) Drills on cometcli-gen-1's spec (primium-evm v0.7.2, 0.2s blocks): triage 5/5, fixed 5/5, 4/5 without the model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abhijitkrm
changed the title
abhijitkrm
added this pull request to stack #16
October 10, 2026 09:27
…al-first # Conflicts: # README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal: the model is called only when cometcli can't work out what to do itself.
Runs locally now
{sig:…},{peers}) and steps can be conditional (when: a && b).playbook_kind: report): they gather the facts and leave the decision to the operator (gov votes, disk, fee balance, active set, upgrades).Learning: when the model handles a known case that has no playbook, its working steps are offered as one, saved to
~/.cometcli/kb/<case>.yaml. The next identical incident then runs locally.Measurement
cometcli route stats [--since 7d]: share of prompts answered locally, top intents and cases, and what still goes to the model.New tools:
node.set-config(in-place TOML edit, parse check, diff approval, backup) andnode.recreate(compose force-recreate). Node files are read and written through the bind mount, so a stopped container works too.Fixes found along the way
max-txs >= 0) needstype = "app", butappwith-1is valid. A stopped node was being misdiagnosed as a mempool mismatch.node.logs level=error|infoactually filters now (it was ignored).Results
Drills on a copy of your network (
drill testnet up --spec <network>)networktool.CreateNetwork).<id>-drill.network checkpasses on it;downalso removes the drill's spec and genesis.Stacked on #14.
🤖 Generated with Claude Code