Skip to content

fix: let anvil finish loading its state - #91

Merged
naps62 merged 2 commits into
mainfrom
fix/anvil-slow-boot
Aug 26, 2026
Merged

fix: let anvil finish loading its state#91
naps62 merged 2 commits into
mainfrom
fix/anvil-slow-boot

Conversation

@naps62

@naps62 naps62 commented Aug 26, 2026

Copy link
Copy Markdown
Member

angus3 could not start: every request killed the boot after 10s and returned "Stack is taking too long to start". Anvil replays its persisted state before it answers anything, and with state_interval plus preserved historical states that file grows past what a 10s readiness cap allows. #90 turned that cap into a kill, so the stack could never come back.

  • A boot that is still loading is left running; the stack reports starting and later requests poll it instead of killing it or starting a second anvil.
  • Readiness polling watches the process, so a boot that exits reports its status immediately instead of after the full readiness budget.
  • The readiness request has a timeout. It defaulted to infinity, so an anvil that accepted the connection and then went quiet parked the whole GenServer.
  • Suspend gives anvil 30s to finish its state dump instead of muontrap's 500ms. A SIGKILL landing mid-dump leaves a truncated state.json that anvil then refuses to parse, exiting 2.
  • A state.json anvil cannot parse is moved to state.json.corrupt and the stack boots clean instead of staying down.
  • Each boot claims a fresh port, returned to the pool only once anvil has released it, and HttpPorts checks a port binds before handing it out. Reusing a port raced anvil's shutdown, and a stale process answering the probe read as a healthy boot.
Verification

mix test test/ethui/services/anvil_test.exs — 10 tests, 0 failures, including new coverage for the still-loading path, the unreadable state file, and serving again after a suspend.

The full suite is flaky on this machine (docker is unavailable, so the graph and ipfs services flap). Measured over two runs each in the same environment: main 11 and 12 failures, this branch 15 and 1. The websocket proxy test fails on both.

🤖 Generated with Claude Code

naps62 and others added 2 commits August 26, 2026 15:40
#90 kills a boot that has not answered after 10s. Anvil replays its
persisted state before it answers anything, and with --state-interval
and preserved historical states that runs past 10s on a busy chain, so
a stack whose state file grew large could no longer start: each request
killed the load half way through and returned "Stack is taking too long
to start".

- a boot that is still loading is left alone and the stack reports
  :starting; later requests poll it instead of starting a second anvil
- readiness polling watches the process, so a boot that exits reports
  its status immediately instead of after the full readiness budget
- the readiness request has a timeout; it defaulted to infinity, which
  parked the whole GenServer whenever anvil accepted a connection and
  then did not answer
- suspend gives anvil 30s to finish its state dump instead of
  muontrap's 500ms; a SIGKILL landing mid-dump leaves a truncated
  state.json that anvil then refuses to parse, exiting 2
- a state.json anvil cannot parse is moved aside so the stack boots
  clean instead of staying down
- suspend waits for the port to be released before a resume rebinds it

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Holding one port per stack meant every resume raced the previous
anvil's shutdown: it keeps the socket while it dumps state, so the
boot either failed to bind or, worse, looked healthy because the old
process answered the readiness probe.

- each boot claims a fresh port and hands the old one back only once
  anvil has actually let go of it, from outside the GenServer so a
  shutdown never blocks requests to the stack
- HttpPorts checks a port is bindable before handing it out, so a
  process that outlived its stack cannot poison a claim
- the EXIT handler only reacts to anvil's own exit; it treated any
  linked process's exit as the chain dying
- destroy waits for anvil instead of dropping its port immediately

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 26, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
stacks Ready Ready Preview Aug 26, 2026 3:14pm

Request Review

@naps62
naps62 merged commit 9a943ec into main Aug 26, 2026
2 checks passed
@naps62
naps62 deleted the fix/anvil-slow-boot branch August 26, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant