Skip to content

feat(keeper): retry transient RPC failures on the read path #6

Description

@0dillon

Summary

The public testnet RPC intermittently fails calls for accounts and contracts that exist. Measured
during release verification: getAccount returned Account not found three times in a row for a
funded, live account and then succeeded, all within eight seconds, and a scan on the registry
returned fetch failed and succeeded on an immediate retry. One lkeep extend --footprint run died
on it outright.

The daemon survives this — it logs, moves to the next contract, and picks the work up next tick — so
this is not a correctness bug. But a one-shot lkeep scan or lkeep extend exits non-zero on a
fault that clears in under a second, and a keeper skips a tick's worth of maintenance for the same
reason.

Acceptance Criteria

  • Read-path calls — getLedgerEntries, getAccount, simulateTransaction, registry paging —
    retry a small, documented number of times with backoff.
  • Submission is not retried. sendTransaction and pollTransaction are left alone: a
    resubmission risks paying twice, and extendViaContract already refuses to poll a hash the
    network never queued.
  • Retries are logged at warn with the attempt number, so a flapping endpoint is visible rather
    than hidden.
  • Total time spent retrying is bounded, and cannot make one contract exceed the scan interval.
  • An abort during a retry backoff stops promptly — ctx.signal is already threaded through
    runTick and must be honoured here too.
  • Tests use a fake that fails N times then succeeds; assert the call count, not just the result.

Tech Stack

TypeScript 5.7 on Node 22+ (CI runs 22 and 24), ESM with moduleResolution: "NodeNext" — relative
imports need the .js extension. @stellar/stellar-sdk is pinned to an exact version; verify RPC
API shape against the installed copy, not against documentation. Tests are Vitest under test/ and
run without a network. npm run format:check, npm run lint, npm run build and npm test must
all be clean.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: keepersrc/keeper, the daemon and its decisionscomplexity: mediumHalf a day, some design judgement neededtype: featureAdds behaviour that does not exist yet

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions