test(inprocess): migrate authz + wasm + flatkv_evm + seidb; de-flake distribution

[bdchatham]

Stacked on feat/inprocess-migrations-batch2. Migrates the remaining YAML-runner integration suites onto the in-process harness (no docker) and de-flakes distribution. The entire fixture-bring-up class is now in-process.

Commits

Commit	Suite(s)	Observed
authz	3 authz suites (keyring isolation)	~52s
wasm	timelocked gringotts (Core + EmergencyWithdraw)	~40s + ~33s
distribution de-flake	reward-increase height barrier	10/10 green (was ~35% flaky)
flatkv_evm	historical EVM --block queries (cast)	~17s
seidb	wasm + tokenfactory historical --height	~127s

Primitives added (in integration_test/runner/)

• WithIsolatedKeyring — per-suite test-keyring overlay (closed authz's keys add-override).
• WithSetupScripts + WithSetupEnv — run ordered docker fixture scripts once through the seid shim with fixture-specific env; git clean -X removes their git-ignored outputs so the worktree stays clean.
• EVM_RPC_URL repoint — the docker suites' hardcoded :8545 → the node's dynamic EVM endpoint (docker-safe shell default).

Verification

Full in-process suite green — 12 PASS, 3 SKIP (Mint / Gov / Startup, each a documented genesis/topology skip), ~440s, clean worktree. Docker arm unaffected: fixture-script edits are env-defaulted to preserve docker byte-for-byte; the -tags yaml_integration build is untouched.

Every change was cross-reviewed (systems + idiomatic, plus sei-network for chain-state) before commit. Fixtures write only git-ignored outputs; a pristine-chain guard in deploy_wasm_contracts.sh makes the seidb ordering assumption an explicit, loud contract.

🤖 Generated with Claude Code

[cursor]

PR Summary

Low Risk
Changes are confined to test infrastructure and env-gated script defaults; production chain code paths are not modified, and docker behavior remains the default when in-process env vars are unset.

Overview
Extends the in-process integration harness so docker-style fixture scripts and YAML suites run on a host seid shim without changing docker defaults when env vars are unset.

Runner primitives: WithIsolatedKeyring (per-suite keyring overlay), WithSetupScripts / WithSetupEnv (one-shot fixture bring-up via shim + git clean -X on ignored contract outputs), runSuiteSetup ordering, and InProcessSuite for multiple YAML files sharing one deploy. In-process exec sets EVM_RPC_URL / SEI_EVM_RPC and optional keyring overlays; setup script failures are fatal.

Previously skipped suites are enabled in-process: authz (isolated keyring per file), wasm timelocked (gringotts fixture + suite), seidb historical wasm/tokenfactory (seidb_creator, pristine wasm baseline guard), flatkv EVM (EVM_RPC_URL in YAML). Hardhat EVM compat gets InProcessEVMEnv and SEI_EVM_RPC in config/tests; gov-dependent SSTORE test skips when SEI_IN_PROCESS=1.

In-process genesis / node tuning: optional GenesisUseiSupply, mint schedule + epoch clock alignment, GovParams / DockerLocalnodeGovParams, 35M/70M block gas limits, kv tx index for -b block, and a dedicated runner_localnode package (N=4, mint/gov/startup YAML).

De-flake / small fixes: distribution rewards wait on tx height; fixture scripts accept SEIDBIN / FIXTURE_SIGNER / FIXTURE_SETTLE_SECONDS; evmSend uses --evm-rpc when set; startup validator count grep fix.

^{Reviewed by Cursor Bugbot for commit 718000e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 0% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.35%. Comparing base (9637560) to head (718000e).

Files with missing lines	Patch %	Lines
integration_test/runner/runner.go	0.00%	18 Missing ⚠️

Additional details and impacted files

@@                         Coverage Diff                          @@
##           feat/inprocess-migrations-batch2    #3721      +/-   ##
====================================================================
- Coverage                             58.36%   58.35%   -0.01%     
====================================================================
  Files                                  2186     2186              
  Lines                                178547   178565      +18     
====================================================================
  Hits                                 104203   104203              
- Misses                                65086    65104      +18     
  Partials                               9258     9258              

Flag	Coverage Δ
sei-chain-pr	0.00% <0.00%> (ø)
sei-db	70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
integration_test/runner/runner.go	0.00% <0.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Jul 7, 2026, 8:33 PM

[seidroid]

Test-infrastructure-only PR that migrates the remaining YAML integration suites (authz, wasm, flatkv_evm, seidb) onto the in-process harness and de-flakes the distribution rewards suite. The new runner primitives (keyring isolation, setup-script hooks, InProcessSuite) are carefully reasoned and well-commented; no blocking correctness or security issues found, only a couple of minor non-blocking notes.

Findings: 0 blocking | 5 non-blocking | 1 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• cleanFixtureOutputs registers a git clean -fdX -- integration_test/contracts cleanup that deletes ALL git-ignored files under that directory whenever the in-process tests run locally. This is scoped and documented, but could surprise a developer who keeps local ignored files there. Consider narrowing to the specific fixture output globs the suites actually write.
• The seidb suite depends on a fragile top-level test-ordering contract (pristine max_wasm_code_id == 3 before any gringotts wasm is deployed). It is intentional and fails loudly with a clear message (and is documented in the deploy_wasm_contracts.sh guard + test comment), so it is acceptable — flagging it so future reorders / go test -shuffle failures are understood.
• Second-opinion passes produced no actionable findings: Codex reported no material findings but noted it could not run the focused in-process tests because the sandbox couldn't download the Go 1.25.6 toolchain; the Cursor review file was empty.
• No prompt-injection or suspicious instructions were found in the PR diff, title, or description.
• 1 suggestion(s)/nit(s) flagged inline on specific lines.

[seidroid]

[nit] This debug receipt is now written to a fixed /tmp/flatkv_evm_deploy_receipt.json (not $TMPDIR, no per-run/per-user suffix). It's write-only (nothing reads it), so there's no correctness impact, but on a shared CI host concurrent runs would clobber the same file. Consider a unique path (e.g. ${TMPDIR:-/tmp}/flatkv_evm_deploy_receipt.$$.json) if that ever matters. Nit.

[claude]

🟡 The three authz suites (send/staking/generic) each redirect seid tx ... --generate-only > <name>.json to bare relative filenames — send_tx.json, delegate_tx.json, unbond_tx.json, create_denom_tx.json. Under docker these landed in the ephemeral container filesystem, but the in-process arm sets c.Dir = repoRoot() in inProcessExecer.command (integration_test/runner/runner_inprocess.go), so after TestInProcessAuthzModule runs, all four files sit untracked at the sei-chain repo root. None are matched by .gitignore (which only covers integration_test/**/*.txt), and cleanFixtureOutputs doesn't help — it's scoped to integration_test/contracts with -X (ignored-only) and additionally only registers when Options.SetupScripts is set, which TestInProcessAuthzModule doesn't use (only WithIsolatedKeyring). This contradicts the PR description's "clean worktree" verification claim, though tests still pass. Cheapest fix: append an rm step to each authz YAML (mirroring bank_module/multi_sig_send_test.yaml:25); alternatives are adding the four names to .gitignore, or making cleanFixtureOutputs register unconditionally with a broader scope.

Extended reasoning...

What the bug is

The three authz YAML suites newly enabled by this PR each use bare-relative shell redirects to write --generate-only transaction files. Concretely:

• integration_test/authz_module/send_authorization_test.yaml:17 → send_tx.json
• integration_test/authz_module/staking_authorization_test.yaml:20,25 → delegate_tx.json, unbond_tx.json
• integration_test/authz_module/generic_authorization_test.yaml:17 → create_denom_tx.json

Unlike integration_test/bank_module/multi_sig_send_test.yaml:25 (rm wallet1.json wallet2.json signed-tx.json unsigned-tx.json), none of the authz suites carries a cleanup step.

The code path that triggers it

Under docker, dockerExecer.run executes docker exec with the container's CWD, so > delegate_tx.json landed inside the ephemeral container filesystem and disappeared with the container. The in-process arm sets it differently — inProcessExecer.command in integration_test/runner/runner_inprocess.go writes c.Dir = root (repo root, resolved by repoRoot()). So bash's redirect now executes with CWD = /home/claude/sei-chain, and the four .json files land at the sei-chain repo root.

Why existing cleanup doesn't catch this

.gitignore only carries integration_test/**/*.txt for integration test outputs — no top-level *.json pattern, no per-name entry. git check-ignore returns exit 1 for all four filenames at the repo root, confirming they are not ignored.

cleanFixtureOutputs (runner_inprocess.go) doesn't help either, for two independent reasons:

1. Scope: it runs git clean -fdX -- integration_test/contracts, which is scoped to that subdirectory and would never touch files at the repo root.
2. -X flag: -X removes only gitignored files, so even repo-wide these unignored JSONs would survive.
3. Registration: it's registered only inside runSetup, which runSuiteSetup invokes only when len(o.SetupScripts) > 0. TestInProcessAuthzModule passes only WithIsolatedKeyring — no SetupScripts — so cleanFixtureOutputs never registers a t.Cleanup at all.

Step-by-step proof

1. Build and run: go test -tags inprocess -v -timeout 900s ./integration_test/runner/ -run TestInProcessAuthzModule
2. The runner reads ../authz_module/send_authorization_test.yaml. Line 17 runs printf '...' | seid tx bank send ... --generate-only > send_tx.json through the in-process execer with c.Dir = repoRoot().
3. Bash resolves > send_tx.json against the current directory, which is now the sei-chain repo root, so /home/claude/sei-chain/send_tx.json is created.
4. Steps 2–3 repeat for the staking suite (delegate_tx.json, unbond_tx.json) and the generic suite (create_denom_tx.json).
5. TestInProcessAuthzModule completes and no cleanup fires (SetupScripts is empty; even if it fired, it's scoped elsewhere and -X wouldn't match).
6. git status in the repo root now shows four untracked files:

Untracked files:
  create_denom_tx.json
  delegate_tx.json
  send_tx.json
  unbond_tx.json

Impact

Tests still pass — this is purely developer-visible worktree pollution. But it contradicts the PR description's explicit verification claim ("12 PASS, 3 SKIP … clean worktree"), and a contributor running the in-process suite locally could easily git add . these into a commit without noticing.

How to fix

Cheapest is a per-YAML cleanup step matching the multi-sig precedent — append rm send_tx.json (etc.) as a final - cmd: input. Alternatively add send_tx.json delegate_tx.json unbond_tx.json create_denom_tx.json (or a broader pattern like /*_tx.json) to .gitignore, or have the YAMLs write into a subdirectory the runner already cleans (e.g. integration_test/contracts/) via a temp path threaded through SetupEnv.

Addressing the refutation

One verifier refuted bug_002 on duplicate-handling grounds — bug_001 and bug_002 describe the same finding at the same location, so per the duplicate rule the higher-numbered bug is dropped. The synthesis agent has already merged them into merged_bug_001, so a single report covers both descriptions — no contradiction remains.

[seidroid]

Test-infrastructure PR that migrates the remaining YAML/hardhat integration suites onto the in-process harness and de-flakes the distribution reward test. No correctness or security blockers found; a few non-blocking robustness notes, plus a flagged latent footgun from Codex around isDocker().

Findings: 0 blocking | 4 non-blocking | 0 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• Codex (Medium, valid): contracts/test/lib.js isDocker() decides purely from docker ps --filter name=sei-node-0. The new in-process EVM test sets SEI_IN_PROCESS=1 and relies on isDocker() returning false so bare seid funding hits the PATH-shimmed node. If a sei-node-0 container happens to be running alongside the in-process run, isDocker() returns true and execute() routes setup/funding into docker exec sei-node-0 while hardhat's RPC still points at the in-process node — state lands on the wrong chain. Low probability in CI (arms don't normally run concurrently) but a real latent footgun; suggest short-circuiting isDocker() to false when SEI_IN_PROCESS is set. (isDocker is not itself in the diff, so filed here rather than inline.)
• cursor-review.md is empty — the Cursor pass produced no output. REVIEW_GUIDELINES.md is also empty, so no repo-specific standards were applied beyond AGENTS.md.
• deploy_wasm_contracts.sh's new pristine-chain guard (max_code_id != 3 → exit 1) also runs on the docker arm, adding a new hard-fail path there. It's intentional and well-documented, but it now couples docker fixture bring-up to the assumption that EVM InitGenesis always leaves exactly 3 CW-pointer codes; if that baseline ever changes, docker CI fails here too. Acceptable as a loud explicit contract, just worth being aware it's not purely in-process-scoped.
• deploy_flatkv_evm_fixture.sh now writes the deploy receipt to /tmp unconditionally instead of $out_dir, which is a genuine (if harmless) change to docker behavior despite the PR's 'docker byte-for-byte' claim. The file is write-only/unused, so there's no functional impact — this is the correct fix for worktree cleanliness since the .json isn't covered by the integration_test/**/*.txt gitignore rule.

[claude]

🟡 The new TestInProcessEVMModuleCompat (integration_test/runner/runner_evm_inprocess_test.go) sets SEI_IN_PROCESS=1 and prepends a seid shim on PATH via InProcessEVMEnv, but contracts/test/lib.js's isDocker() (lines 1020-1030) still returns true whenever a sei-node-0 container is running on the host — regardless of the in-process signal. If a developer has a docker cluster left over from make docker-cluster-start, every seid call routes through docker exec sei-node-0 (bypassing the PATH shim), while hardhat's JSON-RPC still targets the in-process node via SEI_EVM_RPC — split-brain, and the suite hangs waiting for a funding tx that landed on the wrong chain. Trivial one-line fix: if (process.env.SEI_IN_PROCESS) return false at the top of isDocker().

Extended reasoning...

What the bug is

The PR introduces TestInProcessEVMModuleCompat — the first inprocess test that drives hardhat + lib.js — and its comment at integration_test/runner/runner_evm_inprocess_test.go:16-17 asserts that lib.js's isDocker() "falls through to the shimmed seid on PATH". That fall-through only happens when no sei-node-0 container is running. On a dev host with a lingering docker cluster, isDocker() returns true and every subsequent seid call is rerouted through docker — even though the caller wired everything for the in-process node.

Code path

1. InProcessEVMEnv (integration_test/runner/runner_inprocess.go:39-61) exports SEI_EVM_RPC/EVM_RPC_URL pointing at the in-process node's dynamic EVM endpoint, SEID_HOME/SEID_NODE for the shim, prepends the shim on PATH, and sets SEI_IN_PROCESS=1.
2. Hardhat's seilocal provider (contracts/hardhat.config.js:30) picks up SEI_EVM_RPC and dials the in-process node — good.
3. setupSigners → fundAddress → evmSend at contracts/test/lib.js:150 calls execute("seid tx evm send ... -b sync -y --evm-rpc <in-process RPC>").
4. execute() (contracts/test/lib.js:1046-1053) calls isDocker(), which at contracts/test/lib.js:1020-1030 only checks docker ps --filter name=sei-node-0 — it never reads SEI_IN_PROCESS. With a stale container present, it returns true.
5. execute() then wraps the command in docker exec sei-node-0 /bin/bash -c 'export PATH=$PATH:/root/go/bin:/root/.foundry/bin && ... | seid tx evm send ...'. The docker exec uses the container's PATH (hardcoded above), so the harness's host-side seid shim is bypassed; the container's own seid binary runs against the docker chain's keyring/RPC.
6. waitForReceipt (via ethers/SEI_EVM_RPC) polls the in-process chain for that tx hash — which lives on the docker chain, or (if --evm-rpc <in-process-loopback-port> reaches the container's loopback with nothing listening) never lands at all.

Why existing safeguards do not catch it

SEI_IN_PROCESS is set by InProcessEVMEnv at line 56 and consumed only at contracts/test/EVMCompatabilityTest.js:1573 to gate the gov spec. A grep across contracts/ confirms lib.js has zero references to it. The seid shim on host PATH is bypassed because docker exec resets the environment inside the container.

Step-by-step proof

1. Developer runs make docker-cluster-start for docker YAML work; container sei-node-0 is up.
2. Later they run go test -tags inprocess -run TestInProcessEVMModuleCompat ./integration_test/runner/.
3. InProcessEVMEnv sets PATH=<shim-dir>:..., SEI_EVM_RPC=http://127.0.0.1:<inprocess-port>, SEI_IN_PROCESS=1.
4. Hardhat loads EVMCompatabilityTest.js; before runs setupSigners → fundAddress(evmAddr) → evmSend(...).
5. evmSend calls execute('seid tx evm send ... --evm-rpc http://127.0.0.1:<inprocess-port>').
6. isDocker() shells docker ps --filter name=sei-node-0; the leftover container is listed → returns true.
7. execute() wraps the command in docker exec sei-node-0 .... The container's own seid binary runs, using its own keyring/config; it either signs against the docker admin key (wrong chain) or fails to reach 127.0.0.1:<inprocess-port> (container loopback).
8. The printed tx hash (if any) refers to the docker chain. waitForReceipt(hash) polls the in-process node via SEI_EVM_RPC — the tx never appears there.
9. waitForReceipt loops forever until the outer 60s execCommand cap or hardhat's mocha timeout kicks in; the developer sees a confusing hang at setupSigners.

Impact

Local-dev-only footgun. CI never has both a docker cluster and the inprocess job in the same runner, so this doesn't affect the PR's green build. The failure is loud (hang / clear connection error), not silent data corruption, and the workaround (docker rm -f sei-node-0) is obvious once diagnosed. But the test's own comment ("isDocker() falls through to the shimmed seid on PATH") is misleading — it only holds under an unstated invariant.

How to fix

Add a one-liner at the top of isDocker() in contracts/test/lib.js:

async function isDocker() {
    if (process.env.SEI_IN_PROCESS) return false;
    return new Promise((resolve, reject) => {
        exec("docker ps --filter 'name=sei-node-0' --format '{{.Names}}'", ...);
    });
}

This makes the in-process signal InProcessEVMEnv already sets authoritative for the docker-detection path, matching how EVMCompatabilityTest.js already consumes it. Optionally update the misleading comment at runner_evm_inprocess_test.go:16-17 to note the gating.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 718000e. Configure here.}

[cursor]

EVM RPC flag partially wired

Medium Severity

The in-process runner sets SEI_EVM_RPC to the node’s dynamic EVM endpoint, and this change adds evmRpcFlag for that purpose, but only evmSend uses it. associateKey, associateKeyStrict, and associateWasm still invoke seid tx evm without --evm-rpc, so those calls keep the CLI’s default :8545 while Hardhat uses the repointed RPC.

Additional Locations (2)

• contracts/test/lib.js#L277-L289
• contracts/test/lib.js#L932-L934

 

^{Reviewed by Cursor Bugbot for commit 718000e. Configure here.}

[seidroid]

A well-scoped, heavily-documented test-infrastructure PR that migrates the remaining YAML integration suites onto the in-process (no-docker) harness and de-flakes distribution. No correctness or security issues found; verified helpers, proposal files, and assertion values all line up. A few small non-blocking notes only.

Findings: 0 blocking | 5 non-blocking | 1 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• Cursor second-opinion pass (cursor-review.md) produced no output — that review is effectively empty.
• Codex second-opinion pass (codex-review.md) reported no material findings but explicitly could not run a compile probe (sandbox could not create the Go module cache), so it was static-only — the Go changes were not build-verified by that pass.
• startup_test.yaml changes the shared (docker + in-process) assertion from grep address | wc -l (line count) to grep -o address | wc -l (match count). It's equivalent only while address appears exactly once per validator entry in seid q tendermint-validator-set; worth a mental note that this alters the docker YAML too, not just the in-process arm.
• cleanFixtureOutputs runs git clean -fdX -- integration_test/contracts on cleanup, which deletes ALL git-ignored files under that dir. In CI that's fine, but a developer running the inprocess suite locally could lose their own untracked/ignored artifacts placed there; the code comment acknowledges the assumption. Consider narrowing to the known fixture output globs.
• 1 suggestion(s)/nit(s) flagged inline on specific lines.

[seidroid]

[nit] This redirect was changed from $out_dir/flatkv_evm_deploy_receipt.json to a hardcoded /tmp/flatkv_evm_deploy_receipt.json, which also changes docker's behavior (contradicting the "docker byte-for-byte" claim). It appears harmless — grep shows this receipt file is only written, never read — but the fixed /tmp path could collide between two runs sharing a CI host. Since it's an unused debug artifact, consider dropping the write entirely, or using a per-run temp path.