Skip to content

Hunt for flaky tests weekly, with every run replayable by its seed - #95

Merged
HackTuah merged 1 commit into
mainfrom
ci/flake-hunt
Oct 20, 2026
Merged

HackTuah merged 1 commit into
mainfrom
ci/flake-hunt

Conversation

@HackTuah

Copy link
Copy Markdown
Member

CI and tooling, plus one census test and a CONTRIBUTING paragraph. Nothing a consumer of the package can hit changes.

Why. A test that fails one run in tens is invisible to a gate that runs each leg once per push. The flaky tests fixed so far were each found as one red leg that a rerun turned green. ExUnit, as this project runs it, prints no seed, so a failure seen once could not be replayed. A contributor has also reported tests failing on an unmodified main that a hundred local runs here did not reproduce.

What changes

  • tools/flake_hunt.sh <runs> runs the suite <runs> times, each in a fresh VM with a seed of its own. It does not use --repeat-until-failure, which reruns in one VM, where state left by one run can fail the next. A red run prints its failed tests and the command that replays it, and its output is kept. A run that exits green with no summary counts as failed. ERL_FLAGS passes through.
  • tools/probe_flake_hunt.sh runs the script against a fake mix: each run gets its own seed; one red run among greens fails the hunt; a silent run fails it; a bad count is refused.
  • .github/workflows/flake-hunt.yml runs weekly and by hand, on the gate's four pairs and images. Each leg hunts at the runner's schedulers and then at two (+S 2). It never runs on a push or a pull request and is not a required check: a flake it finds is a finding to fix, not a blocked merge.
  • governance_test.exs: the hunt's matrix must equal the gate's, and the hunt must have no push or pull-request trigger.
  • CONTRIBUTING.md: what to report when a test fails only sometimes (its name and the seed), and the tool that gives both.

Evidence

  • The probe fails before the script exists and passes after. Six mutants of the script each fail it: no seed passed, one seed for every run, a silent green counted as a pass, red output dropped, failures ignored, zero runs accepted.
  • On the real suite with a planted failing test, the script names the test and its replay command and exits 1. Two runs on a clean tree at +S 2 exit 0.
  • The census fails without the workflow, with the box leg's image changed in the hunt only, and with a pull_request trigger added.

Not yet measured

  • The hunt on CI. Its first run is a manual dispatch after this merges, since a workflow can be dispatched only from the default branch.

A test that fails one run in tens is invisible to a gate that runs each
leg once per push. The flaky tests fixed here so far were each found as
one red leg that a rerun turned green, and ExUnit, as this project runs
it, prints no seed, so a failure seen once could not be replayed.

- tools/flake_hunt.sh <runs>: the suite <runs> times, each in a fresh
  VM with a seed of its own. A red run prints its failed tests and the
  command that replays it, and its output is kept. A run that exits
  green with no summary counts as failed. ERL_FLAGS passes through.
- tools/probe_flake_hunt.sh: the script against a fake mix (each run
  its own seed, one red run among greens, a silent run, a bad count).
- .github/workflows/flake-hunt.yml: weekly and by hand, on the gate's
  four pairs and images; each leg hunts at the runner's schedulers and
  at two. Not run on push or pull request, and not a required check.
- governance_test.exs: the hunt's matrix equals the gate's, and the
  hunt has no push or pull_request trigger.
- CONTRIBUTING.md: what to report when a test fails only sometimes.

Measured locally: the probe fails before the script exists and passes
after; six mutants of the script each fail it. The script on the real
suite with a planted failing test names the test and its replay
(exit 1); two runs on a clean tree at +S 2 exit 0. The census fails
without the workflow, with the box leg's image changed in the hunt
only, and with a pull_request trigger added.

Signed-off-by: Ayla Croft <aylacroft@proton.me>
@HackTuah
HackTuah merged commit 9a24a83 into main Oct 20, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant