Repository navigation
Hunt for flaky tests weekly, with every run replayable by its seed - #95
Merged
Merged
Conversation
A test that fails one run in tens is invisible to a gate that runs each leg once per push. The flaky tests fixed here so far were each found as one red leg that a rerun turned green, and ExUnit, as this project runs it, prints no seed, so a failure seen once could not be replayed. - tools/flake_hunt.sh <runs>: the suite <runs> times, each in a fresh VM with a seed of its own. A red run prints its failed tests and the command that replays it, and its output is kept. A run that exits green with no summary counts as failed. ERL_FLAGS passes through. - tools/probe_flake_hunt.sh: the script against a fake mix (each run its own seed, one red run among greens, a silent run, a bad count). - .github/workflows/flake-hunt.yml: weekly and by hand, on the gate's four pairs and images; each leg hunts at the runner's schedulers and at two. Not run on push or pull request, and not a required check. - governance_test.exs: the hunt's matrix equals the gate's, and the hunt has no push or pull_request trigger. - CONTRIBUTING.md: what to report when a test fails only sometimes. Measured locally: the probe fails before the script exists and passes after; six mutants of the script each fail it. The script on the real suite with a planted failing test names the test and its replay (exit 1); two runs on a clean tree at +S 2 exit 0. The census fails without the workflow, with the box leg's image changed in the hunt only, and with a pull_request trigger added. Signed-off-by: Ayla Croft <aylacroft@proton.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI and tooling, plus one census test and a CONTRIBUTING paragraph. Nothing a consumer of the package can hit changes.
Why. A test that fails one run in tens is invisible to a gate that runs each leg once per push. The flaky tests fixed so far were each found as one red leg that a rerun turned green. ExUnit, as this project runs it, prints no seed, so a failure seen once could not be replayed. A contributor has also reported tests failing on an unmodified
mainthat a hundred local runs here did not reproduce.What changes
tools/flake_hunt.sh <runs>runs the suite<runs>times, each in a fresh VM with a seed of its own. It does not use--repeat-until-failure, which reruns in one VM, where state left by one run can fail the next. A red run prints its failed tests and the command that replays it, and its output is kept. A run that exits green with no summary counts as failed.ERL_FLAGSpasses through.tools/probe_flake_hunt.shruns the script against a fakemix: each run gets its own seed; one red run among greens fails the hunt; a silent run fails it; a bad count is refused..github/workflows/flake-hunt.ymlruns weekly and by hand, on the gate's four pairs and images. Each leg hunts at the runner's schedulers and then at two (+S 2). It never runs on a push or a pull request and is not a required check: a flake it finds is a finding to fix, not a blocked merge.governance_test.exs: the hunt's matrix must equal the gate's, and the hunt must have no push or pull-request trigger.CONTRIBUTING.md: what to report when a test fails only sometimes (its name and the seed), and the tool that gives both.Evidence
+S 2exit 0.pull_requesttrigger added.Not yet measured