Repository navigation
Run the full Python suite in CI, turn hypothesis contract v3 on by default, and add a grounding replay standard - #19
Merged
Conversation
added 3 commits
October 23, 2026 17:55
… default
Full Python suite in CI
- The PR check ran no Python tests; the full suite (over 5,500 tests, more than four hours
on one machine) was run locally before each merge. CI now runs it on every pull request,
on main and nightly (18:00 UTC), in nine parallel Windows jobs:
- python-fast: every test outside tests/integration.
- python-integration: tests/integration split into eight shards by measured file duration.
The shard script fails the job unless the shards cover every integration test file
exactly once; new files without a measured time go to the lightest shard.
- python-summary: fails on a missing, empty, damaged or duplicated JUnit report, any test
failure or error, or a failed, cancelled or skipped job.
- Same pytest options and test selection as the local full run. No test, expectation or
time limit changed. Manual dispatch is also available.
Hypothesis contract v3 on by default
- Projects with no saved choice now use the v3 contract (expected result per hypothesis for
each test, and refutation conditions). A saved False keeps the v2 contract, and saved
settings are not rewritten. Live comparisons on two synthetic datasets passed the fixed
acceptance criteria in all recent runs; predictions remain labelled as not confirmed by
evidence. README and KNOWN_LIMITATIONS describe the default and the extra model call.
Verification: shard and summary tool tests (24); collected node IDs across shards match the
full collection (5,613 tests, no overlap); hypothesis contract, settings and research
request tests and web switch/settings tests pass; actionlint, ruff, types, lint, budget and
manifest checks pass. This PR's CI run is the first full-suite run of the new jobs.
On GitHub runners GITHUB_ACTIONS makes Typer force terminal mode and ignore COLUMNS=120. The contract test and capture script now remove GITHUB_ACTIONS, FORCE_COLOR, PY_COLORS and TERMINAL_WIDTH before rendering help. The recorded contract is unchanged.
…ing replay tests Scripted research models answer the v3 contract and table filler role; the CLI help test ignores CI terminal flags; integration tests run in sixteen shards; model-free prediction grounding replay data and scorer added.
hyperdrivebeep
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full Python suite in CI
and nightly (18:00 UTC) in parallel Windows jobs: python-fast (everything outside
tests/integration) and python-integration split into sixteen shards by measured file
duration. The shard script fails the job unless the shards cover every integration file
exactly once. python-summary fails on a missing, empty, damaged or duplicated JUnit report,
any failure or error, or a failed, cancelled or skipped job.
changed. Eight shards took 47 to 73 minutes on hosted runners, so sixteen are used.
TERMINAL_WIDTH before rendering help, since GITHUB_ACTIONS makes Typer ignore COLUMNS.
Hypothesis contract v3 on by default
each test, and refutation conditions). A saved False keeps v2; saved settings are not
rewritten. README and KNOWN_LIMITATIONS describe the default and the extra model call.
only supplied context. Product code is unchanged by this test update.
Prediction grounding replay standard (tests only)
restatements of recorded observations must link the right source span, hypothesis-only
predictions may have no evidence, and false verification, out-of-scope or invented links and
synthetic-as-measured labels are rejected. No product behaviour changes.
Verification: previous CI runs of this PR ran all 5,613 Python tests; their failures (CLI help
under GITHUB_ACTIONS, scripted models without the table filler role) are fixed here and pass
locally. One research-recovery timeout seen once in CI passed locally three times and is not
changed. This push is the first run with sixteen integration shards.