Skip to content

Preserve duplicate collected items in grouped scheduling - #1383

Open
AdemVessell wants to merge 1 commit into
pytest-dev:masterfrom
AdemVessell:fix/grouped-collection-occurrences
Open

AdemVessell wants to merge 1 commit into
pytest-dev:masterfrom
AdemVessell:fix/grouped-collection-occurrences

Conversation

@AdemVessell

@AdemVessell AdemVessell commented Sep 14, 2026 •

Copy link
Copy Markdown

Summary

Fixes #1212.

The grouped schedulers use nodeids as work-unit keys, so duplicate collected
items collapse into one entry. Looking those names up with list.index() also
always selects the first occurrence and repeatedly scans the collection.

This change keeps collection indices as work-unit keys through dispatch,
completion and worker replacement. Nodeids still determine the groups and the
text of crash reports. The worker protocol is unchanged.

Reproduction

Given test_repeat.py:

def test_selected_occurrence():
    pass

Run:

python -m pytest --keep-duplicates test_repeat.py test_repeat.py -n2 --dist=loadscope

On the current base, both workers collect two items, but only one test runs and
the command exits successfully. With this patch, both occurrences run.
loadfile and loadgroup have the same problem; ordinary pytest and load
already execute both occurrences.

Tests

  • Nine new regression cases across the three schedulers: duplicate dispatch,
    separate completion, unfinished work after worker replacement, and repeated
    file selection through the normal CLI. They fail before the production fix
    and pass after it.
  • Python 3.10.20 / pytest 7.0.0: all 25 selected regression, grouping and restart
    tests pass.
  • Python 3.12.13 / pytest 9.1.1: the nine new tests pass; mypy checks all 27
    source/test files; changed Python files pass Ruff lint and formatting.

There is one full-suite caveat. Two runs passed 231 tests with 6 skips and 10
expected failures. The final run had 230 passes and one failure in
test_keyboardinterrupt_hooks_issue79: debug logging in the worker receiver
thread raised OSError: [Errno 9] Bad file descriptor, followed by a missing
workeroutput error. That test uses the unchanged default load scheduler.
A fixed 12-round comparison reproduced the same failure on unmodified source
(1/12 failures; patched 0/12). I have not changed or suppressed that test, and
those counts are not evidence that this patch changes shutdown reliability.

These checks were local on macOS ARM64, not a complete platform matrix.
Prepared with AI assistance; no independent human review is claimed.

CI follow-up: all 18 released-pytest test jobs passed on Linux and Windows.
The two py311-pytestmain jobs failed five existing TestGroupScope cases.
I reproduced those same five failures on both unmodified xdist at eba6a447
and this patch, using pytest commit aceb21ab2 locally on macOS ARM64.
That pytest revision reads nodeid from _id, while xdist's unchanged group
hook writes _nodeid. I have left that separate compatibility issue out of
this patch. CI results.

Compatibility note

workqueue and assigned_work now store collection-index keys within each
group instead of nodeid keys. Subclasses that manipulate those dictionaries
directly may need adjustment. Public hook signatures and the wire protocol
are unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance issue with LoadScopeScheduling when a node has a large number of tests

1 participant