Skip to content

Support Python 3.15 and free-threaded builds - #170

Open
jameshilliard wants to merge 1 commit into
scrapinghub:masterfrom
jameshilliard:python315-freethreading
Open

jameshilliard wants to merge 1 commit into
scrapinghub:masterfrom
jameshilliard:python315-freethreading

Conversation

@jameshilliard

@jameshilliard jameshilliard commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #28.

Adds Python 3.15 and 3.15t wheel builds and CI coverage, adds 3.14t to the test matrix, and declares the Cython extension compatible with free-threading. Per-instance locks protect native state, including input setup and decoding within a shared Tagger.tag() call. Training callbacks can inspect their trainer but cannot mutate active training state. Model dumps avoid closing file descriptors that another thread may have reused.

Free-threaded builds route CRFsuite's shuffle calls through a mutex around libc's rand(). This preserves the platform's generator and sequential shuffle sequence while protecting concurrent trainers on platforms such as macOS and musl, whose rand() implementations update shared state without locking. Native sources compile normally, and the CRFsuite submodule is unchanged. ItemSequence retains its lock because calling __init__() on an existing instance replaces its native vector; a regression test covers concurrent reinitialization and reads.

Committed regression tests cover shared-tagger calls from multiple threads, callback mutation rejection, concurrent ItemSequence reinitialization, closed-tagger ValueError behavior, and import without enabling the GIL. The Cython minimum is reduced to 3.1 after verifying builds with that version. Setuptools 75.7 remains the minimum because it defines Py_GIL_DISABLED for free-threaded Windows extension builds (upstream change).

Validated on macOS ARM64 using wheels built from the source distribution with Cython 3.1.0 and setuptools 75.7.0: Python 3.14t and 3.15t each passed 80 tests with 5 existing expected failures; Python 3.10 and 3.15 each passed 78 tests with 2 skips and 5 expected failures. These totals include additional temporary stress tests. A temporary native RNG probe verified that serialized and concurrent wrapper calls consume the same libc RNG sequence. Seeded smoke tests produced identical model files on 3.15 and 3.15t for L-BFGS, L2-SGD, AP, PA, and AROW.

@jameshilliard
jameshilliard force-pushed the python315-freethreading branch 3 times, most recently from 423bbca to 16268ff Compare October 21, 2026 18:07
@jameshilliard
jameshilliard marked this pull request as ready for review October 21, 2026 18:26
@AdrianAtZyte

Copy link
Copy Markdown
Member
  • Please, commit the concurrency tests, at least a threaded tag() on a shared Tagger and a test for the new RuntimeError when a callback mutates the trainer during train().
  • I am not sure about the rand override. It makes free-threaded builds shuffle differently from regular builds, so the same data trains a different model on 3.15t than on 3.15. glibc rand() is already locked and MSVC’s is per-thread; what problem did you hit without it?
  • ItemSequence is not mutated after __init__, so I think its lock can go, which also lets items() iterate in place again.
  • cibuildwheel already builds cp314t wheels, which this now marks as free-threading compatible, so I think 3.14t should be in the test matrix too.
  • Why cython>=3.3.0 and setuptools>=75.7? freethreading_compatible is available since Cython 3.1.

@jameshilliard
jameshilliard force-pushed the python315-freethreading branch from 16268ff to 95db3d9 Compare October 22, 2026 15:51
@AdrianAtZyte

Copy link
Copy Markdown
Member

Please, drop the set() override in tagger_wrapper.hpp. CRFSuite::Tagger::set() already does the same check and throws std::invalid_argument, so tag() on a closed tagger raised ValueError before and raises RuntimeError now:

>>> import pycrfsuite
>>> pycrfsuite.Tagger().tag([{"a": 1.0}])
# master
ValueError: The tagger is not opened
# this branch
RuntimeError: Tagger is closed

@jameshilliard
jameshilliard force-pushed the python315-freethreading branch from 95db3d9 to b0b4d18 Compare October 22, 2026 16:45
@AdrianAtZyte
AdrianAtZyte requested a review from wRAR October 22, 2026 18:30
@jameshilliard
jameshilliard force-pushed the python315-freethreading branch from b0b4d18 to 040230a Compare October 22, 2026 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tagger.tag segfaults if Tagger.open is not called

2 participants