Freeze legacy into _legacy.py, reorganize tests, ship 9.1.0

CHANGELOG.md

@@ -1,3 +1,10 @@
+## 9.1.0 — unreleased
+
+- Modern mode only: decode uppercase `&#X..;` hexadecimal references as HTML allows, in addition to lowercase `&#x..;`. Legacy output is unchanged (Rupayon Haldar, #195).
+- Modern mode only: preserve post-replacement output when the whole slug already fits `max_length`, so intentional repeated/trailing delimiters are not collapsed at the length limit. Public `smart_truncate` is unchanged (emme1t, #193).
+- Fix `add_uppercase_char` to apply insertions atomically, leaving the input list unchanged if iteration fails. Built-in transliteration tables are unaffected (Cristian Ramirez, #194).
+- Modern mode only: validate argument types up front, raising `TypeError` for a `bool`/non-int `max_length` (`bool` is an `int` subclass) or a non-str `separator`. Legacy behavior is unchanged and still treats `max_length=True` as its historical single-character truncation (Jon Bailey, #196).
+
 ## 9.0.0 — unreleased
 
 - Add keyword-only `algorithm='legacy'` (permanent default) and explicit `algorithm='modern'` opt-in, plus CLI `--algorithm`. Preserve historical default entity ordering, numeric handling, iterator consumption, separator truncation, and public `smart_truncate` behavior.

DOJO.md

@@ -1,9 +1,12 @@
-# Project guidance
+# DOJO.md
+
+The source repository is `python-slugify/`. This is the single, top-level guidance file; there is no per-repo `DOJO.md` inside `python-slugify/`.
 
 ## Compatibility comes first
 
+- We do not change legacy architecture. Legacy stays in the past: its behavior, output pipeline, and code paths are frozen and must not be altered, refactored, "improved", or "fixed". The frozen legacy implementation lives in its own module (`slugify/_legacy.py`); any change to that file that alters legacy output is rejected on principle. The only permitted changes to legacy are those that do not alter its behavior at all (verified against the frozen baseline). All new work targets the explicit `algorithm='modern'` opt-in.
 - Many deployed sites depend on exact slug output. Existing calls must retain legacy behavior: `algorithm='legacy'` is the permanent default. Output-changing improvements require explicit `algorithm='modern'` opt-in; do not silently switch the default in a future release.
-- Preserve the original upstream `test.py` unchanged. Add new coverage in separate files. Compare default and explicit legacy behavior against the frozen baseline using identical transliteration backends/dependency versions.
+- Preserve the original upstream legacy test suite unchanged. It lives frozen at `tests/test_legacy.py` (the original `test.py`, contents unchanged), mirroring the frozen `slugify/_legacy.py` code split. Add new/modern coverage in separate files under `tests/` (e.g. `tests/test_release.py`); keep legacy and modern test coverage separate and do not edit the legacy suite. Compare default and explicit legacy behavior against the frozen baseline using identical transliteration backends/dependency versions.
 - Preserve the public `smart_truncate` contract. Modern slug generation uses separate internal handling; do not silently apply its semantics to old helper calls.
 - Keep existing stored URLs/keys unchanged. Applications own migration, aliases, redirects and collision detection. Slug generation cannot automatically know which URL exists in a database.
 - Passing tests establishes tested behavior, not universal compatibility. Report concrete gaps honestly. The CLI regex-forwarding correction is a documented intentional exception to old CLI behavior.
@@ -32,7 +35,8 @@
 
 - Read each item and verify what the implementation actually addresses. Distinguish incorporated, superseded, partially addressed, declined and deferred work, including modern-only fixes.
 - When authorized to consolidate/close items, add an individual explanatory cross-reference to the aggregate PR and a disposition index there. Credit contributors; never represent a closed-unmerged PR as merged or deferred work as implemented.
-- Identify agent-authored follow-ups as Dojo. Do not blanket-close unrelated items or merge PRs without authorization. A merged PR does not mean a package was published.
+- Identify agent-authored interactions as Dojo. For every Dojo-authored interaction in this repository—including opening, replying to, reviewing, or closing issues, pull requests, discussions, and releases—end the authored body or comment with the following attribution as the final line, separated from preceding content by a blank line: `🚀 Generated with [Dojo](https://heydojo.ai) ⛩️`
+- Do not blanket-close unrelated items or merge PRs without authorization. A merged PR does not mean a package was published.
 - Historical review drafts may predate compatibility decisions: check current source and migration documentation before posting them.
 - Licensing questions deserve calm, factual answers, not dismissal or blanket assurances. Separate this project's MIT license, installed dependency licenses, and runtime backend selection. Link the current README licensing section, migration guide, and relevant upstream license files; explain available choices and their limits.
 - When authorized to close a licensing item that is answered or duplicates an existing discussion, leave a respectful explanation and direct links to the authoritative documentation or tracking item. State whether it was answered, superseded, declined or deferred. Do not close a new unresolved licensing defect merely because similar questions recur. Never claim that an optional extra removes base dependencies, that GPL-associated means GPL-only, or that documentation constitutes legal advice.

MANIFEST.in

@@ -1,5 +1,6 @@
 include LICENSE README.md CHANGELOG.md
-include test.py test_release.py tox.ini dev.requirements.txt
+include tox.ini dev.requirements.txt
+recursive-include tests *.py
 recursive-include docs *.md
 recursive-include tools *.py
 recursive-include slugify *.py py.typed

README.md

@@ -82,6 +82,9 @@ slugify(
 ```
 
 - `algorithm`: `'legacy'` is the permanent default, preserving the historical output pipeline. `'modern'` explicitly opts into the changes below. Unknown values raise `ValueError`.
+
+  > **Note for contributors:** the legacy algorithm is frozen. Its behavior and output are intentionally kept as-is for backward compatibility, and we do **not** accept changes that alter legacy output — please do not open PRs to "fix" or "improve" legacy. All improvements target `algorithm='modern'`.
+
 - `text`: `str`, or UTF-8 `bytes`/`bytearray` (invalid bytes ignored). Other objects raise `TypeError`.
 - `entities`, `decimal`, `hexadecimal`: independently decode named HTML entities, decimal references, and hexadecimal references. Legacy accepts lowercase `x`; modern accepts both `x` and `X` as specified by HTML. Legacy decodes after transliteration and numeric substitutions are all-or-nothing per reference kind. Modern decodes before transliteration and handles invalid references independently.
 - `max_length`: legacy budgets internal dashes before separator mapping, so wide separators can exceed the limit. Modern budgets final Python characters, including emitted delimiters. Nonpositive means unlimited for slugify; this is not a byte or grapheme limit.
@@ -173,3 +176,5 @@ about your application's obligations. Evaluate the actual versions, distribution
 ## Sponsors
 
 [Neekware Inc.](https://neekware.com) — creator of [Dojo Workspace](https://heydojo.ai), your AI workspace for building, learning, and getting things done.
+
+🚀 Created with [Dojo](https://heydojo.ai) ⛩️

pyproject.toml

@@ -84,7 +84,7 @@ sqlite_cache = true
 # ------
 
 [tool.pytest.ini_options]
-testpaths = ["test.py", "test_release.py"]
+testpaths = ["tests"]
 addopts = "--color=yes"
 filterwarnings = [
     "error",

slugify/__version__.py

@@ -5,4 +5,4 @@
 __url__ = 'https://github.com/un33k/python-slugify'
 __license__ = 'SPDX-License-Identifier: MIT'
 __copyright__ = 'Copyright 2022 Val Neekman @ Neekware Inc.'
-__version__ = '9.0.0'
+__version__ = '9.1.0'

slugify/_legacy.py

@@ -0,0 +1,158 @@
+"""Frozen legacy slugify implementation.
+
+DO NOT MODIFY. This module is the permanently frozen legacy output pipeline.
+Many deployed sites depend on its exact output, so its behavior must never
+change. All improvements belong in the modern path (slugify/slugify.py).
+Any change to this file that alters legacy output is rejected on principle;
+the only acceptable edits are non-behavioral (e.g. this docstring) verified
+byte-for-byte against the frozen 2,688-case differential baseline.
+"""
+from __future__ import annotations
+
+import re
+import unicodedata
+from collections.abc import Iterable
+from html.entities import name2codepoint
+from importlib import import_module
+from typing import Literal
+
+CHAR_ENTITY_PATTERN = re.compile(r'&(%s);' % '|'.join(name2codepoint))
+DECIMAL_PATTERN = re.compile(r'&#(\d+);')
+HEX_PATTERN = re.compile(r'&#x([\da-fA-F]+);')
+QUOTE_PATTERN = re.compile(r"[']+")
+DISALLOWED_CHARS_PATTERN = re.compile(r'[^-a-zA-Z0-9]+')
+DISALLOWED_UNICODE_CHARS_PATTERN = re.compile(r'[\W_]+')
+DUPLICATE_DASH_PATTERN = re.compile(r'-{2,}')
+NUMBERS_PATTERN = re.compile(r'(?<=\d),(?=\d)')
+DEFAULT_SEPARATOR = '-'
+
+Backend = Literal['auto', 'text-unidecode', 'unidecode', 'anyascii']
+ReplacementStage = Literal['both', 'pre', 'post']
+
+
+def _decode_entities(text: str, entities: bool, decimal: bool, hexadecimal: bool) -> str:
+    if entities:
+        text = CHAR_ENTITY_PATTERN.sub(lambda m: chr(name2codepoint[m.group(1)]), text)
+    if decimal:
+        # Legacy substitution is all-or-nothing for each numeric reference kind.
+        try:
+            text = DECIMAL_PATTERN.sub(lambda m: chr(int(m.group(1))), text)
+        except (ValueError, OverflowError):
+            pass
+    if hexadecimal:
+        try:
+            text = HEX_PATTERN.sub(lambda m: chr(int(m.group(1), 16)), text)
+        except (ValueError, OverflowError):
+            pass
+    return text
+
+
+def _transliterate(text: str, backend: Backend) -> str:
+    if backend == 'auto':
+        try:
+            module = import_module('unidecode')
+        except ModuleNotFoundError as error:
+            if error.name != 'unidecode':
+                raise
+            module = import_module('text_unidecode')
+    else:
+        module = import_module(backend.replace('-', '_'))
+    # These third-party modules share a string-to-string API; some are untyped.
+    result: str = getattr(module, 'anyascii' if backend == 'anyascii' else 'unidecode')(text)
+    return result
+
+
+def smart_truncate(
+    string: str,
+    max_length: int = 0,
+    word_boundary: bool = False,
+    separator: str = " ",
+    save_order: bool = False,
+) -> str:
+    """Historical public truncation behavior, including character-set stripping.
+
+    Zero means unlimited; negative limits retain Python slicing semantics.
+    An empty separator raises ValueError, as in the legacy implementation.
+    """
+    string = string.strip(separator)
+    if not max_length:
+        return string
+    if len(string) < max_length:
+        return string
+    if not word_boundary:
+        return string[:max_length].strip(separator)
+    if separator not in string:
+        return string[:max_length]
+    truncated = ''
+    for word in string.split(separator):
+        if word:
+            next_len = len(truncated) + len(word)
+            if next_len < max_length:
+                truncated += '{}{}'.format(word, separator)
+            elif next_len == max_length:
+                truncated += '{}'.format(word)
+                break
+            elif save_order:
+                break
+    if not truncated:
+        truncated = string[:max_length]
+    return truncated.strip(separator)
+
+
+def slugify(
+    text: str | bytes | bytearray,
+    entities: bool = True,
+    decimal: bool = True,
+    hexadecimal: bool = True,
+    max_length: int = 0,
+    word_boundary: bool = False,
+    separator: str = DEFAULT_SEPARATOR,
+    save_order: bool = False,
+    stopwords: Iterable[str] = (),
+    regex_pattern: re.Pattern[str] | str | None = None,
+    lowercase: bool = True,
+    replacements: Iterable[Iterable[str]] = (),
+    allow_unicode: bool = False,
+    *,
+    replacement_stage: ReplacementStage = 'both',
+    backend: Backend = 'auto',
+) -> str:
+    """Frozen legacy output pipeline. Do not change its behavior."""
+    if not isinstance(text, str):
+        if not isinstance(text, (bytes, bytearray)):
+            raise TypeError(f'text must be str, bytes or bytearray, not {type(text).__name__}')
+        text = text.decode('utf-8', 'ignore')
+    if replacement_stage not in ('both', 'pre', 'post'):
+        raise ValueError("replacement_stage must be 'both', 'pre' or 'post'")
+    if backend not in ('auto', 'text-unidecode', 'unidecode', 'anyascii'):
+        raise ValueError("backend must be 'auto', 'text-unidecode', 'unidecode' or 'anyascii'")
+
+    if replacements and replacement_stage in ('both', 'pre'):
+        for old, new in replacements:
+            text = text.replace(old, new)
+
+    text = QUOTE_PATTERN.sub(DEFAULT_SEPARATOR, text)
+    if allow_unicode:
+        text = unicodedata.normalize('NFKC', text)
+    else:
+        text = _transliterate(unicodedata.normalize('NFKD', text), backend)
+    text = _decode_entities(text, entities, decimal, hexadecimal)
+    text = unicodedata.normalize('NFKC' if allow_unicode else 'NFKD', text)
+    if lowercase:
+        text = text.lower()
+    text = QUOTE_PATTERN.sub('', text)
+    text = NUMBERS_PATTERN.sub('', text)
+    pattern = regex_pattern or (DISALLOWED_UNICODE_CHARS_PATTERN if allow_unicode else DISALLOWED_CHARS_PATTERN)
+    text = re.sub(pattern, DEFAULT_SEPARATOR, text)
+    text = DUPLICATE_DASH_PATTERN.sub(DEFAULT_SEPARATOR, text).strip(DEFAULT_SEPARATOR)
+
+    if stopwords:
+        excluded = [word.lower() for word in stopwords] if lowercase else stopwords
+        text = DEFAULT_SEPARATOR.join(word for word in text.split(DEFAULT_SEPARATOR) if word not in excluded)
+    if replacements and replacement_stage in ('both', 'post'):
+        for old, new in replacements:
+            text = text.replace(old, new)
+
+    if max_length > 0:
+        text = smart_truncate(text, max_length, word_boundary, DEFAULT_SEPARATOR, save_order)
+    return text.replace(DEFAULT_SEPARATOR, separator) if separator != DEFAULT_SEPARATOR else text