Skip to content

Handle invalid numeric entity references individually - #181

Closed
CodingFeng101 wants to merge 1 commit into
un33k:masterfrom
CodingFeng101:codex/preserve-valid-numeric-entities
Closed

CodingFeng101 wants to merge 1 commit into
un33k:masterfrom
CodingFeng101:codex/preserve-valid-numeric-entities

Conversation

@CodingFeng101

Copy link
Copy Markdown

Summary

  • convert decimal and hexadecimal numeric character references independently
  • keep invalid numeric references unchanged so later slug cleanup can handle them
  • add regression coverage for mixed valid and invalid numeric references

Why

The previous implementation wrapped the whole regex substitution in one try block. If one numeric character reference could not be converted to a valid Unicode code point, conversion was skipped for every other match in the same string.

For example, Ž � produced 381-9999999999, even though the first reference is valid and should become z.

Checks

  • python -m pytest -q -> 84 passed
  • python -m mypy -> Success: no issues found in 5 source files
  • python -m pycodestyle --ignore=E128,E261,E225,E501,W605 slugify test.py setup.py
  • git diff --check

@un33k

un33k commented Sep 8, 2026

Copy link
Copy Markdown
Owner

This is Dojo, posting a maintainer-authorized follow-up linking this PR to #191.

Per-reference invalid numeric entity recovery is incorporated ONLY in opt-in algorithm='modern', including invalid-neighbor and surrogate cases. Legacy retains historical substitution behavior. This standalone PR is superseded.

Closing this item under the maintainer’s consolidation decision. The default algorithm remains legacy; improved output rules are opt-in. This note does not announce a published release, and no individual PR is being merged by this follow-up. Thank you for the contribution and discussion.

🚀 Generated with Dojo ⛩️

@un33k un33k closed this Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants