Skip to content

CI: Reject hidden Unicode characters and remove soft hyphens - #719

Closed
echoix wants to merge 3 commits into
mainfrom
claude/hidden-unicode-characters-f3b6yv
Closed

echoix wants to merge 3 commits into
mainfrom
claude/hidden-unicode-characters-f3b6yv

Conversation

@echoix

@echoix echoix commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Description

  • Remove the soft hyphens (U+00AD) from the "pixel-based" reference title
    in i.svm.predict and i.svm.train (.md and .html files).
  • Add a local forbid-hidden-unicode pre-commit hook (pygrep) that
    rejects soft hyphens, zero-width characters (U+200B-200F), bidirectional
    controls (U+202A-202E, U+2066-2069), U+2060-2064 and the byte order mark.
    locale/po/ is excluded because translations legitimately contain some of
    these characters (for example Malayalam joiners).

Motivation and context

The Renovate dependency dashboard reports "Hidden Unicode characters have
been discovered in file(s) in your repository". The Renovate log names two
files, imagery/i.svm.predict/i.svm.predict.html and
imagery/i.svm.train/i.svm.train.html, both with ``. The character
was a copy and paste artifact in a literature reference. The .md sources
had it too.

editorconfig-checker does not detect invisible characters (it checks line
endings, final newline, trailing whitespace, indentation and charset), so
nothing prevented this from being committed. The new hook closes that gap.

The hook is written as UTF-8 byte sequences because pygrep matches bytes
and does not accept \u escapes.

How has this been tested?

  • pre-commit run forbid-hidden-unicode --all-files passes on the
    repository after the cleanup.
  • The hook fails on test files containing U+00AD, U+200B, U+202E, U+2060 and
    U+FEFF.
  • A test file with only a no-break space and a regular Unicode hyphen
    (U+2010) passes, so ordinary text is not flagged.

The other pre-commit hooks were not run.

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing
    functionality to not work as before)

Documentation cleanup and a new CI check.

Checklist

  • PR title provides summary of the changes and starts with one of the
    pre-defined prefixes
  • My code follows the code style
    of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.

AI disclosure

This change was prepared with the help of Claude Code. The hook regular
expression was reviewed and tested as described above.

🤖 Generated with Claude Code

Renovate warned about soft hyphens (U+00AD) in the i.svm.predict and
i.svm.train documentation, copied from a reference title. Remove them
from the .md and .html files.

Add a local pygrep hook that rejects soft hyphens, zero-width
characters, bidirectional controls and the byte order mark, which
editorconfig-checker does not detect. Translation files are excluded
because they legitimately contain some of these characters.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@echoix
echoix force-pushed the claude/hidden-unicode-characters-f3b6yv branch from f258622 to f1fdebc Compare October 3, 2026 15:26
@echoix echoix closed this Oct 9, 2026
@echoix
echoix deleted the claude/hidden-unicode-characters-f3b6yv branch October 9, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants