[echoix]

Renovate’s dependency dashboard is showing us warnings of hidden Unicode characters. It’s finally only soft-hyphens in a citation. But to prevent something similar to being introduced, add a simple pre-commit hook that will catch those.

Description

• Remove the soft hyphens (U+00AD) from the "pixel-based" reference title
  in i.svm.predict and i.svm.train (.md and .html files).
• Add a local forbid-hidden-unicode pre-commit hook (pygrep) that
  rejects soft hyphens, zero-width characters (U+200B-200F), bidirectional
  controls (U+202A-202E, U+2066-2069), U+2060-2064 and the byte order mark.
  locale/po/ is excluded because translations legitimately contain some of
  these characters (for example Malayalam joiners).

Motivation and context

The Renovate dependency dashboard reports "Hidden Unicode characters have
been discovered in file(s) in your repository". The Renovate log names two
files, imagery/i.svm.predict/i.svm.predict.html and
imagery/i.svm.train/i.svm.train.html, both with ``. The character
was a copy and paste artifact in a literature reference. The .md sources
had it too.

editorconfig-checker does not detect invisible characters (it checks line
endings, final newline, trailing whitespace, indentation and charset), so
nothing prevented this from being committed. The new hook closes that gap.

The hook is written as UTF-8 byte sequences because pygrep matches bytes
and does not accept \u escapes.

How has this been tested?

• pre-commit run forbid-hidden-unicode --all-files passes on the
  repository after the cleanup.
• The hook fails on test files containing U+00AD, U+200B, U+202E, U+2060 and
  U+FEFF.
• A test file with only a no-break space and a regular Unicode hyphen
  (U+2010) passes, so ordinary text is not flagged.

The other pre-commit hooks were not run.

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing
  functionality to not work as before)

Documentation cleanup and a new CI check.

Checklist

• PR title provides summary of the changes and starts with one of the
  pre-defined prefixes
• My code follows the code style
  of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.

AI disclosure

This change was prepared with the help of Claude Code. The hook regular
expression was reviewed and tested as described above.

🤖 Generated with Claude Code