From 84ca53150d99215bdbe6ca33107bc0c17708ea3e Mon Sep 17 00:00:00 2001 From: Aaron Wang Date: Wed, 14 Oct 2026 07:35:39 -0400 Subject: [PATCH] Repair locale-dependent reads in test_research_current_status (no new kappa) On a native Windows cp1252 host, the documented check python3 -B -m unittest discover -s tests -p 'test_research_*.py' fails with three errors when PYTHONUTF8 is not set: STATUS.read_text() decodes docs/research/current-status.md with the locale codec and the document carries byte 0x9d (U+201D). Pass explicit UTF-8 and add a static locale-independence guard plus a load-bearing byte control in the same module, following the test_research_onboarding.TextEncodingTests pattern. Adds research/test-suite-locale-reads: task card, base-commit source pins and a bounded registered replay with cp1252/bare-read failure controls and a forced legacy codec (cp1252 on Windows, C locale elsewhere). Credit: Aaron Wang, with help from opencode. Predecessor: the merged #1299/#1293-rgb Windows encoding line, whose five-script static guard did not cover the suite's document readers. --- research/test-suite-locale-reads/README.md | 166 ++++++++++++++++++ research/test-suite-locale-reads/SOURCES.json | 30 ++++ research/test-suite-locale-reads/TASK.json | 67 +++++++ research/test-suite-locale-reads/VERIFY.json | 21 +++ research/test-suite-locale-reads/verify.py | 75 ++++++++ tests/test_research_current_status.py | 29 ++- 6 files changed, 386 insertions(+), 2 deletions(-) create mode 100644 research/test-suite-locale-reads/README.md create mode 100644 research/test-suite-locale-reads/SOURCES.json create mode 100644 research/test-suite-locale-reads/TASK.json create mode 100644 research/test-suite-locale-reads/VERIFY.json create mode 100644 research/test-suite-locale-reads/verify.py diff --git a/research/test-suite-locale-reads/README.md b/research/test-suite-locale-reads/README.md new file mode 100644 index 0000000000..a0431c3c32 --- /dev/null +++ b/research/test-suite-locale-reads/README.md @@ -0,0 +1,166 @@ +# test-suite-locale-reads + +**No kappa, no certificate change.** One reproducible environment failure +repaired: the repository's documented test command fails on a native Windows +cp1252 host because `tests/test_research_current_status.py` reads +`docs/research/current-status.md` with the locale codec. + +## Task card + +- **Task ID / scope key:** `verifier-engineering` / `portable-replay` + (scope key `831cc1a3c8f9ff01b197`, chooser OS nonce + `c7669648496a57422bfa478f1d356cad`, priority revision 3). +- **Base commit:** `0b1cccf86e2f5132ed286a26ef0a681a495740ea` (selected-record + SHA-256 `10b9e340fd8e354a37a0c4813c90e5de1dd42044b5f6aca4a3fc5a42b685fa32`). +- **Concrete object and question:** the suite's own document reads. On Windows + with the Git-for-Windows defaults, `python3 -B -m unittest discover -s tests + -p 'test_research_*.py'` (without `PYTHONUTF8`) fails with three errors in + `test_research_current_status.CurrentStatusTests`, all from + `STATUS.read_text()` decoding `docs/research/current-status.md` with cp1252 + while the document contains byte `0x9d` (in U+201D). Can this be repaired + minimally and pinned by a regression? +- **Scope status:** refined, within the card's premise ("one reproducible + environment failure repaired") and its allowed variation (one + interpreter/environment mismatch). Reason recorded in [TASK.json](TASK.json). +- **Existing restriction being changed:** none in the mathematics. The prior + Windows-encoding work (merged #1299/#1293-rgb line, credited below) covered + `scripts/research_package.py`, `scripts/find_overlap.py` and + `tests/test_research_onboarding.py`; its static guard + (`TextEncodingTests.SCRIPTS`) covers five onboarding scripts only, and does + not cover the suite's other document readers. +- **Method; parameter range:** reproduce on native Windows cp1252 (Python + 3.13.15), fix the module's text reads to explicit UTF-8, add a static + locale-independence guard (the same AST pattern the onboarding tests use) and + a load-bearing byte control, and register a bounded replay that forces a + legacy codec (cp1252 on Windows, the C locale elsewhere). +- **Nearest overlapping PRs:** https://github.com/CrocSwap/integer-mult-bounds/pull/1702 + (makes the CP1252 overlap regression in `test_research_onboarding.py` + self-contained; different file and object, no conflict with this change) and + the merged #1299 line (predecessor). The gzip/canonical-payload variation of + the card is under active concurrent work + (https://github.com/CrocSwap/integer-mult-bounds/pull/1559, + https://github.com/CrocSwap/integer-mult-bounds/pull/1824) and was not + duplicated. Overlap refreshed at 2026-10-14 11:10Z and again before + submission. +- **Optimistic full cost budget:** trivial runtime (the suite runs in about + 15 s). Required observable improvement: the documented command exits 0 with + and without `PYTHONUTF8` on native Windows, and the new guard fails on the + base-commit module. +- **Success evidence; falsification; cap:** see below. Falsification test: the + static guard reports exactly the base module's bare reads (lines 10 and 26), + and a bare locale read of the document fails in a forced-legacy child. One + session, one pivot (the historical-failure candidate was killed first). + +## Scope and evidence + +**What was reproduced (before).** Windows 11, cp1252, Git 2.55.0.windows.5 +(`core.autocrlf=true`), CPython 3.13.15. At base `0b1cccf8`, without +`PYTHONUTF8`: + +```text +ERROR: test_current_section_matches_selected_record ... setUp + File ".../tests/test_research_current_status.py", line 26, in setUp + self.text = STATUS.read_text() +UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1014 +Ran 42 tests in 13.787s — FAILED (errors=3, skipped=2) +``` + +The same command with `PYTHONUTF8=1`, and everything under CPython 3.12, +passed. The three errors are one root cause: a locale-dependent read of a UTF-8 +document. + +**Obstruction recorded (killed first candidate).** The historical Windows +failures cited in many PR bodies no longer reproduce at base: +`test_research_onboarding` runs green natively (Python 3.13.15: 18 tests with 2 +environment-gated skips; Python 3.12: all 18 pass, `core.autocrlf=true`), +including the formerly failing scaffold-pin and +`os.killpg` tests. Credit for that repair goes to the merged +#1299/#1293-rgb line ("UTF-8 metadata/pipe handling and Windows child-tree +timeout fallback", recorded in `research/community-efficient400-audit`) and to +the feedback report in +https://github.com/CrocSwap/integer-mult-bounds/discussions/1200#discussioncomment-18910079, +which identified the same byte (`0x9d`) in `research_package.py`. This package +fixes the instance that line missed and pins it. + +**What is established (after).** The module's reads now pass explicit +`encoding="utf-8"`. Two regression tests were added in the same module, in the +style of `test_research_onboarding.TextEncodingTests` ("a UTF-8 CI host cannot +observe locale-codec bugs, so check the calls statically"): an AST guard +requiring `encoding=` on every text read/write in the module, and a +load-bearing control asserting the document contains `e2 80 9d` and is +undecodable as cp1252 (deterministic on every host). Evidence: + +```text +python -B -m unittest discover -s tests -p 'test_research_*.py' # no PYTHONUTF8, 3.13.15 +Ran 44 tests in 14.672s — OK (skipped=2) +python3.12 -B -m unittest discover -s tests -p 'test_research_*.py' +Ran 44 tests in 16.541s — OK +``` + +Failure controls (both run in this session): the static guard applied to the +base-commit module reports missing `encoding=` at exactly lines 10 and 26; a +bare `read_text()` of the document in a `PYTHONUTF8=0` child fails with the +same `UnicodeDecodeError` on this host. The registered replay +(`verify.py`) reruns all of this with a forced legacy codec. + +**Limitations (not checked).** No mathematics: no kappa, certificate, frozen +pin, `upstream/`, selected-record or cold-replay change; the acceptance gate +"preserve immutable source hashes and proof-relevant content" is untouched +because no comparison logic changed. The roughly 100 remaining +locale-dependent text calls in `tests/` and the one-off `scripts/` generators +were inventoried but not swept (open question 1). The registered replay forces +the legacy codec only for the repaired module; whole-suite behavior under a +true POSIX ASCII locale is unmeasured (open question 2). macOS and Linux hosts +were not executed here. [SOURCES.json](SOURCES.json) pins +`tests/test_research_current_status.py` at its **base-commit** bytes as +provenance (the defective version this PR repairs); it cannot be a live input +under the pin contract because this contribution changes it. +`docs/research/current-status.md` is the live input and matches its pin. + +## Reproduce + +From repo root, on any host: + +```sh +python3 scripts/research_package.py --package research/test-suite-locale-reads --check-only +python3.12 scripts/research_package.py --package research/test-suite-locale-reads +``` + +The second command runs `verify.py`: the cp1252 load-bearing control, the bare +legacy-locale read failure control, the module's unittest under a forced legacy +codec and the default environment, and the documented suite command in the +default environment. The native-Windows reproduction of the original failure +needs no `PYTHONUTF8`; on POSIX the replay forces `LC_ALL=C` with +`PYTHONCOERCECLOCALE=0` and `PYTHONUTF8=0`. + +## Open next questions + +1. **Sweep the remaining locale-dependent reads in `tests/`** (about 100 calls + across about 40 modules). Why it matters: only `current-status.md` carries + cp1252-undefined bytes today, so only one module crashes; the next + typographic quote added to any read document recreates the same bogus + errors elsewhere. First step: extend the static AST guard to all + `tests/*.py` and fix the call sites that read documents already containing + non-ASCII bytes. Blocker: mechanical churn across frozen-era test modules + wants maintainer buy-in. +2. **Whole-suite guarantee under a POSIX ASCII locale.** Why it matters: + "green on Windows cp1252" and "green on `LC_ALL=C`" are different claims; + the registered replay only forces the legacy codec for the repaired module. + First step: run the full discover command under `PYTHONUTF8=0 + PYTHONCOERCECLOCALE=0 LC_ALL=C` on Linux and triage failures. Blocker: this + session ran on Windows only. + +## Credit + +Prepared by Aaron Wang, with help from opencode. New work is Apache-2.0. + +Predecessor credit: the merged #1299/#1293-rgb Windows encoding line +(UTF-8 metadata and pipe handling, child-tree timeout fallback, cp1252 +simulations and the `TextEncodingTests` static-guard pattern, summarized in +`research/community-efficient400-audit`); the failure report and environment +details from +https://github.com/CrocSwap/integer-mult-bounds/discussions/1200#discussioncomment-18910079; +the `docs/research/current-status.md` consistency tests themselves +(`tests/test_research_current_status.py`, authored upstream of this base). This +package adds the instance their guard missed, a regression in the module that +failed, and a replayable legacy-locale check. No mathematical claim is made. diff --git a/research/test-suite-locale-reads/SOURCES.json b/research/test-suite-locale-reads/SOURCES.json new file mode 100644 index 0000000000..be5f08378a --- /dev/null +++ b/research/test-suite-locale-reads/SOURCES.json @@ -0,0 +1,30 @@ +{ + "schema_version": 1, + "base_commit": "0b1cccf86e2f5132ed286a26ef0a681a495740ea", + "files": [ + { + "path": "certificates/selected-result.json", + "usage": "provenance", + "git_blob": "ae6a99f28aa468649eb66a361e764d769363bf11", + "sha256": "10b9e340fd8e354a37a0c4813c90e5de1dd42044b5f6aca4a3fc5a42b685fa32" + }, + { + "path": "docs/research/agent-task-catalog.json", + "usage": "provenance", + "git_blob": "06c96f5579bb67ed1b86d334e78fbed7da6945f1", + "sha256": "6686ad58f1fbdbc87134a78901b60fd53a20d3984355e91defbbebc9e9fa8075" + }, + { + "path": "docs/research/current-status.md", + "usage": "input", + "git_blob": "d73b4ff9da41f044e793422db4318be17e234a6c", + "sha256": "adb54af005eb11edaf84d782fe729338680ff908f6d4fbf39ba1062d2e7d8a77" + }, + { + "path": "tests/test_research_current_status.py", + "usage": "provenance", + "git_blob": "356f49df3c7f8a3d6266322fabc8f47cc4666f93", + "sha256": "9698ecb1f989a2a8282aecaabb8e13e433fc5dfdd0423b6a3ca6e690757c2b4a" + } + ] +} diff --git a/research/test-suite-locale-reads/TASK.json b/research/test-suite-locale-reads/TASK.json new file mode 100644 index 0000000000..2c7ef5d50b --- /dev/null +++ b/research/test-suite-locale-reads/TASK.json @@ -0,0 +1,67 @@ +{ + "schema_version": 1, + "task_id": "portable-replay", + "priority_revision": 3, + "base_commit": "0b1cccf86e2f5132ed286a26ef0a681a495740ea", + "original_card": { + "id": "portable-replay", + "question": "Can one reproducible environment or artifact-comparison failure be repaired?", + "starting_points": [ + "research/gzip-container-reproducibility/README.md", + "docs/ci-verification.md" + ], + "methods": [ + "Reproduce one portability failure and add a minimal cross-environment fix.", + "Audit one canonical-payload comparison for omitted proof-relevant fields." + ], + "acceptance_gate": "Preserve immutable source hashes and proof-relevant content; a different compressed byte stream alone is not a mathematical mismatch.", + "first_step": { + "command": [ + "python3", + "-B", + "scripts/research_baseline.py" + ], + "expected": "PASS_RESEARCH_BASELINE_BINDINGS_ONLY", + "inspect": "Read the gzip-container reproducibility examples and distinguish frozen source bytes from regenerated payloads.", + "scope": "Positive control / source-bound orientation only; this is not a new construction or full cold replay." + }, + "first_experiment": "Reproduce one specific mismatch or audit one canonical comparison for omitted proof-relevant fields.", + "bottleneck_link": "Portable exact checks prevent environment failures from blocking valid contributions.", + "allowed_variations": [ + "One interpreter/version mismatch or one payload field comparison." + ], + "scope_change_triggers": [ + "Removing immutable-source pins because generated compressed bytes differ." + ], + "overlap_terms": [ + "gzip container", + "reproducibility", + "canonical payload" + ] + }, + "assignment_seed": "c7669648496a57422bfa478f1d356cad", + "scope_status": "refined", + "actual_scope": "Repair one reproducible environment failure: on a native Windows cp1252 host the documented check python3 -B -m unittest discover -s tests -p 'test_research_*.py' errors in tests/test_research_current_status.py because its text reads use the locale codec while docs/research/current-status.md carries byte 0x9d (U+201D). Fix the reads to explicit UTF-8, add a static locale-independence guard and a load-bearing byte control in the same module, and register a bounded legacy-locale replay.", + "scope_change_reason": "Refined within the card's premise (one reproducible environment failure repaired) and its allowed variation (one interpreter/environment mismatch). The card's obvious first candidates were killed first: the historical Windows failures cited by many PR bodies (autocrlf temp-fixture pin mismatch in test_research_onboarding.PackageTests, POSIX-only os.killpg, cp1252 metadata reads in research_package.py/find_overlap.py) no longer reproduce at base 0b1cccf8 on native Windows (test_research_onboarding runs green: Python 3.13.15 has 18 tests with 2 environment-gated skips, Python 3.12 passes all 18 with core.autocrlf=true; the merged #1299/#1293-rgb line already repaired them). What remains is the suite's own document reads, which that line's five-script static guard (test_research_onboarding.TextEncodingTests.SCRIPTS) does not cover. The gzip/canonical-payload variation of the card is under active concurrent work (https://github.com/CrocSwap/integer-mult-bounds/pull/1559, https://github.com/CrocSwap/integer-mult-bounds/pull/1824) and was deliberately not duplicated.", + "bottleneck_link": "Portable exact checks prevent environment failures from blocking valid contributions. The documented suite command is the check every reviewer asks contributors to run; on the most common desktop configuration (Windows, Git-for-Windows defaults) it reports three bogus errors unless the contributor knows to set PYTHONUTF8=1, which is how several PR bodies came to disclaim 'pre-existing Windows failures' instead of getting a clean signal.", + "budget_minutes": 60, + "stop_rule": "One repair of tests/test_research_current_status.py plus one bounded package and one focused PR. No sweep of the ~100 remaining locale-dependent text calls in tests/ or the scripts/ one-off generators (inventoried as an open question). No frozen-pin, upstream, selected-record or certificate changes. One pivot maximum: the pivot from the killed historical-failure candidate to the suite's document reads is the one pivot used.", + "outcome": "completed", + "evidence": "Native Windows 11 (cp1252, Git 2.55 autocrlf=true), CPython 3.13.15. BEFORE (base 0b1cccf8): python -B -m unittest discover -s tests -p 'test_research_*.py' without PYTHONUTF8 -> 'Ran 42 tests ... FAILED (errors=3)', all three in test_research_current_status.CurrentStatusTests.setUp, UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1014. Same command with PYTHONUTF8=1 -> OK (skipped=2); python3.12 -> OK. AFTER (this change): same command without PYTHONUTF8 -> 'Ran 44 tests ... OK (skipped=2)' on Python 3.13.15 and 'Ran 44 tests ... OK' on Python 3.12, with and without PYTHONUTF8. Failure controls (run recorded): the base-commit module carries bare reads at lines 10 and 26 under the new static guard (guard reports exactly those lines); a bare locale read of docs/research/current-status.md in a PYTHONUTF8=0 child fails with the same UnicodeDecodeError on this host, and the module's new test asserts the load-bearing byte sequence e2 80 9d and cp1252 undecodability on every host. Registered replay: python3.12 scripts/research_package.py --package research/test-suite-locale-reads passes; metadata check --check-only passes. scripts/research_baseline.py reports PASS_RESEARCH_BASELINE_BINDINGS_ONLY (card first step, unchanged).", + "author": "Aaron Wang", + "ai_assistance": "with help from opencode", + "open_questions": [ + { + "question": "Should the remaining locale-dependent text reads in tests/ (~100 calls across ~40 modules) be swept the way test_research_onboarding.TextEncodingTests guards the five onboarding scripts?", + "why_it_matters": "Only tests/test_research_current_status.py crashes today because only docs/research/current-status.md carries cp1252-undefined bytes among the documents the suite reads; any typographic quote added to formal/lean/SOURCES.md, a workflow yml or a vendored .tex reintroduces the same bogus errors in another module.", + "first_step": "Extend the static AST guard to all tests/*.py, triage which reads target documents that already contain non-ASCII bytes (a byte scan of the repo finds 1536 non-ASCII files, mostly under research/), and fix those call sites first.", + "blocked_by": "Nothing technical; the mechanical churn across frozen-era test modules wants maintainer buy-in before a sweep PR." + }, + { + "question": "Can the whole suite, not just tests/test_research_current_status.py, be guaranteed under a forced legacy codec on POSIX?", + "why_it_matters": "The registered replay forces the C locale only for the repaired module; other modules' reads could still fail under a true ASCII locale on Linux even though they pass under cp1252, so 'green on Windows' and 'green on LC_ALL=C' are different claims.", + "first_step": "Run the full discover command under PYTHONUTF8=0 PYTHONCOERCECLOCALE=0 LC_ALL=C on a Linux host and triage whatever fails.", + "blocked_by": "This session ran on Windows only; the POSIX C-locale behavior of the unmodified modules is unmeasured." + } + ] +} diff --git a/research/test-suite-locale-reads/VERIFY.json b/research/test-suite-locale-reads/VERIFY.json new file mode 100644 index 0000000000..bcb20f7cd8 --- /dev/null +++ b/research/test-suite-locale-reads/VERIFY.json @@ -0,0 +1,21 @@ +{ + "schema_version": 1, + "status": "ready", + "python": "3.12", + "requirements": [], + "inputs": [ + "docs/research/current-status.md", + "tests/test_research_current_status.py" + ], + "commands": [ + { + "argv": [ + "{python}", + "-B", + "{package}/verify.py" + ], + "timeout_seconds": 300, + "checks": "Locale-independence of tests/test_research_current_status.py: cp1252 and bare-read failure controls on docs/research/current-status.md, the module's unittest run under a forced legacy codec (cp1252 on Windows, C locale elsewhere) and under the default environment, and the documented suite command in the default environment. No frozen-pin, replay or selected-record checks." + } + ] +} diff --git a/research/test-suite-locale-reads/verify.py b/research/test-suite-locale-reads/verify.py new file mode 100644 index 0000000000..8183c4d8c9 --- /dev/null +++ b/research/test-suite-locale-reads/verify.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +"""Locale-independence check for the repository's document-reading tests. + +Scope: tests/test_research_current_status.py reads docs/research/current-status.md. +On a legacy-locale host (cp1252 on Windows, the C locale elsewhere) a bare text +read of that document raises UnicodeDecodeError because it contains byte 0x9d +(in U+201D). Controls: + - failure control 1: the document must still be undecodable as cp1252, + - failure control 2: a bare locale read in a legacy-locale child must fail, + - positive: the repaired module passes in a legacy-locale child and in the + default environment, and the whole documented suite passes by default. + +Not checked: the ~100 remaining locale-dependent text calls in tests/ and the +one-off scripts/ generators; frozen pins, replays and the selected record. +""" +import os +import subprocess +import sys +from pathlib import Path + +ROOT = Path.cwd() +STATUS = ROOT / "docs/research/current-status.md" +MODULE = "tests.test_research_current_status" + + +def fail(message): + raise SystemExit("FAIL_TEST_SUITE_LOCALE_READS " + message) + + +def legacy_env(): + env = dict(os.environ) + env["PYTHONUTF8"] = "0" + env.pop("PYTHONLEGACYWINDOWSUTF8", None) + if os.name != "nt": + env["PYTHONCOERCECLOCALE"] = "0" + env["LC_ALL"] = "C" + env["LANG"] = "C" + return env + + +# Failure control 1: the fixture is load-bearing (byte 0x9d, undefined in cp1252). +try: + STATUS.read_text(encoding="cp1252") +except UnicodeDecodeError: + pass +else: + fail("document decodes as cp1252; the legacy-locale fixture is not load-bearing") + +# Failure control 2: a bare locale read fails under the forced legacy codec. +probe = "from pathlib import Path; Path('docs/research/current-status.md').read_text()" +bare = subprocess.run([sys.executable, "-c", probe], env=legacy_env(), + capture_output=True, text=True) +if bare.returncode == 0 or "UnicodeDecodeError" not in bare.stderr: + fail("host cannot exhibit a legacy-codec read failure: " + bare.stderr[-200:]) + +# Positive: the repaired module passes under the forced legacy codec. +legacy = subprocess.run([sys.executable, "-B", "-m", "unittest", MODULE], + env=legacy_env(), capture_output=True, text=True) +if legacy.returncode != 0: + fail("legacy-locale unittest run failed:\n" + legacy.stderr[-2000:]) + +# Positive: the module passes in the default environment. +default = subprocess.run([sys.executable, "-B", "-m", "unittest", MODULE], + capture_output=True, text=True) +if default.returncode != 0: + fail("default-environment unittest run failed:\n" + default.stderr[-2000:]) + +# Positive: the documented suite command passes in the default environment. +suite = subprocess.run([sys.executable, "-B", "-m", "unittest", "discover", + "-s", "tests", "-p", "test_research_*.py"], + capture_output=True, text=True) +if suite.returncode != 0: + fail("documented suite command failed:\n" + suite.stderr[-2000:]) + +print("PASS_TEST_SUITE_LOCALE_READS legacy-locale reads repaired in tests/test_research_current_status.py") diff --git a/tests/test_research_current_status.py b/tests/test_research_current_status.py index 356f49df3c..125464489a 100644 --- a/tests/test_research_current_status.py +++ b/tests/test_research_current_status.py @@ -1,5 +1,6 @@ """The status page's only "current" section must agree with the selected record.""" +import ast import json from pathlib import Path import re @@ -7,7 +8,7 @@ ROOT = Path(__file__).resolve().parents[1] STATUS = ROOT / "docs/research/current-status.md" -SELECTED = json.loads((ROOT / "certificates/selected-result.json").read_text()) +SELECTED = json.loads((ROOT / "certificates/selected-result.json").read_text(encoding="utf-8")) def headings(text): @@ -23,7 +24,7 @@ def headings(text): class CurrentStatusTests(unittest.TestCase): def setUp(self): - self.text = STATUS.read_text() + self.text = STATUS.read_text(encoding="utf-8") self.lines = self.text.splitlines() self.heads = headings(self.text) @@ -46,5 +47,29 @@ def test_history_banner_precedes_historical_sections(self): self.assertIn("selected-result.json", "\n".join(self.lines[:end])) +class LocaleIndependenceTests(unittest.TestCase): + def test_this_modules_text_io_does_not_depend_on_host_locale(self): + # A UTF-8 CI host cannot observe locale-codec bugs, so check the calls + # statically (the approach of test_research_onboarding.TextEncodingTests). + source = Path(__file__).read_text(encoding="utf-8") + missing = [] + for node in ast.walk(ast.parse(source)): + if not isinstance(node, ast.Call): + continue + func = getattr(node.func, "attr", getattr(node.func, "id", None)) + keywords = {k.arg: k.value for k in node.keywords} + if func in ("read_text", "write_text", "open") and "encoding" not in keywords: + missing.append(node.lineno) + self.assertEqual(missing, [], "pass encoding= to every text read/write in this module") + + def test_status_document_carries_the_load_bearing_byte(self): + # Byte 0x9d (in U+201D) is undefined in cp1252, the default Windows + # locale codec, so a bare locale read of this document fails there. + raw = STATUS.read_bytes() + self.assertIn(b"\xe2\x80\x9d", raw) + with self.assertRaises(UnicodeDecodeError): + STATUS.read_text(encoding="cp1252") + + if __name__ == "__main__": unittest.main()