[ppcvote]

• resolves doctools writes generated files using the locale encoding, corrupting 34 of them on non-UTF-8 machines #1237

Every generated-artifact writer opened its file in text mode with no encoding=, so it used the host's locale encoding. The descriptions' curly quotes exist in cp950 and cp1252, so those writes succeeded and emitted non-UTF-8 bytes with no warning and exit 0. The CSV writer separately lacked newline="", so to_csv's line endings became \r\r\n.

That matters because the doctools-regenerate hook and run_doctools.yml's own failure message both tell a contributor to regenerate and push data/.

Running the hook on a cp950 host, before and after:

	before	after
files under data/ modified	185	0
no longer valid UTF-8	34	0
CSVs containing \r\r\n	16	0

Seven one-argument changes across doctools.py, md_gen.py and decision_tables/helpers.py: encoding="utf-8" everywhere, newline="\n" on the text writers so output stays LF, and newline="" plus an explicit lineterminator on the two CSV writers. Nothing under data/ is committed here, so run_doctools.yml regenerates and verifies on its own runner.

The test runs a full generation under -X warn_default_encoding (PEP 597) and fails on any EncodingWarning raised from inside the package. That is what makes it work on your UTF-8 CI: asserting on the generated bytes would pass there whether or not the defect is present. It fails on the current code naming doctools.py:345 and :197, and passes with the change.

black --check is clean. The four failures in test_dp_helpers.py and test_mixins.py are present on an unmodified checkout here too.

[sei-vsarvepalli]

The question for me mostly was - if we ever have fp.write for csv's. I think we don't it should all be csv.writer that should work fine. the @ppcvote 's tests seem to be useful indicator.

[Copilot]

🟢 Approval recommended

The focused writer changes address the portability defects, with no blocking issues identified during static review; tests were not executed.

No open issues 🎉

What changed in this PR

Makes SSVC’s generated artifacts consistently use UTF-8 and LF line endings across platforms, addressing #1237.

Changes:

• Sets explicit encoding and newline handling for artifact writers.
• Forces LF line endings in both CSV writers.
• Adds a full-generation regression test for implicit encoding warnings.

File	Description
src/​test/​test_doctools_encoding.py	Adds encoding-warning regression coverage.
src/​ssvc/​md_gen.py	Writes Markdown as UTF-8 with LF.
src/​ssvc/​doctools.py	Standardizes JSON, schema, registry, and CSV output.
src/​ssvc/​decision_tables/​helpers.py	Standardizes CSV encoding and line endings.

_{Review effort: Balanced}