fix(evaluation): Correct CSV encoding issues

[AkhileshNegi]

Issue

Closes #PLEASE_TYPE_ISSUE_NUMBER

Summary

• Before: The flow failed with 422 errors for both Excel CSV formats due to encoding issues.
• Now: The flow correctly decodes uploaded CSVs using multiple encodings, eliminating errors for common cases.
• Added logic to probe for BOMs and fallback through encoding options.
• Implemented handling for cp1252 and latin-1 to recover intended characters.

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran uv run pytest app/tests/services/evaluations/ — 57 passed. Also mypy --strict and ruff clean on the touched files.
• Added test cases: 6 new cases in TestParseCsvItemsEncoding covering UTF-8, UTF-8+BOM, cp1252, UTF-16+BOM, UTF-32+BOM (guarding the probe order), and a byte undefined in cp1252 falling through to latin-1.
• Did not run the server. Verified instead by calling parse_csv_items directly against a byte-for-byte reproduction of the failing upload (0x96 at position 292, matching the AppSignal message exactly), plus cp1252 smart quotes and a curly apostrophe. All six rows now decode to the intended characters:

Notes

• Behaviour change: files that previously 422'd now upload. Nothing that used to succeed decodes differently — utf-8-sig is byte-identical to utf-8 on BOM-less input, and the later rungs are only reached after utf-8-sig raises.
• Updated docs/wiki/modules/evaluations.md per the maintenance rule.
• Not addressed here: this only fixes the decode. Guessing a legacy encoding is a heuristic, so a file in some other single-byte codepage (e.g. ISO-8859-7 Greek) will now decode without error to the wrong characters rather than 422. That trade seems right for the actual user population, but flagging it.

Original PR description

Issue

No linked issue — found via an AppSignal error on the Glific side (org 290):

Glific.ThirdParty.Kaapi.Error: Failed to upload evaluation dataset to Kaapi
reason: %{status: 422, body: %{error: "Invalid CSV file: 'utf-8' codec can't
decode byte 0x96 in position 292: invalid start byte", ...}}

Summary

parse_csv_items decoded uploaded bytes with a hard-coded utf-8. Both of Excel's CSV export options produce bytes that fails, so there was no Excel path that worked:

Excel "Save As"	Bytes written	Result before this PR
CSV (Comma delimited)	cp1252	422 Invalid CSV file: 'utf-8' codec can't decode byte 0x96
CSV UTF-8	UTF-8 + BOM	422 CSV must contain 'question' and 'answer' columns. Missing: ['question']

Two independent bugs:

1. cp1252 — Excel autocorrects typed punctuation into smart quotes, curly apostrophes and dashes. cp1252 encodes those in 0x80–0x9F, exactly the range UTF-8 reserves for continuation bytes, so an en dash (–) is the lone byte 0x96 and UTF-8 rejects it as an invalid start byte. A cp1252 file containing only straight ASCII punctuation decoded fine, which is why this surfaced sporadically rather than on every Windows upload.

2. BOM — "CSV UTF-8" prefixes EF BB BF. field.strip().lower() does not strip , so the first header stayed question, failed the required-column check, and returned a misleading error naming a column the user could plainly see in their file. This also means the usual workaround advice ("re-save as CSV UTF-8") was itself broken.

Fix

decode_csv_bytes in services/evaluations/validators.py:

• Probe BOMs first, longest-first — the UTF-32-LE BOM starts with the UTF-16-LE BOM, so the reverse order misreads a UTF-32 file as UTF-16.
• Then fall through utf-8-sig → cp1252 → latin-1. utf-8-sig strips a BOM when present and is otherwise identical to utf-8, so rung one alone kills bug 2.
• cp1252 precedes latin-1 deliberately: latin-1 would decode 0x96 to U+0096, an invisible C1 control character, silently putting garbage in the dataset instead of the en dash the user typed. cp1252 recovers the intended character.
• latin-1 is total, so the ladder cannot fall through; the trailing HTTPException is there to keep the function total if that catch-all is ever removed.

No new dependency — codecs and the codecs themselves are stdlib. (chardet is present transitively in uv.lock, but an undeclared transitive is worse than a three-rung ladder that is provably correct for the encodings spreadsheets actually emit.)

Both upload routes already funnel through parse_csv_items (api/routes/evaluations/dataset.py → services/evaluations/dataset.py:88, dataset_v2.py → services/evaluations/fast.py:116), so v1 and v2 are both covered by the one change.

Checklist

• Ran uv run pytest app/tests/services/evaluations/ — 57 passed. Also mypy --strict and ruff clean on the touched files.
• Added test cases: 6 new cases in TestParseCsvItemsEncoding covering UTF-8, UTF-8+BOM, cp1252, UTF-16+BOM, UTF-32+BOM (guarding the probe order), and a byte undefined in cp1252 falling through to latin-1.
• Did not run the server. Verified instead by calling parse_csv_items directly against a byte-for-byte reproduction of the failing upload (0x96 at position 292, matching the AppSignal message exactly), plus cp1252 smart quotes and a curly apostrophe. All six rows now decode to the intended characters:

  What is the fee range? | The fee is 10 – 20 rupees
  Any quoted text?       | He said “hello” and left.
  Whose file is it?      | It’s the partner’s file.
  

Notes

• Behaviour change: files that previously 422'd now upload. Nothing that used to succeed decodes differently — utf-8-sig is byte-identical to utf-8 on BOM-less input, and the later rungs are only reached after utf-8-sig raises.
• Updated docs/wiki/modules/evaluations.md per the maintenance rule.
• Not addressed here: this only fixes the decode. Guessing a legacy encoding is a heuristic, so a file in some other single-byte codepage (e.g. ISO-8859-7 Greek) will now decode without error to the wrong characters rather than 422. That trade seems right for the actual user population, but flagging it.

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

• ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 54c17f3c-e113-48da-8d36-f1e7bdf06b2f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

OpenAPI changes   ⚪ No API surface changes

Note

This PR does not modify the API contract.

_{main ↔ cd7d52a3 · generated by oasdiff}