Harden BCal aggregation against shard setup failures

[martinsrui-msft]

Summary

This PR hardens the BCal evaluation workflow against matrix shards that fail before BC-Bench can write a result. It preserves the M365 LLM API migration in #962 and is the top PR in native stack #964 (#962 -> #963).

It intentionally excludes the cancelled validation-only/Kusto experiment and all unrelated local dashboard changes.

Incident

• Evaluation run 37908218345
• Failing job 113748785670
• Failed matrix entry: nl2al__sales-order-rush-flag-1
• The shard failed during Python/UV setup, before bcbench evaluate bcal started.
• Exact root cause: uv sync failed downloading urllib3==2.8.0 from the Azure Artifacts blob after UV exhausted its 3 internal retries. The final transport error was an HTTP/2 stream no longer needed failure.
• The always-running artifact step then warned because evaluation_results/**/*.jsonl did not exist. The matrix expected 110 entries, but only 109 result artifacts were available.

Behavior after this change

• Python/UV setup gets one bounded outer retry. The first setup step uses continue-on-error only so the workflow can wait 20 seconds and attempt setup once more. The retry itself is not suppressed: if it also fails, the original UV failure remains visible and the matrix shard stays failed.
• Summarization runs after any non-cancelled matrix outcome and receives the exact expected entry IDs, expected count, and attempt-scoped artifact pattern.
• A missing setup-time result is synthesized as an explicit infrastructure-failure row. For the incident shape, completeness becomes expected 110 / produced 110 / missing 0 / infrastructure failures 1. The matrix failure remains visible; synthesis does not turn the failed shard green.
• Synthetic rows carry infrastructure-error metadata (provider, status when available, and error message), preserve the expected dataset payload, and emit a null test_passed score rather than fabricating a pass or fail.
• Ordinary evaluated rows remain strict. The LM Checklist wrapper places a nonnumeric per-row core-score sentinel before judging and replaces it only after exactly one numeric, non-null test_passed score is returned. If judging raises or omits that score, BC Eval conversion fails the summarization command before storage.
• Completeness is an exact-set gate: missing, duplicate, or unexpected entry IDs prevent publication. Only a complete expected set proceeds to scoring/storage.
• Full runs upload the complete result set to Braintrust and Kusto. Test runs execute the same scoring and test_passed validation but stop before external storage.

This builds on the aggregation hardening from #944 while covering the remaining pre-evaluation setup-failure gap.

Validation

• Focused affected suite: 174 passed.
• Full equivalent tree: 1147 passed, 2 skipped, 1 deselected.
• Ruff check and format passed for changed Python files.
• Changed-file ty checks passed.
• Generic pre-commit hooks passed.
• git diff --check passed.
• Published tree 4f66a0dc9bbc7382cd072418431ce58827030c1a exactly matches the validated source tip.

[github-code-quality]

Code Coverage Overview

Languages: Python

Python / code-coverage/pytest

The overall line coverage in commit 830f866 in the martinsrui-publish-b... branch is 87%. The line coverage in commit 92af447 in the main branch is 86%.

Show a line coverage summary of the most impacted files.

File	main 92af447	martinsrui-publish-b... 830f866	+/-
src/bcbench/commands/result.py	93%	87%	-6%
src/bcbench/types.py	92%	92%	0%
src/bcbench/results/display.py	93%	93%	0%
src/bcbench/results/base.py	94%	94%	0%
src/bcbench/results/summary.py	96%	96%	0%
src/bcbench/com...ands/dataset.py	25%	26%	+1%
src/bcbench/com...nds/evaluate.py	77%	78%	+1%
src/bcbench/evaluate/base.py	91%	92%	+1%
src/bcbench/age...m_api_bridge.py	0%	79%	+79%
src/bcbench/res...completeness.py	0%	95%	+95%

---

_{Updated October 09, 2026 10:48 UTC}

[Sun Haoran (haoranpb)]

I think the PR might be a bit much to address the transient uv failure.

The current behavior: partial failures (e.g. one or more tasks out of the entire dataset) will fail the whole job.

This has been serving us well because it can mean some tasks are malformed, or upstream caused failures.

---

In this case, I guess it is a signal that we should invest a in the CI run stabilities, i.e. tasks fail but then succeed after one re-run. I will take a note of this and see if we could improve that for all workflows