Migrate BCal evaluation to M365 LLM API

[martinsrui-msft]

Summary

• Replaces the temporary CAPI bridge with a thin BCal external-command bridge over bc-eval==0.6.1 and the direct M365 llm_api provider.
• Removes CAPI certificate staging, CAPI secrets, the [capi] extra, and --use-capi; both workflow installs use plain bc-eval==0.6.1 and scoring uses --llm-provider llm_api.
• Reuses GitHub Actions Azure OIDC authentication. The workflow identity must have Key Vault Crypto User access to the configured/default LLM API certificate vault.
• Preserves resilient matrix aggregation, exact completeness reporting, timeout rows scored with test_passed=0, and HTTP 5xx infrastructure rows uploaded with a null core score while the individual matrix job remains failed.
• Bumps BC-Bench to 0.15.0 and updates the directly related configuration, documentation, and tests.

This is the permanent successor to temporary DO NOT MERGE PR #958. It does not modify #958 or stack #945.

Upstream dependency

The direct API implementation merged in bic/BC-LanguageModelTools#218. bc-eval==0.6.1 is now published in BC_PythonFeed; it repackages the same direct-API release after bic/BC-LanguageModelTools#223 fixed Twine compatibility and release/tag success gating. The direct M365 LLM API contract is unchanged.

Workflow run 37789235996 failed all entries solely because the workflows requested nonexistent bc-eval==0.6.0. Follow-up commit 1ad5f078 changes both workflow pins and directly related diagnostics/tests/docs to the published 0.6.1.

Validation

• Authenticated plain bc-eval==0.6.1 install on Python 3.13 succeeds and exposes the expected create_openai_compatible_client signature.
• 0.6.1 follow-up: 70 focused tests passed, plus Ruff check/format, strict Ty, full pre-commit, uv lock, and git diff --check.
• Original migration: 1173 passed, 1 skipped, 1 deselected in the full suite; 203 focused tests passed.
• Provider signature, bridge JSON contract, parameter translation, typed HTTP 5xx handling, timeout/null-score behavior, completeness, and workflow assertions pass.
• Confirmed no stale intentional 0.6.0 references and no runtime CAPI references, [capi] installs, --use-capi, or 0.3.14 pins remain.
• No expensive/full BCal evaluation workflow was dispatched manually.

Keep this PR in draft until the normal push-triggered checks for 1ad5f078 pass.

[github-code-quality]

Code Coverage Overview

Languages: Python

Python / code-coverage/pytest

The overall line coverage in commit 1ad5f07 in the martinsrui-bcal-llm-... branch is 87%. The line coverage in commit 92af447 in the main branch is 86%.

Show a line coverage summary of the most impacted files.

File	main 92af447	martinsrui-bcal-llm-... 1ad5f07	+/-
src/bcbench/commands/result.py	93%	89%	-4%
src/bcbench/results/display.py	93%	92%	-1%
src/bcbench/com...nds/evaluate.py	77%	77%	0%
src/bcbench/results/base.py	94%	94%	0%
src/bcbench/com...ands/dataset.py	25%	26%	+1%
src/bcbench/evaluate/base.py	91%	93%	+2%
src/bcbench/age...t/bcal/agent.py	86%	89%	+3%
src/bcbench/exceptions.py	71%	75%	+4%
src/bcbench/age...m_api_bridge.py	0%	84%	+84%
src/bcbench/res...completeness.py	0%	96%	+96%

---

_{Updated October 09, 2026 07:58 UTC}