fix(libsy): count routing-time model calls that stop the run

[grahamking]

When a routing-time model call failed, serve in libsy-llm-client returned the error from drive. That cancelled the algorithm task before Driver::call_model could record the call. So switchyard.llm_calls and switchyard.llm_call_duration_ms missed these failed calls.

The host now records call metrics when it finishes a CallModel.

• CallModel::respond records the call, then sends the result to the algorithm.
• New CallModel::fail records a failed call and returns the error to stop drive. serve uses it when a model call fails.
• Drop for CallModel records a call that was never answered as an error. This covers hosts that drop the call without calling respond or fail.
• Driver::call_model still records the outcome and token counts on its libsy.llm_call span.

Fixes: https://linear.app/nvidia/issue/SWITCH-1544

Assisted-by: Pi:GPT 6 Astra medium
Assisted-by: Claude:Opus 5.5 medium
Signed-off-by: Graham King grahamk@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Model-call failures can now be handled by routing logic without immediately stopping the driver.
  • Unresolved or failed calls are consistently recorded as failures, while successful responses remain marked as successful.

• Observability

  • LLM call metrics now consistently report successful and failed outcomes.
  • Classifier calls record the selected model, task, and outcome, including call and response-decoding errors.
  • Call timing, response status, and token details are recorded more reliably.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-830/
Built to branch gh-pages at 2026-09-22 20:53 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA-NeMo/Switchyard/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fc0a1f73-27ef-4a70-b837-1b08e9cf21c9

📥 Commits

Reviewing files that changed from the base of the PR and between bec0cb5 and 959c377.

📒 Files selected for processing (4)

• crates/libsy-llm-client/src/run.rs
• crates/libsy-llm-client/tests/observability.rs
• crates/libsy/src/core/algorithm.rs
• crates/libsy/src/observability.rs

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

Walkthrough

The change separates recoverable model-call failures from routing-stopping failures. CallModel records call outcomes and dropped calls. Observability now records call metrics separately from span outcomes and token usage. Serving and classifier tests cover the new behavior.

Changes

Call Failure Observability

Layer / File(s)	Summary
Observability recording split crates/libsy/src/observability.rs	Call metrics now use an explicit success flag. Span outcomes and available token usage are recorded separately from the resolved response.
CallModel lifecycle tracking crates/libsy/src/core/algorithm.rs	CallModel records start time, success, failure, and unresolved dropped calls. Driver::call_model uses the shared span recorder. Documentation describes the separate failure paths.
Serving failure routing and tests crates/libsy-llm-client/src/run.rs, crates/libsy-llm-client/tests/observability.rs	serve passes call_one failures to call.fail. Tests verify one classifier metric per judge call and validate ok and error outcomes.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: ⚪ Minimal · up to 959c3

The failure paths preserve routing behavior and record the intended call metrics; no merge-blocking risk is established.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: recording routing-time model calls that fail and stop the run.
Docstring Coverage	✅ Passed	Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 4 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{A rabbit checks each call with care
Errors hop through the proper snare
Metrics mark the outcome bright
Spans record the ending right
Dropped calls leave a failure trail
The driver follows every trail}

_{Comment @coderabbitai help to get the list of available commands.}