fix(pylon): derive fallback throughput from observed requests

[barrygreengus]

Why

Pylon deployments with --engine-stats-stream off do not receive authoritative engine statistics. Without a request-derived fallback, request load remains on bootstrap values and output throughput cannot reliably use observed or exact usage.

This throughput slice is stacked on #1452 and uses the shared-cluster statistics contract from #1448.

What changed

• Estimate streamed output from cumulative ASCII and non-ASCII character counts so chunk fragmentation does not change the result.
• Keep raw output units separate from displayed output tokens, and allow trusted exact usage to correct estimates without accepting regressing counters.
• Derive input TPS from a bounded, request-keyed union of submission-to-first-output intervals, including exact-token correction, eviction, overlap, and reused request IDs.
• Record one output TPS sample for each successful terminal request. Failed requests add no completed sample, and final exact usage wins over live estimates.
• Preserve duration-based throughput for callers of the public PylonRuntimeState::observe_request API.
• Retain the public configured-TPS pin field for source compatibility while removing internal pin authority after valid observations.
• Add the default-off --force-chat-completions-include-usage option for direct and reverse Pylon modes.
• Validate and rewrite only streaming Chat Completions JSON, remove stale Content-Length after mutation, and leave already-enabled bodies byte-identical.
• Bound blocking JSON rewrites with one semaphore shared by cloned tunnel applications. Request-body I/O and validation occur before permit acquisition. Queued blocking work is cancelled when its request handler is dropped.
• Advertise request.load.proxy_local even when current request load is zero.

Customer Release Notes

Pylon can derive input and output throughput from observed requests when the engine statistics stream is disabled. Operators can also opt in to exact streaming Chat Completions usage requests.

Plan Summary

No infrastructure or resource changes.

Deployment order:

1. fix(stargate): aggregate proxy-local Pylon stats #1448 shared Stargate aggregation.
2. fix(pylon): derive request phases from generated output #1452 request lifecycle foundation.
3. This throughput slice.
4. feat(pylon): calibrate fallback output token estimates #1459 optional single-Pylon output-token calibration.

Usage

Enable exact streaming Chat Completions usage requests with:

pylon --force-chat-completions-include-usage ...

The option defaults to disabled.

Testing

• cargo test -p pylon-lib: 448 unit tests and 4 public API tests passed on this branch.
• cargo test -p pylon: 67 unit tests and 1 CLI test passed on this branch.
• cargo clippy -p pylon-lib -p pylon --all-targets -- -D warnings: passed.
• Combined four-PR integration: cargo test --workspace passed 1,545 tests with 3 ignored.
• Combined integration: cargo test -p stargate passed 571 tests.
• Combined integration: strict Pylon and Stargate Clippy, focused Rust formatting, and diff checks passed.
• Fresh final integrated Rust, Tokio, and YAGNI reviews found zero actionable findings.

No separate QA environment is required for this code-only slice. The unchanged Pylon library Criterion benchmark target was not run under --all-targets because it exceeds the local test timeout.

Notes

The rewrite worker limit applies only to CPU work. Slow request uploads do not reserve worker capacity. Each body remains independently bounded by the existing request-body limit.

Full-workspace cargo fmt --all -- --check reports only four inherited formatting differences in unchanged Stargate files: crates/stargate/src/http_proxy.rs, crates/stargate/src/main.rs, crates/stargate/src/runtime.rs, and crates/stargate/tests/suite/health_lifecycle.rs. Every changed Rust file passes formatting checks.

No dependency, license, NOTICE, protobuf, generated-file, metric-name, span-name, dashboard, or alert changes are included.

Issues

Relates to #1447

References

• Aggregate proxy-local Pylon stats for shared Stargate clusters #1447

Related Pull Requests

• Parent lifecycle slice: fix(pylon): derive request phases from generated output #1452
• Shared Stargate prerequisite: fix(stargate): aggregate proxy-local Pylon stats #1448
• Optional calibration child: feat(pylon): calibrate fallback output token estimates #1459

Dependencies

None. No license review or NOTICE update is required.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f84e3976-2708-4d12-96c1-5cbd53a14efd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}