Skip to content

fix(sdk): name why exports fail and stop experiments on a revoked key - #193

Merged
aikins01 merged 1 commit into
mainfrom
fix/evals-key-errors
Nov 3, 2026
Merged

aikins01 merged 1 commit into
mainfrom
fix/evals-key-errors

Conversation

@aikins01

@aikins01 aikins01 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

A refused export now says why. With a key that has Run experiments but not Send telemetry, the first failed trace export used to report only telemetry.dev trace ingest failed: 403 (TypeScript) or OTLP span export failed (check the API key and ingest URL) (Python). Both SDKs now report telemetry.dev trace ingest failed: 403 This API key cannot send telemetry. Create a key with Send telemetry access., the message of the ingest's JSON or protobuf Status.

  • Error reason: the TypeScript transport reads at most 4096 bytes of a refused response with a JSON or protobuf content type, then cancels the body, and throws the new exported IngestError with the status. Another content type, or a body that can't be read, keeps the old message. Python gets the response from a requests session hook on its OTLP exporters and raises the same message as an IngestError; an export with no response keeps the generic message.
  • Revoked key: runExperiment and run_experiment (sync and async) start no more items after the ingest refused an export with 401, when the telemetry client sends with the key and base URL of the run. They then complete the experiment and reject or raise invalid_api_key. Before, a run whose project was deleted kept running every item and failed only at the end. A 403 does not stop the run, since the key still works for the experiment API.
  • Runs without traces: the results line for the experiment action gets traced, false without an enabled client or after a 401 or 403 refusal, and tracesError with the refusal. The job summary and comment then say the runs have no traces, with the ingest's reason, or with the init() hint when there was no client. An outage (5xx, timeouts) does not count.
  • Agent and tool names: a TypeScript type: "agent" or type: "tool" span without agentName or toolName now takes its span name, as Python already did. In both SDKs a span named after its operation (invoke_agent, invoke_agent AGENT123, execute_tool) gets no name, so the Bedrock agent spans keep none.
  • Docs: the READMEs drop the claim that a key for another environment can't send the experiment traces (a key with Run experiments can), and say with a traceId a score belongs to that trace's session, and that a session's trace keeps the environment of its first turn. The action README and docs/sdk-conformance.md match.

Python now lists requests>=2.28 and googleapis-common-protos>=1.63.1 as direct dependencies. The OTLP exporter already required both, with floors that do not import on Python 3.10 (requests 2.7) or with protobuf 5 (googleapis-common-protos before 1.63.1). Because the SDK passes its own session to the exporters, the exporters no longer read the OpenTelemetry credential provider variables; that is #192.

Tests run:

  • vp test run in packages/otel (102 passed), packages/sdk (320), packages/bedrock (88), packages/eve (50), packages/ai (98), packages/opencode (24), packages/omp (20) and packages/pi (24).
  • vp check: one type error in packages/omp/tests/omp.test.ts and one lint warning in packages/sdk/tests/capture.test.ts, both also on main in my local setup.
  • py:check for the 7 Python packages, pytest in sdks/python (474 passed), and .github/scripts/python_dependency_bounds.sh for all 7 packages.
  • The product repository's TypeScript and Python OTLP e2e tests against this branch (4 passed).

Each new assertion fails when the code it covers is removed.

A refused OTLP export now carries the reason the ingest gave, such as
"This API key cannot send telemetry", in both SDKs. runExperiment and
run_experiment stop starting items after a 401 for the key and base URL
of the run, and the results line tells the experiment action when the
runs sent no traces. Agent and tool spans in TypeScript take the span
name as their agent or tool name, as in Python.
@aikins01 aikins01 self-assigned this Nov 3, 2026
@aikins01
aikins01 merged commit 012b778 into main Nov 3, 2026
10 checks passed
@aikins01
aikins01 deleted the fix/evals-key-errors branch November 3, 2026 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant