Skip to content

fix(sdk): name why exports fail and stop experiments on a revoked key #193

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aikins01 merged 1 commit into from
Nov 3, 2026
Merged

fix(sdk): name why exports fail and stop experiments on a revoked key #193

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions actions/experiment/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,17 +70,18 @@ changes of each release.
| `command` | A shell command that runs the eval script, for example `uv run python evals/reply_eval.py` or `pnpm eval`. |
| `dataset` | The dataset to run on. The action puts it in `TELEMETRY_DEV_EXPERIMENT_DATASET`. When the script also names a dataset, the two must be the same. |
| `min-scores` | Pass marks, as `name=value` pairs that commas or new lines separate. A pass rate is 0 to 1, and a mean is on the scale of its score. The action puts them in `TELEMETRY_DEV_EXPERIMENT_MIN_SCORES`, and they win over the `minScores` of the script for the same name. |
| `api-key` | A telemetry.dev API key with the `datasets:read` and `experiments:write` scopes, for all environments or for `experiment`. Default: the `TELEMETRY_DEV_API_KEY` variable of the job. |
| `api-key` | A telemetry.dev API key with the `datasets:read` and `experiments:write` scopes (Run experiments) and, for the traces of the items, Send telemetry. Default: the `TELEMETRY_DEV_API_KEY` variable of the job. |
| `base-url` | The telemetry.dev API. Default: `https://api.telemetry.dev`. |
| `working-directory` | The directory that the script or the command runs in. Default: `.`. |
| `comment` | `true` or `false`. Set to `false` to write the job summary only. Another value gets a warning and counts as `true`. Default: `true`. |
| `github-token` | The token that writes the comment. Default: `github.token`, which needs the `pull-requests: write` permission. When it is empty on a pull request, the action writes no comment and logs a warning. |

The action sets `TELEMETRY_DEV_ENVIRONMENT` to `experiment` when the job does not set it, so the
traces of the items stay out of your production pages. Thus the key must be for all environments
or for `experiment`. A key for another environment, for example your production key, cannot send
the traces of the items: the experiment then completes with its outputs, but with no traces and no
cost. `CI` is `true` in GitHub Actions, thus the SDK runs in CI mode: it records the repository,
traces of the items stay out of your production pages. A key with Run experiments sends the
`experiment` environment even when it is for another environment, but it needs Send telemetry for
the traces of the items. Without it, the experiment completes with its outputs, but with no traces
and no cost, and the job summary says that the runs have no traces, with the reason of the ingest.
`CI` is `true` in GitHub Actions, thus the SDK runs in CI mode: it records the repository,
the commit, the run, and the pull request with the experiment, and it exits with an error when the
check fails.

Expand Down
23 changes: 22 additions & 1 deletion actions/experiment/report.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
// Writes the job summary of the experiment action, and the comment with the result on the pull
// request. runExperiment and run_experiment add one line of JSON to the results file for each
// experiment: the body of GET /v1/experiments/{id}, with `ci`, false when the CI mode was off, and
// `runsPerItem`. Lines from SDKs before those fields have neither.
// `runsPerItem`, and `traced`, false without a telemetry client or when the ingest refused its
// exports (401 or 403), with `tracesError` for a refusal. Lines from SDKs before those fields do
// not have them.
import { createHash } from "node:crypto";
import { appendFileSync, readFileSync } from "node:fs";

Expand Down Expand Up @@ -180,6 +182,25 @@ function render(experiments, code) {
lines.push("", "The CI mode of this experiment was off, thus it does not fail the job.");
}

if (experiment.traced === false) {
const missing =
"The runs have no traces: the items have no trace to open, and Cost leaves out the model calls of your code.";

if (typeof experiment.tracesError === "string" && experiment.tracesError !== "") {
lines.push(
"",
`${missing} The ingest refused them:`,
"",
`> ${text(experiment.tracesError)}`,
);
} else {
lines.push(
"",
`${missing} No telemetry client sent them: before the experiment runs, call \`init({ environment: "experiment" })\` in TypeScript, or \`init(environment="experiment")\` in Python, with an API key.`,
);
}
}

if (typeof experiment.error === "string" && experiment.error !== "") {
lines.push("", `> ${text(experiment.error)}`);
}
Expand Down
14 changes: 8 additions & 6 deletions docs/sdk-conformance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ repository, not this SDK workspace. Paths below refer to that product repository
| `finishReason` | `gen_ai.response.finish_reasons = [value]` | single-element array |
| sampling params | `gen_ai.request.{temperature,top_p,top_k,max_tokens,stop_sequences,seed,frequency_penalty,presence_penalty}` | flat fields |
| `timeToFirstChunkMs` / `time_to_first_chunk_ms` | `gen_ai.response.time_to_first_chunk` | **seconds** on the wire (key has no `ms` suffix; ingest multiplies by 1000) |
| `toolName`, `toolCallId`, `toolDescription` | `gen_ai.tool.{name,call.id,description}` | |
| `agentName`, `agentId` | `gen_ai.agent.{name,id}` | |
| `toolName`, `toolCallId`, `toolDescription` | `gen_ai.tool.{name,call.id,description}` | `gen_ai.tool.name` defaults to the span name of a `type: "tool"` span, unless it is `execute_tool` or starts with `execute_tool ` |
| `agentName`, `agentId` | `gen_ai.agent.{name,id}` | `gen_ai.agent.name` defaults to the span name of a `type: "agent"` span, unless it is `invoke_agent` or starts with `invoke_agent ` |
| `metadata.<k>` (per span) | `td.metadata.<k>` | reserved keys `userId`/`sessionId`/`user_id`/`session_id` dropped with a debug warning; never passed through the mask |
| `propagateAttributes userId` | `user.id` on EVERY span/log in scope | stamped by the span processor's onStart |
| `propagateAttributes sessionId` | `gen_ai.conversation.id` on EVERY span/log in scope | never `session.id` |
Expand Down Expand Up @@ -367,8 +367,10 @@ the `dataset` option has no value, and a different `dataset` option is a `TypeEr
The value must match `-?\d+(\.\d+)?` (ASCII digits), else it is a `TypeError`. These marks win over
the `minScores` option for the same name. When `TELEMETRY_DEV_EXPERIMENT_RESULTS` names a file, the
SDK adds the JSON of the `GET /v1/experiments/{id}` body to it as one line, then `ci`, the CI mode
of the run (a boolean), and `runsPerItem`. The action fails the job for a failed experiment only
when its line has `ci` true.
of the run (a boolean), `runsPerItem`, and `traced`: false when there is no enabled client, or
when the ingest refused an export of the client with 401 or 403 since an export last reached it,
and then `tracesError`, the message of that export error. The action fails the job for a failed
experiment only when its line has `ci` true, and notes runs without traces in the job summary.

The SDKs print one line when the experiment starts and the result when it ends. The result and the
`check_failed` error show a rate as a whole percent and any other score, with its pass mark, to one
Expand Down Expand Up @@ -420,7 +422,7 @@ eventName: "e2e.inside")` inside it
`prompted-private` (the same with captureInput false)
2. plain span `background-job` started after the agent ended, with
`parent = <agent traceparent captured via getTraceparent()>`
3. agent span `follow-up-turn` (`agentName: "support"`) started after the agent ended with no
3. agent span `follow-up-turn` (no `agentName`) started after the agent ended with no
active parent: a second root call of the same session
4. standalone `log("outside spans", level: "warn", eventName: "e2e.outside")`
5. `flush()` + `shutdown()`
Expand Down Expand Up @@ -462,7 +464,7 @@ do not sanitize raw OpenTelemetry events or arbitrary attributes.
| C17 | Exact span census (20). Without a key the SDK sends nothing and does not throw; the census cannot see that, so the TypeScript test counts captured requests and the Python scenario checks only that nothing raised |
| C18 | Streaming generation: `timeToFirstChunkMs` stored as 250 ms; wire attribute `gen_ai.response.time_to_first_chunk` = 0.25 s |
| C19 | OpenRouter generation: provider stored as `openrouter` |
| C20 | Session trace: `follow-up-turn` (a second root call of the same session) lands in the same trace, parented to the session parent, started at or after the agent ended |
| C20 | Session trace: `follow-up-turn` (a second root call of the same session) lands in the same trace, parented to the session parent, started at or after the agent ended, with the agent name `follow-up-turn` |
| C21 | Process mode: two otherwise independent roots and a standalone log share one generated UUID session; a nested explicit-session span/log subtree consistently uses the explicit ID instead |
| C22 | Streaming truncation: `telemetry.dev.capture.truncated = true` survives OTLP transport and ingest on the streamed generation |
| C23 | Captured JSON matches an independently specified JSON.stringify byte sequence, including number spelling, key ordering, non-finite values and Unicode |
Expand Down
1 change: 1 addition & 0 deletions packages/otel/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ export {
createMetricExporter,
createTraceExporter,
type ExportFailures,
IngestError,
maybeGzip,
otlpHeaders,
type OtlpTarget,
Expand Down
Loading
Loading