Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/evals/experiments.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Experiments
description: Run a prompt version or your own code on each item of a dataset, score the outputs, and compare them with the baseline in the dashboard or in CI.
lastModified: 2026-10-30
lastModified: 2026-10-31
---

An experiment runs a prompt version or your code on each item of a [dataset](/evals/datasets), then scores each output with [evaluators](/evals/evaluators). The experiment page compares the scores with the baseline of the dataset, so you can see which items became better or worse.
Expand Down Expand Up @@ -212,7 +212,7 @@ jobs:
with:
node-version: 24
- run: npm ci
- uses: telemetry-dev/telemetry/actions/experiment@main
- uses: telemetry-dev/telemetry/actions/experiment@experiment-action-v1
with:
dataset: support-golden
script: evals/reply.eval.ts
Expand All @@ -235,7 +235,7 @@ The action sets `TELEMETRY_DEV_ENVIRONMENT` to `experiment` if the job does not

The comment shows the status of each experiment and the count of items that ran. Its table of scores has the **Value**, the **Change**, and the **Pass mark** of each score. The action writes one comment for each workflow, job, and input, and changes that comment on each later push.

To keep a workflow the same when `main` changes, use a commit SHA of `telemetry-dev/telemetry` in place of `@main`.
The tag `experiment-action-v1` moves to each 1.x release of the action, and a change that can break a workflow gets a new major tag. To keep a workflow the same, use the tag of one release, such as `experiment-action-v1.0.0`, or its commit SHA. The [releases](https://github.com/telemetry-dev/telemetry/releases) of `telemetry-dev/telemetry` list the changes.

## Stop, run again, or delete an experiment

Expand Down
6 changes: 3 additions & 3 deletions docs/platform/analytics.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Analytics
description: Track a project's request volume, latency, errors, tokens, and cost over time.
lastModified: 2026-10-26
lastModified: 2026-10-31
---

The **Overview** page shows how a project is doing over a period: how many requests it served, how fast, how many failed, and what the model calls cost. It combines trace data with the OTLP metrics that your SDKs report.
Expand All @@ -27,7 +27,7 @@ Each card shows a total for the period and a chart of the value over the period:
| **LLM tokens** | Input tokens plus output tokens |
| **Active users** | Number of distinct user IDs in the traces |

The charts use hourly points for the 7-day period and daily points for longer periods. Select a card to open **Traces**; the **Error rate** card shows only traces with errors.
The charts use hourly points for the 7-day period and daily points for longer periods. Select a card to open **Traces** for the same period; the **Error rate** card shows only traces with errors.

Each card except **Active users** compares the period with the period of the same length just before it. For **Error rate**, **p95 latency**, **LLM cost**, and **LLM tokens**, a decrease counts as an improvement.

Expand All @@ -37,7 +37,7 @@ A comparison appears only if the project's retention covers both periods. For ex

The **Breakdown** table groups traces by **Model** at first. Select **Provider**, **User**, or **Operation** to change the grouping. Each row shows the group's **Requests**, **Error rate**, **p95**, **Tokens**, and **Cost**.

The table shows up to 6 groups, the ones with the most requests first. Select a row to list its traces, or its spans for a model, provider, or operation.
The table shows up to 6 groups, the ones with the most requests first. Select a row to list its traces, or its spans for a model, provider, or operation. The list opens with the period of the Overview. The rows of models and providers count only model calls, so their lists show only model calls too, with the **Operation: model calls** chip. Read [Find a trace or span](/platform/traces#find-a-trace-or-span).

## Read metric series

Expand Down
16 changes: 14 additions & 2 deletions docs/platform/mcp-server.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Connect an AI agent
description: Let Claude, ChatGPT, Cursor, VS Code, or another MCP client read traces, logs, and usage from one workspace.
lastModified: 2026-10-30
lastModified: 2026-10-31
---

telemetry.dev runs an MCP server at `https://telemetry.dev/mcp`. An AI agent connected to it can find traces, read a trace's spans and errors, list logs, and summarize token usage and cost, so you can ask questions such as "why did the last checkout run fail?" in your editor or chat app.
Expand Down Expand Up @@ -63,7 +63,7 @@ If the page says **telemetry.dev has not verified this app**, the app's name was
| `list_projects` | The workspace's projects. Every other tool needs a project from this list. |
| `search_sessions` | Sessions with traces in a time window, newest first, with trace count, errors, tokens, and cost. |
| `search_traces` | Traces in a time window, newest first, filtered by name, status, environment, framework, service, session, or end user, or by the model, provider, operation, agent, or tool of one of their spans. The tool filter matches a tool call by its recorded tool name, or by the span name when there is none. |
| `get_trace` | One trace's spans with models, tools, timing, tokens, cost, and error text, and its logs, in pages. It also returns `trace.url`, the trace's page in the dashboard, so an agent can give you a link to the run. |
| `get_trace` | One trace's spans with models, tools, timing, tokens, cost, and error text, and its logs, in pages. Next to its `cost_usd`, each span has `input_cost_usd`, `cache_read_cost_usd`, `cache_write_cost_usd`, `reasoning_cost_usd`, and `output_cost_usd`, which add up to it; they are null for a span without this split, such as one whose cost the SDK reported. Read [Cost by token class](/reference/cost#cost-by-token-class). It also returns `trace.url`, the trace's page in the dashboard, so an agent can give you a link to the run. |
| `get_span_payload` | The captured input, output, or error of one span, in pages. Needs the extra permission below. |
| `list_logs` | Log records in a time window, newest first, filtered by level, with counts per level for the whole window. |
| `get_usage_summary` | Requests, errors, latency, tokens, cost, and active users for 7, 30, or 90 days, compared with the period before, broken down by model, provider, end user, or operation. The breakdown lists the top 6 groups by requests. |
Expand Down Expand Up @@ -92,6 +92,18 @@ While a result's `next_cursor` is not null, pass it as `cursor` to get the next

`search_sessions` lists sessions by the start of their latest trace in the window. A trace that started before the window ended but reached telemetry.dev after the first page was read can move its session to a page already read, so a session can be missed or listed twice. `list_logs` counts levels over the whole window on every page, so a late log can change the counts between pages.

### Errors

Each tool that takes arguments refuses one that it does not know, so a misspelled filter returns an error instead of results without that filter.

A tool error has a text that the agent reads and, for a client that acts on it, a code in `_meta`:

```json
"_meta": { "dev.telemetry/error": { "code": "invalid_cursor", "retryable": false } }
```

The codes are `invalid_cursor`, `payloads_not_allowed`, `access_denied`, `project_not_found`, `trace_not_found`, `span_not_found`, `not_found`, `invalid_arguments`, `plan_limit`, and `server_error`. Only `server_error` is retryable. An argument that fails the tool's input schema returns an error without a code, so treat a missing code as unknown.

### Usage reads that fail

`get_usage_summary` still answers when one of its reads fails. `data_incomplete` is `true` when any read failed, such as the read of the period before. `current_unavailable` is `true` when the totals of the range were not read, and `breakdown_unavailable` is `true` when the breakdown was not read. In those cases the zeros in `current`, or the empty `breakdown`, are not data, so read them as unknown rather than as no usage.
Expand Down
15 changes: 14 additions & 1 deletion docs/platform/projects.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Projects
description: Create a project, connect an application to it, and rename or delete it.
lastModified: 2026-10-26
lastModified: 2026-10-31
seo:
description: "Use projects to keep telemetry, API keys, environments, and privacy settings separate. Create, set up, rename, or delete a project in telemetry.dev."
---
Expand Down Expand Up @@ -67,6 +67,19 @@ Select **Settings** in the project sidebar to open the project's **General** set

Deleting a project removes the project and its API keys, so the keys stop working at once. It also deletes the project's stored telemetry. You cannot undo it.

## Stop accepting telemetry

A workspace owner or admin can stop a project from taking new data and keep what it has. Select **Settings** in the project sidebar, and turn off **Accept telemetry** under **Ingestion**.

While it is off:

- The project's API keys get `403 project_not_accepting_telemetry` when they send traces, logs, or metrics, start an experiment, send experiment runs, or send scores, and nothing new is stored. Read [OTLP error responses](/reference/otlp#error-responses) and [Evals API errors](/reference/evals-api#errors).
- Apps can still fetch prompts and datasets.
- The project's stored data stays.
- telemetry.dev does not score the project. Live and session evaluators, backfills, and experiment items wait, and LLM judges start no new model calls, apart from up to a minute of calls in a scoring run that began before you turned it off. An experiment stays queued or running until you turn the setting on again or stop the experiment.

When you turn **Accept telemetry** on again, the waiting evals continue where they stopped. Live and session evaluators look back at most one hour, so if the setting was off for longer, traces that were not scored before it was turned off stay unscored.

<span id="rename-the-organization" />

## Rename the workspace
Expand Down
8 changes: 7 additions & 1 deletion docs/platform/prompts.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Prompts
description: Keep prompts in telemetry.dev, save versions, deploy one with a label, fetch it with getPrompt or get_prompt, and see its calls on traces.
lastModified: 2026-10-30
lastModified: 2026-10-31
---

Prompts keeps the prompts of your app in telemetry.dev. You save each change as a version, and a label such as `production` says which version your app gets. To deploy a version, you move the label. Your code does not change.
Expand Down Expand Up @@ -45,6 +45,8 @@ In a chat prompt, a **Chat history** message stands for a list of messages that

Only text prompts can be included. Includes can nest at most 3 levels deep, one version can include at most 50 prompts, and includes can add at most 1,000,000 characters. telemetry.dev puts in the included text when your app fetches the prompt, so a move of a label of the included prompt changes the text that your app gets. A save, a label move, or a rename that makes includes loop back or nest too deep is refused.

You can save a version with an include that gets no version, such as one that names a label that does not exist yet, and the editor warns you. A label cannot point to such a version. Apps that fetch it with `latest` or by its version number get `include_not_found` until the include gets a version.

## Save versions

Each change to the prompt is a draft until you save it. Select **Save as v**_n_, enter an optional note in **What changed**, and select **Save version**. In the same dialog you can move labels to the new version.
Expand All @@ -61,6 +63,8 @@ A label points to one version of the prompt. Your app fetches the prompt by its
- **latest** is not a label that you set. It always gives the newest version.
- Other labels, such as `staging`, are yours to name. Open **Manage labels** in the **Versions** tab to add, move, or remove a label.

Removing a label asks first and says which apps will get an error, including old versions of other prompts that include this prompt through that label. The removal is refused while the newest version or a labeled version of another prompt includes this prompt through that label.

Until a version has the production label, apps that fetch the prompt without a label get the error `label_not_found`.

### Protected labels
Expand All @@ -71,6 +75,8 @@ Until a version has the production label, apps that fetch the prompt without a l

When you promote a version to production, select **Part of the calls** and enter a share from 1 to 99 percent. That share of the calls gets the new version, and the rest get the version that production had. One session keeps one version for all of its calls. Promote the new version again with **All calls** to end the rollout, or roll production back to an earlier version.

Moving the label to one version, or removing it, ends the rollout. **Manage labels**, **Promote to production**, **Roll back**, and the Save dialog show the rollout and say that the change ends it. A rollout reaches only apps that fetch this prompt. A prompt that includes it through the label gets the version that the label had before the rollout.

## Rename, move, or delete a prompt

Open the prompt menu in the row of the prompt or on the prompt page:
Expand Down
Loading