Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions packages/ai/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,38 @@ Alongside spans, each run emits structured logs (shown on the trace and the Logg
Logs are emitted as OTLP span events on the owning span (sent on the same `/v1/traces` request), so
they correlate to the turn/tool they describe.

## Testing

To check your telemetry in tests without a network, pass a second argument to `telemetryDev()`.
`sendSpans` receives each flushed batch of sampled spans instead of posting it, and
`recordDuration` and `recordTokens` receive each metric point instead of recording it. The media
and batch wrappers, and the `/v6` entry, take the same object as their last argument.

```ts
import { telemetryDev } from "@telemetry-dev/ai-sdk";

const spanNames: string[] = [];
const tokens: number[] = [];

const telemetry = telemetryDev(
{ apiKey: "td_live_test" },
{
sendSpans: async (batch) => {
spanNames.push(...batch.map((span) => span.name));
},
recordDuration: () => {},
recordTokens: (_type, count) => {
tokens.push(count);
},
},
);
```

The integration still needs an API key, or it does nothing; any non-empty value works. Each
function you leave out keeps its default, so pass all three to keep the test offline. An error
thrown by `sendSpans` goes to `onError`. Each span in a batch is an OpenTelemetry `ReadableSpan`, so
a test can also read its `attributes` and `events`.

## Limitations

On `ai@7` (root entry):
Expand Down
2 changes: 0 additions & 2 deletions packages/ai/tests/integration.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,6 @@ interface MetricRecord {
const requestUrl = (input: string | Request | URL): string =>
input instanceof Request ? input.url : input.toString();

// Inject a capturing transport via the otel.ts test seam: spans are captured as the real
// ReadableSpans the emitter would serialize, and metric points are captured in-process.
function makeCapture() {
const spanBatches: ReadableSpan[][] = [];
const metrics: MetricRecord[] = [];
Expand Down
6 changes: 5 additions & 1 deletion packages/otel/src/generation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,11 @@ export interface GenerationEmitter {
flush(spans: Span[]): Promise<void>;
}

/** Replaces span export or metric collection without changes to sampling and flush behavior. */
/**
* Replaces span export or metric collection without changes to sampling and flush behavior.
* `sendSpans` receives each flushed batch of sampled spans, and an error it throws goes to
* `onError`. A function left out keeps exporting to telemetry.dev.
*/
export interface GenerationEmitterOverrides {
sendSpans?: (spans: ReadableSpan[]) => Promise<void>;
recordDuration?: (seconds: number, attributes: Attributes) => void;
Expand Down
31 changes: 31 additions & 0 deletions packages/tanstack-ai/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,37 @@ telemetryDev({ waitUntil: (p) => ctx.waitUntil(p) });
- **Metrics:** `gen_ai.client.operation.duration` per iteration and tool call, and
`gen_ai.client.token.usage` per iteration, both following the OTel GenAI semantic conventions.

## Testing

To check your telemetry in tests without a network, pass a second argument to `telemetryDev()`.
`sendSpans` receives each flushed batch of sampled spans instead of posting it, and
`recordDuration` and `recordTokens` receive each metric point instead of recording it.

```ts
import { telemetryDev } from "@telemetry-dev/tanstack-ai";

const spanNames: string[] = [];
const tokens: number[] = [];

const telemetry = telemetryDev(
{ apiKey: "td_live_test" },
{
sendSpans: async (batch) => {
spanNames.push(...batch.map((span) => span.name));
},
recordDuration: () => {},
recordTokens: (_type, count) => {
tokens.push(count);
},
},
);
```

The middleware still needs an API key, or it does nothing; any non-empty value works. Each
function you leave out keeps its default, so pass all three to keep the test offline. An error
thrown by `sendSpans` goes to `onError`. Each span in a batch is an OpenTelemetry `ReadableSpan`, so
a test can also read its `attributes` and `events`.

## Limitations

- **TTFT:** time-to-first-chunk is not currently reported.
Expand Down
2 changes: 0 additions & 2 deletions packages/tanstack-ai/tests/middleware.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,6 @@ interface MetricRecord {
attributes: Attributes;
}

// Inject a capturing transport via the otel.ts test seam: spans are captured as the real
// ReadableSpans the emitter would serialize, and metric points are captured in-process.
function makeCapture() {
const spanBatches: ReadableSpan[][] = [];
const metrics: MetricRecord[] = [];
Expand Down
Loading