Repository navigation
Merge latest main into v4 - #2680
Merged
Merged
Conversation
…ironment variables (#2600) ## Summary Empty `BRAINTRUST_API_URL` and `BRAINTRUST_PROXY_URL` environment variables are now ignored, and the SDK uses the API and proxy URLs from the organization's login response. The logger reads both variables with `iso.getEnv(name) ?? org.api_url`. `??` only falls back for `undefined`, so an empty value is used as the URL. Empty values are common: Docker Compose sets `BRAINTRUST_API_URL: ${BRAINTRUST_API_URL}` to `""` when the host variable is unset, and GitHub Actions does the same for an undefined `${{ vars.* }}`. The effects: - `BRAINTRUST_API_URL=` makes `login()`, `initLogger()` and `init()` fail with `Unable to log into any organization with the provided credential.`, even though the credential is valid. - `BRAINTRUST_API_URL=` makes `loadPrompt()` and `loadParameters()` with an explicit `apiKey` fail with the same error. - `BRAINTRUST_PROXY_URL=` sets `state.proxyUrl` to `""`, so proxy requests go to the API URL instead of the organization's proxy URL, and `BraintrustState.serialize()` throws `Cannot serialize BraintrustState without all login attributes`. This change uses `||` for these lookups, which matches how the same file already reads `BRAINTRUST_APP_URL` and `BRAINTRUST_APP_PUBLIC_URL`. The Python SDK already treats empty values of these variables as unset. ## Test plan - Added tests in `js/src/logger.test.ts`: - `loginToState` with empty `BRAINTRUST_API_URL` and `BRAINTRUST_PROXY_URL` uses the organization's `api_url` and `proxy_url`. - `loadPrompt` with an explicit API key and an empty `BRAINTRUST_API_URL` sends the prompt request to the organization's `api_url`. - Before the change, both new tests fail with `Unable to log into any organization with the provided credential.` After the change, both pass, and all 133 tests in the file pass. ```sh cd js && npx vitest run src/logger.test.ts -t "login with empty URL environment variables" ```
## AI Summary Implements ONLY [span-customizer Hook failures](https://github.com/braintrustdata/braintrust-spec/blob/main/skills/instrumentation-spec/references/features/span-customizers.md#hook-failures). - Exceptions and invalid returns stop the chain and export only original identity, routing/protocol fields, and existing duration timestamps, with context replaced by { customizer_error: true }. - Removes other content and previous hook mutations; no failed-record attachment uploads. Plain-object validation accepts null-prototype and cross-realm objects, rejects custom prototypes and promises, and consumes rejected promises. - Safe failure logging is once per JavaScript global environment, shared across SDK bundles; diagnostics contain no payload or exception details. - Memoized stripped records bypass downstream masking and form ordered, separate transport groups. This avoids mutation by in-place merging and preserves transaction ordering without duplicate same-span IDs in a request. Successful records keep normal merge/masking behavior within those boundaries. - Removes obsolete null/drop plumbing and updates regression tests, API docs, instrumentation docs, and changelog. ## Verification - pnpm --filter braintrust check:typings — passed (source and test TypeScript projects). - vitest run src/span-customizer.test.ts src/masking.test.ts src/logger.test.ts — 181 tests passed. - Runtime HTTP exporter regression covers failed/successful incremental records, ordered requests, HTTP 503 retry, unchanged original timestamps, masking immunity, attachment exclusion, unrelated attachment upload, and hook/log counts. - pnpm --filter braintrust build — all JS entrypoints and declarations built successfully. - Node smoke against built ESM and CJS SDK bundles with injected fetch transport — 6 stripped records exported, exactly 1 safe failure log across both bundles, no payload leak. - Scoped ESLint — 0 errors, 117 warnings (including tests ignored by repository ESLint configuration). - Scoped Prettier check — passed. ## Scope / limitations Native SDK spans only; existing OTel compatibility restrictions are unchanged. Incremental stripping cannot retract earlier successful exports. Runtime transport smoke used an injected local fetch implementation, not a live Braintrust service. Existing untracked js/tests/auto-instrumentations/fixtures/test-files/ was preserved and excluded from the commit.
bump deps to fix security alerts
This is a purely internal refactor that lets us clean up a bunch of
incredibly unnecessary complex code. It is only net -2k LOC, however the
spaghetti it removes is much more important.
- API wrapping previously carried tracing lifecycle machinery.
`intercept` and `invoke` now only wrap function calls, so they can be
reused without creating spans or initializing Braintrust.
- Rename the internal definition API to `defineInterceptor`.
Interceptors receive the original function, receiver, and arguments:
```ts
const hooks = defineInterceptor("client", {
request: channel<[string], Promise<string>>({ channelName: "request" }),
});
// Wrap calls without enabling tracing.
hooks.request.intercept((target, receiver, args) => {
console.log("Request:", args[0]);
return Reflect.apply(target, receiver, args);
});
```
- Move span creation, context propagation, and completion into separate
tracing functions. Provider integrations explicitly register these
through interceptors; manual and generated wrappers only invoke hooks.
- Remove the old tracing hook protocol. Applications with previously
transformed bundles need to rebuild after upgrading.
- Previously, `decisions.create()` calls weren’t traced. Capture them through `wrapOpenAI()` and automatic instrumentation, recording questions, answers, probabilities, model, and token usage. ```ts const client = wrapOpenAI(new OpenAI()); await client.decisions.create(params); // Now traced automatically ``` - Store inline image inputs as attachments instead of embedding base64 data in traces. If uploading fails after retries, preserve the original inline image while respecting redaction and disabled image capture. - Preserve OpenAI’s response helpers, including `.withResponse()` and `.asResponse()`, so tracing doesn’t consume the response body callers need.
Closes [#2619](#2619) - Trace AI SDK v7’s `experimental_decide` alongside `experimental_evaluate`, which was previously the only supported decision API. Both work with wrappers and native telemetry without duplicate spans. ```ts const tracedAI = wrapAISDK(ai); await tracedAI.experimental_decide({ model, state, questions }); // Now traced ``` - Upload embedded decision media as Braintrust attachments instead of leaving it inline in traces. Uploads happen after masking and customization; upload failures still allow traces to be exported. - Respect telemetry settings for disabling tracing or input/output capture, and omit embedded media when attachment capture is disabled.
This is tbh more a hack until we can do breaking changes to the export. In the new major I'd like to just do everything super properly. - Previously, the bundled default export kept unused SDK APIs in application bundles, even when importing only a single function. Preserve native ES module re-exports so bundlers can remove unused APIs. - Share logging and runtime state across the split modules so runtime configuration still applies to every SDK export.
…als (#2623) the bot found this while working on braintrustdata/braintrust-sdk-python#934, and I think it's legit. - Revalidate active login credentials after a queued non-forced login waits for its predecessor. - Reuse matching completed logins and reject conflicting credentials with `forceLogin: true` guidance.
Ref: braintrustdata/braintrust-spec#68 Adds image and box instrumentation for TypeSafe SDK 0.7 while retaining coverage for 0.6. Calls emit one `decision` span containing all questions and answers, including mixed box, Noul, Choice, and Score results. Images stay in their logged input locations, and box answers get an `image_path` when the image association is unambiguous. Attachment capture remains opt-in and uses the shared export pipeline for customization and upload fallback. Adds pinned/latest e2e coverage for both version lines, recorded provider cassettes, and paired JSON/text span-tree snapshots. The cassette harness can opt into inline request bodies so multipart image requests match during replay. Verified build/lint, 33 focused tests, and three e2e replay runs with 36 passing tests each. Verified all four summary links against 24 production traces.
## Summary Fixes agent instruction gaps found by reviewing recent coding agent sessions on this repo. - **AGENTS.md:** Replace `pnpm test -- -t "test name"`, which makes Vitest ignore the filter and run the full core suite, with `pnpm exec vitest run <file> [-t "name"]`. - **AGENTS.md:** Document `pnpm run check:typings` and require it before opening or updating a PR. Vitest does not typecheck, and a recent PR failed `check-typings` in CI because of type errors in a test file. - **Changesets:** Remove the root `pnpm changeset` script and every reference to it. Its interactive package picker forced agents to drive it with simulated keystrokes. The create-changeset skill now says to write `.changeset/*.md` files directly and shows the file format. `PUBLISHING.md` points to that skill, and the `enforce-changeset` error message points to `PUBLISHING.md`. Release workflows are unaffected because they call `pnpm exec changeset` directly. ## Verification - `pnpm exec vitest run src/logger-caching.test.ts -t "queued"` runs 12 tests and skips 44. `pnpm test -t "<no match>"` skips every file, while `pnpm test -- -t "<no match>"` runs the full suite. - `pnpm run check:typings` passes in `js/`. - Prettier passes, and `node scripts/release/validate-changesets.mjs` passes. No changeset is included because no published package changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Was looking at adding console.log instrumentation and figured we need to
do this first. But open to other ideas.
## AI Summary
Routes the SDK's remaining `console.*` diagnostics through
`debugLogger`. Adds `debugLogger.notice` for user-facing messages that
should stay visible by default.
```ts
debugLogger.warn("..."); // hidden unless a level is configured
debugLogger.notice.warn("..."); // shown by default
```
```
level configured?
├─ false → silent
├─ "<level>" → filter by level (applies to notice too)
└─ unset → notice.* prints as "warn", everything else silent
```
| Method | Used for |
| --- | --- |
| `notice.*` | wrapper "Not wrapping" warnings, late
`configureInstrumentation`, span customizer failure, dev server errors,
`publish()` warnings, scorer/vitest warnings, auto-instrumentation
transform failures |
| `warn` / `error` / `debug` | internal swallowed errors (stream
patcher, plugins, telemetry, disk cache), auto-instrumentation status |
## Behavior changes
| Before | After |
| --- | --- |
| Internal handler errors always printed | Opt-in via
`BRAINTRUST_DEBUG_LOG_LEVEL` / `debugLogLevel` |
| `DEBUG='@braintrust*'` showed auto-instrumentation status |
`BRAINTRUST_DEBUG_LOG_LEVEL=debug` |
| `[braintrust] [Braintrust] ...` | Duplicate prefixes removed |
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
- Add tracing for `@ai-sdk/code-mode`, capturing executed JavaScript and
nesting the functions it calls underneath the execution.
- Extend `wrapAISDK` to trace direct code-mode calls without a loader
hook:
```ts
import * as codeMode from "@ai-sdk/code-mode";
import { wrapAISDK } from "braintrust";
// Now supports code-mode execution.
const { experimental_runCodeMode } = wrapAISDK(codeMode);
```
- Start tool traces before running the tool so nested calls attach to
the correct parent and synchronous errors are captured.
- Avoid duplicate traces when automatic instrumentation and explicit
Braintrust telemetry are both enabled.
Example trace:
https://www.braintrust.dev/app/Braintrust%20SDKs/p/tmp-luca/logs?r=586cb08b3efce34c640c3f045f83b40a&sid=22ffb096f6fc1e7e&tvt=trace
<img width="801" height="950" alt="Screenshot 2026-10-19 at 12 46 37"
src="https://github.com/user-attachments/assets/0050d15f-7133-4251-94c5-7d9b0f33b450"
/>
…enAI models (#2632) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: lforst <8118419+lforst@users.noreply.github.com>
resolves https://linear.app/braintrustdata/issue/SDK-345/add-logging-api-to-javascript-sdk resolves https://linear.app/braintrustdata/issue/SDK-346/instrument-consolelog-to-emit-logs-in-js-sdk # AI Summary ## Public API ```js import { initLogger, wrapConsole } from "braintrust"; const logger = initLogger({ projectName: "application-logs" }); ``` | API | Behavior | | --- | --- | | logger.trace(body, options?) | Record a trace-level message; does not start a trace. | logger.debug(body, options?) | Record a debug-level message. | | logger.info(body, options?) | Record an info-level message. | | logger.warn(body, options?) | Record a warning. | | logger.error(body, options?) | Record an error-level message. | | logger.fatal(body, options?) | Record a fatal-level message. | | logger.fmt`text ${value}` | Prepare a message that preserves its template and values. | | wrapConsole(target = console, { logger } = {}) | Capture future console calls; return a cleanup function. | Bodies can be strings or JSON-serializable values. All severity methods accept { metadata: { ... } } as their second argument. By default they return a row ID immediately. With initLogger({ asyncFlush: false, ... }), they return a promise that resolves to the row ID after uploading. ## Track request activity with structured messages ```js await logger.traced(async () => { logger.info("Checkout started", { metadata: { request_id: "req_123", customer_id: "customer_456" }, }); logger.warn({ event: "payment_retry", attempt: 2 }); }, { name: "checkout" }); await logger.flush(); ``` Messages emitted inside an active Braintrust or compatible OpenTelemetry span share its span and trace IDs, while retaining their own log row IDs. They do not create child spans or change the active span. Outside an active span, messages from a logger share that logger's baseline trace ID. ## Keep templates for grouping related messages ```js const userId = "user_123"; const amount = 12.5; logger.info(logger.fmt`User ${userId} paid ${amount.toFixed(2)}`, { metadata: { source: "checkout" }, }); ``` The displayed message is "User user_123 paid 12.50". JavaScript evaluates each expression normally, so use expressions such as amount.toFixed(2) for formatting. The native template tag preserves these metadata fields: | Metadata key | Value | | --- | --- | | braintrust.template | User ${0} paid ${1} | | braintrust.template.parameter.0 | "user_123" | | braintrust.template.parameter.1 | "12.50" | | source | "checkout" | Parameters use positional keys because JavaScript template tags receive values, not variable names. JSON-compatible values retain their types; undefined and bigint become strings. Ordinary strings, including interpolated strings passed without logger.fmt, are recorded verbatim without template metadata. ## Capture an application's existing console calls ```js const stopCapturing = wrapConsole(console, { logger }); console.log("Worker started"); console.warn("Payment %s will retry", "pay_123"); stopCapturing(); await logger.flush(); ``` wrapConsole() without arguments instruments the global console and uses the current project logger at call time. Before a logger is initialized, console calls keep their normal output without uploading logs. Pass a console instance and { logger } to select a specific destination. Node uses its native console message formatter. Other runtimes preserve the arguments as text in Braintrust while the original console retains its native rendering. SDK diagnostics and internal HTTP transport logs are excluded from capture to prevent recursion. Repeated wrapping does not duplicate messages. The cleanup function is safe to call repeatedly and does not disable a newer wrap or reactivate an older wrap. It stops future forwarding; messages already queued still upload, and direct logger calls remain available.
closes [SDK-562](https://linear.app/braintrustdata/issue/SDK-562) - Extend automatic tracing for `@cloudflare/think` from 0.13.x to 0.13–0.20.x. - Keep Think turn traces open until AI SDK 7 finishes streaming, capturing outputs, token usage, and errors while keeping model and tool calls under the correct turn without duplicate traces. - Preserve all arguments passed through `wrapCloudflareThink`, including the conversation history branch identifier that was previously dropped.
- Add explicit tracing helpers for AI SDK text batches, capturing
prompts, outputs, token usage, and errors under one batch trace with a
child span per request.
- Allow tracing to start after submission and finish in another process
using saved context. Retrying completion updates the same spans.
- Keep submission, polling, and result collection in application code.
The new helpers only capture trace data:
```ts
import { startAISDKBatchTrace, completeAISDKBatchTrace } from
"braintrust";
// New: capture submitted requests and save context for later.
const context = await startAISDKBatchTrace({ batch, requests });
// New: finish tracing after collecting the batch results.
await completeAISDKBatchTrace({ batch, requests, context, status,
results });
```
…en the provider omits index (#2641) Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: lforst <8118419+lforst@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ens (#2646) Co-authored-by: Amp <amp@ampcode.com> Co-authored-by: lforst <8118419+lforst@users.noreply.github.com>
Co-authored-by: lforst <8118419+lforst@users.noreply.github.com>
- Export previously hidden LiveKit operations, such as session startup, shutdown, and end-of-turn detection, as `livekit.<name>` spans with their original nesting and metadata. - Keep redundant model and speech synthesis request spans folded into their parents to avoid duplicate tracing. - Allow session cleanup even when LiveKit leaves lifecycle spans open, while still recording them if they finish later.
Closes [#2645](#2645) - Use Node’s synchronous module hooks so automatic tracing works alongside TypeScript loaders such as `ts-exec`, preserving application exports and execution. - Handle ES modules and CommonJS through the same hook, avoiding duplicate instrumentation. Keep the existing loader approach on older Node versions where synchronous hooks aren’t safe. - Correctly decode module source returned as bytes by other loaders; previously, it could be interpreted as comma-separated numbers instead of JavaScript.
resolves #2651 resolves https://linear.app/braintrustdata/issue/SDK-604 > 🤖 **This text was AI-generated.** AI SDK tools can transform their execution result with `toModelOutput` before the model sees it. Tool spans previously kept only the raw execution result, so their recorded output could differ from the value returned in the model response. Capture the converted value on the tool span and include it in the parent call output: ```text tool result -> toModelOutput -> model response └──────────────> tool span output ``` This also updates AI SDK v5, v6, and v7 scenario coverage and recordings. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
> 🤖 **This text was AI-generated.** Pi Coding Agent instrumentation copied each child LLM's token usage onto the AgentSession.prompt task span. Trace rollups counted both, doubling token totals and estimated cost. Before, two calls using 29 prompt tokens each produced: ```text AgentSession.prompt (task): 58 prompt tokens |-- LLM call: 29 prompt tokens `-- LLM call: 29 prompt tokens Trace rollup: 116 prompt tokens ``` Keep usage on the child LLM spans and record only timing metrics on the prompt span. Remove both the stream-result aggregation and the turn_end fallback. ```text AgentSession.prompt (task): timing only |-- LLM call: 29 prompt tokens `-- LLM call: 29 prompt tokens Trace rollup: 58 prompt tokens ``` Regression coverage checks cached usage across multiple turns, deferred prompts, and exported traces for wrapped and automatic instrumentation. Fixes #2659 resolves https://linear.app/braintrustdata/issue/SDK-621
I really need to rewrite the transport, the whole retry logic is messed
but this is a good intermediate in the short term.
> 🤖 **This text was AI-generated.**
The HTTP logger sent rows as soon as they were queued, so nearby events
often used separate requests and span start and end rows could not be
merged. Add a short batching window for long-lived Node processes, while
keeping serverless and other runtimes immediate by default.
```text
span.log() / span.end() flush() / beforeExit
│ │
▼ │
┌─────────────────────────┐ │
│ queue [r1][r2] … [rN] │ │
└────────────┬────────────┘ │
▼ │
╔═════════════════════╗ yes │
║ N ≥ batch threshold ║ ───────────────┐ │
╚══════════╤══════════╝ │ │
│ no ▼ ▼
▼ ┌─────────────────────────────┐
◷ wait until oldest row ──────▶ │ drain ─▶ merge ─▶ POST │
+ BRAINTRUST_FLUSH_DELAY_MS │ same-id rows /logs3 │
└──────────────┬──────────────┘
1000ms long-lived Node │ done
0ms serverless, browser, edge ▼
rows queued meanwhile get
their own deadline ──▶ ◷
```
```text
before r1 r2 r3 r4 after r1 r2 r3 r4
│ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ └─────┴──┬──┴─────┘
[P] [P] [P] [P] ◷ ≤ 1s
▼
4 requests, start + end [P]
rows sent separately 1 request, start + end
rows merged into one
```
An explicit flush still sends immediately. BRAINTRUST_FLUSH_DELAY_MS and
BRAINTRUST_FLUSH_BATCH_THRESHOLD adjust the delay and batch threshold.
Updated request snapshots reflect the new batching behavior.
---------
Co-authored-by: lforst <8118419+lforst@users.noreply.github.com>
…2673) > 🤖 **This text was AI-generated.** Pi 1.4 carries system instructions and tool declarations in transcript system messages. The integration read only `context.systemPrompt` and `context.tools`, so LLM spans lost their system prompt and tool schemas. ```text Before: Pi system messages -> discarded -> prompt and tools missing After: Pi system messages -> replayed -> system input + metadata.tools ``` Keep the initial system prompt and later system updates at their original transcript positions, with explicit section replacements and removals. Resolve tool additions, replacements, and removals into `metadata.tools`. Keep legacy context support, wrap executors from both `agent.state.tools` and legacy/custom `context.tools`, and enable the auto hook for Pi 1.x. Pinned and latest Pi 1.x variants cover manual wrapping and the auto hook using fresh recordings made through each variant against OpenAI Responses (`gpt-5-nano`). Cassette matching retains request bodies, including the system prompt and tool schemas; only the working directory, date, and session cache key are normalized. Four regression cases failed before the initial fix. Two additional context-executor regressions failed before restoring wrapping of context-owned tools. The existing transcript regression also failed before correcting duplicated instructions in later system updates. After consolidating overlapping tests, all 35 focused tests pass and all 24 e2e checks pass across three replay runs. Build, type checks, formatting, and cassette auditing passed. Changing the recorded system prompt or tool schema was also verified to produce a cassette miss. Fixes #2662 resolves https://linear.app/braintrustdata/issue/SDK-622
Prepares a release by updating package versions, changelogs, and consumed changeset files. ## Packages - `braintrust`: `3.40.0` Source-main-sha: abefccb Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Abhijeet Prasad (AbhiPrasad)
approved these changes
Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.