Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This repository contains Braintrust's Python SDKs and integrations, including:

- The main `braintrust` SDK package in [`./py`](./py)
- Built-in integrations under [`py/src/braintrust/integrations`](py/src/braintrust/integrations) and related compatibility packages under [`./integrations`](./integrations)
- Self-contained `uv`-project examples for every integration in [`./examples`](./examples)
- Self-contained `uv`-project examples for the integrations that have a directory under [`./examples`](./examples)
- Tests and local development tooling for Python SDK development

## Quickstart
Expand Down Expand Up @@ -63,6 +63,7 @@ BRAINTRUST_API_KEY=<YOUR_API_KEY> braintrust eval tutorial_eval.py
| [OpenRouter](py/src/braintrust/integrations/openrouter/) | Yes | `openrouter>=0.6.0` |
| [Vercel AI SDK for Python](py/src/braintrust/integrations/ai_sdk/) | Yes | `ai>=0.4.0` |
| [Google GenAI](py/src/braintrust/integrations/google_genai/) | Yes | `google-genai>=1.30.0` |
| [Google Discovery Engine](py/src/braintrust/integrations/google_discoveryengine/) | Yes | `google-cloud-discoveryengine>=0.20.3` |
| [Bedrock Runtime](py/src/braintrust/integrations/bedrock_runtime/) | Yes | `boto3>=1.34.116` |
| [HuggingFace Hub](py/src/braintrust/integrations/huggingface_hub/) | Yes | `huggingface-hub>=0.32.0` |
| [Hugging Face Transformers](py/src/braintrust/integrations/transformers/) | Yes | `transformers>=4.42.0` |
Expand All @@ -71,6 +72,7 @@ BRAINTRUST_API_KEY=<YOUR_API_KEY> braintrust eval tutorial_eval.py
| [LangChain](py/src/braintrust/integrations/langchain/) | Yes | `langchain-core>=0.3.28` |
| [LlamaIndex](py/src/braintrust/integrations/llamaindex/) | Yes | `llama-index-core>=0.13.0` |
| [Instructor](py/src/braintrust/integrations/instructor/) | Yes | `instructor>=1.11.0` |
| [TypeSafe](py/src/braintrust/integrations/typesafe/) | Yes | `typesafe-sdk>=0.6.0` |
| [DSPy](py/src/braintrust/integrations/dspy/) | Yes | `dspy>=2.6.0` |
| [OpenAI Agents](py/src/braintrust/integrations/openai_agents/) | Yes | `openai-agents>=0.0.19` |
| [Claude Agent SDK](py/src/braintrust/integrations/claude_agent_sdk/) | Yes | `claude_agent_sdk>=0.1.10` |
Expand Down
16 changes: 15 additions & 1 deletion examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ braintrust.init_logger(project="example-<name>")
# now import and use the library normally
```

`braintrust.auto_instrument()` patches every supported integration that's installed in the current environment — there's no need to call individual `setup_*()` or `wrap_*()` helpers. The exceptions are examples that demonstrate non-tracing concerns (`evals/`, `langsmith/`, `otel/`, `temporal/`).
`braintrust.auto_instrument()` patches every supported integration that's installed in the current environment — there's no need to call individual `setup_*()` or `wrap_*()` helpers. `setup_*()`, `wrap_*()`, and `patch_*()` in `braintrust.integrations.<name>` are the manual alternatives. The exceptions are examples that demonstrate non-tracing concerns (`evals/`, `langsmith/`, `otel/`, `temporal/`).

If you want to see `auto_instrument()` light up multiple providers in one trace, the `openai/`, `anthropic/`, and any other provider example can be combined trivially — just install both packages in the same environment, call `auto_instrument()` once, and use both clients.

Expand All @@ -66,6 +66,7 @@ Unless noted otherwise, every example below uses `braintrust.auto_instrument()`.
| `claude_agent_sdk/` | Claude Agent SDK subprocess query |
| `cohere/` | Cohere `ClientV2` chat call |
| `crewai/` | CrewAI `Agent` + `Task` + `Crew` end-to-end |
| `deepagents/` | Deep Agents traced through LangChain; there is no `setup_deepagents()` |
| `dspy/` | DSPy `ReAct` agent with two tools (LiteLLM token metrics propagate) |
| `evals/` | The `Eval` framework — does **not** use `auto_instrument()` |
| `google_genai/` | Google GenAI `generate_content` against Gemini |
Expand All @@ -83,6 +84,19 @@ Unless noted otherwise, every example below uses `braintrust.auto_instrument()`.
| `strands/` | Strands `Agent` against `gpt-4o-mini` |
| `temporal/` | Distributed Temporal workflow tracing via `BraintrustPlugin` |

## Integrations without a sample yet

These integrations ship in the SDK and do not have a directory under `examples/` yet:

- `ai_sdk`
- `google_discoveryengine`
- `huggingface_hub`
- `instructor`
- `livekit_agents`
- `pipecat`
- `transformers`
- `typesafe`

## Why no committed `uv.lock`?

Examples are meant to demonstrate the **current** SDK against **current** provider releases, not snapshot a specific point in time. Committing lockfiles would create constant churn (a new lockfile for every transitive bump on PyPI) and would actively work against the demo goal — someone running an example next quarter wants it to work with whatever's on PyPI today, not a 3-month-old pin. Lockfiles are gitignored.
Expand Down
2 changes: 1 addition & 1 deletion examples/adk/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Google ADK + Braintrust

Calls `braintrust.auto_instrument()` so Google ADK is patched automatically, then runs a weather agent against `gemini-3.8-flash` with one tool. The trace shows the runner span, the agent span, the LLM span, and the tool call.
Calls `braintrust.auto_instrument()` so Google ADK is patched automatically, then runs a weather agent against `gemini-3.8-flash` with one tool.

## Run

Expand Down
4 changes: 2 additions & 2 deletions examples/langsmith/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The wrapper supports two modes:
Both LangSmith and Braintrust tracing are active. Use this during migration to verify everything works before fully switching.

```python
from braintrust.wrappers.langsmith import setup_langsmith
from braintrust.wrappers.langsmith_wrapper import setup_langsmith

setup_langsmith(project_name="my-project")
```
Expand All @@ -33,7 +33,7 @@ setup_langsmith(project_name="my-project")
Only Braintrust runs - LangSmith code is completely replaced. Use this when you're ready to fully migrate.

```python
from braintrust.wrappers.langsmith import setup_langsmith
from braintrust.wrappers.langsmith_wrapper import setup_langsmith

setup_langsmith(project_name="my-project", standalone=True)
```
Expand Down
2 changes: 1 addition & 1 deletion examples/openrouter/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# OpenRouter + Braintrust

Calls `braintrust.auto_instrument()` to patch the OpenRouter SDK so chat, embeddings, and responses calls are traced. The resulting LLM spans include the provider that OpenRouter dispatched to (e.g. `metadata.provider == "openai"`).
Calls `braintrust.auto_instrument()` so OpenRouter chat completions are traced.

## Run

Expand Down
2 changes: 1 addition & 1 deletion examples/pydantic_ai/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Pydantic AI + Braintrust

Calls `braintrust.auto_instrument()` so the Pydantic AI agent's underlying provider calls are captured as Braintrust spans automatically. The agent itself is wrapped in `start_span` so the run gets a permalink.
Calls `braintrust.auto_instrument()`, which traces the Pydantic AI agent. The script also opens `start_span` so it can print a permalink.

## Run

Expand Down
2 changes: 2 additions & 0 deletions examples/temporal/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

This example demonstrates distributed tracing for Temporal workflows using Braintrust.

This example passes `BraintrustPlugin` explicitly. Import it from `braintrust.integrations.temporal`.

## Setup

1. Install dependencies. This is a uv project that installs `braintrust[temporal]` from the local checkout:
Expand Down
2 changes: 1 addition & 1 deletion examples/temporal/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import uuid

import braintrust
from braintrust.contrib.temporal import BraintrustPlugin
from braintrust.integrations.temporal import BraintrustPlugin
from temporalio.client import Client
from workflow import TASK_QUEUE_NAME, SimpleWorkflow, TaskInput

Expand Down
2 changes: 1 addition & 1 deletion examples/temporal/worker.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import asyncio
import os

from braintrust.contrib.temporal import BraintrustPlugin
from braintrust.integrations.temporal import BraintrustPlugin

# Import only what we need to avoid loading optional dependencies
from braintrust.logger import init_logger
Expand Down
17 changes: 6 additions & 11 deletions py/src/braintrust/integrations/adk/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@


def setup_braintrust(*args, **kwargs):
"""Deprecated. Call :func:`setup_adk` instead.

Forwards ``*args`` and ``**kwargs`` to :func:`setup_adk`.
"""
warnings.warn("setup_braintrust is deprecated, use setup_adk instead", DeprecationWarning, stacklevel=2)
return setup_adk(*args, **kwargs)

Expand All @@ -40,16 +44,7 @@ def setup_adk(
project_name: str | None = None,
SpanProcessor: type | None = None,
) -> bool:
"""
Setup Braintrust integration with Google ADK. Will automatically patch Google ADK agents, runners, flows, and MCP tools for automatic tracing.

Also enables the Google GenAI integration, which ADK's Gemini models call
through. It owns the ``llm`` span and token usage for each Gemini request;
ADK's ``llm_call`` spans are ``task`` spans so usage is counted once. For
other ADK models (e.g. LiteLLM or Anthropic), enable that provider's
integration, or use ``braintrust.auto_instrument()``, to record usage.

If you prefer manual patching take a look at `wrap_agent`, `wrap_runner`, `wrap_workflow`, `wrap_flow`, and `wrap_mcp_tool`.
"""Set up Braintrust tracing for Google ADK.

Args:
api_key (Optional[str]): Braintrust API key.
Expand All @@ -58,7 +53,7 @@ def setup_adk(
SpanProcessor (Optional[type]): Deprecated parameter.

Returns:
bool: True if setup was successful, False otherwise.
True if patched or already patched, False if ADK is missing or too old.
"""
if SpanProcessor is not None:
warnings.warn("SpanProcessor parameter is deprecated and will be ignored", DeprecationWarning, stacklevel=2)
Expand Down
1 change: 0 additions & 1 deletion py/src/braintrust/integrations/adk/tracing.py
Original file line number Diff line number Diff line change
Expand Up @@ -505,7 +505,6 @@ async def _trace():

async with aclosing(wrapped(*args, **kwargs)) as agen:
async for event in agen:
# Record time to first token
if first_token_time is None:
first_token_time = time.time()

Expand Down
6 changes: 5 additions & 1 deletion py/src/braintrust/integrations/agentscope/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,11 @@ def setup_agentscope(
project_id: str | None = None,
project_name: str | None = None,
) -> bool:
"""Setup Braintrust integration with AgentScope."""
"""Set up Braintrust tracing for AgentScope.

Returns:
True if patched or already patched, False if AgentScope is missing or too old.
"""
if current_span() == NOOP_SPAN:
init_logger(project=project_name, api_key=api_key, project_id=project_id)

Expand Down
7 changes: 3 additions & 4 deletions py/src/braintrust/integrations/agno/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,7 @@ def setup_agno(
project_name: str | None = None,
eval_experiments: bool | None = None,
) -> bool:
"""
Setup Braintrust integration with Agno. Will automatically patch Agno agents, models,
function calls, and evals (``agno.eval``) for tracing.
"""Set up Braintrust tracing for Agno.

Args:
api_key: Braintrust API key (optional, can use env var BRAINTRUST_API_KEY)
Expand All @@ -59,7 +57,8 @@ def setup_agno(
current, so pass eval_experiments=False to keep suite runs in logs too.

Returns:
True if setup was successful, False otherwise
``True`` when patching ran or was already applied. ``False`` when Agno is missing or outside the version
range.
"""
_configure_eval_experiments(eval_experiments)

Expand Down
6 changes: 5 additions & 1 deletion py/src/braintrust/integrations/ai_sdk/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,11 @@ def setup_ai_sdk(
project_id: str | None = None,
project_name: str | None = None,
) -> bool:
"""Initialize Braintrust and register the Vercel AI SDK telemetry adapter."""
"""Set up Braintrust tracing for the Vercel AI SDK.

Returns:
True if patched or already patched, False if the SDK is missing or too old.
"""
if current_span() == NOOP_SPAN:
init_logger(project=project_name, api_key=api_key, project_id=project_id)
return AISDKIntegration.setup()
Expand Down
3 changes: 0 additions & 3 deletions py/src/braintrust/integrations/ai_sdk/tracing.py
Original file line number Diff line number Diff line change
Expand Up @@ -323,9 +323,6 @@ def _loop_turn_start(_upstream_span: Any, _data: Any) -> dict[str, Any]:


def _stream_start(_upstream_span: Any, data: Any) -> dict[str, Any]:
# The AI SDK delegates transport to an instrumentable provider client.
# Keep this framework-level span a task and let that provider own the
# single LLM span for the request.
return {
"name": "ai.stream",
"type": SpanTypeAttribute.TASK,
Expand Down
3 changes: 3 additions & 0 deletions py/src/braintrust/integrations/anthropic/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
"""Braintrust integration for the Anthropic Python SDK."""

import warnings

from .integration import AnthropicIntegration
Expand All @@ -8,6 +10,7 @@


def wrap_anthropic_client(client):
"""Deprecated. Use ``wrap_anthropic``."""
warnings.warn(
"wrap_anthropic_client() is deprecated. Use wrap_anthropic() instead.",
DeprecationWarning,
Expand Down
8 changes: 1 addition & 7 deletions py/src/braintrust/integrations/anthropic/tracing.py
Original file line number Diff line number Diff line change
Expand Up @@ -1016,8 +1016,6 @@ def __close(self, exc_type, exc_value, traceback):


class TracedMessageStream(Wrapper):
"""TracedMessageStream wraps both sync and async message streams."""

def __init__(self, msg_stream, span, request_start_time: float):
super().__init__(msg_stream)
self.__msg_stream = msg_stream
Expand Down Expand Up @@ -2047,11 +2045,7 @@ async def _trace_async_parsed_message_call(method, args, kwargs):


def _wrap_anthropic(client):
"""Wrap an Anthropic object (or AsyncAnthropic) to add tracing.

If the client is already traced (e.g. via ``AnthropicIntegration.setup()``),
it is returned unchanged to avoid double-wrapping.
"""
"""Wrap an Anthropic or async Anthropic client. Already-traced clients are returned unchanged."""
if getattr(client, _BRAINTRUST_TRACED, False):
return client

Expand Down
6 changes: 5 additions & 1 deletion py/src/braintrust/integrations/autogen/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,11 @@ def setup_autogen(
project_id: str | None = None,
project_name: str | None = None,
) -> bool:
"""Setup Braintrust integration with AutoGen."""
"""Set up Braintrust tracing for AutoGen.

Returns:
True if patched or already patched, False if AutoGen is missing or too old.
"""
if current_span() == NOOP_SPAN:
init_logger(project=project_name, api_key=api_key, project_id=project_id)

Expand Down
45 changes: 39 additions & 6 deletions py/src/braintrust/integrations/base.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
"""Shared integration orchestration primitives."""
"""Orchestration for third-party integrations.

Subclass :class:`BaseIntegration` and keep the provider import inside the patcher.
:meth:`BaseIntegration.setup` returns ``False`` when the package is missing or
outside ``min_version`` / ``max_version``. An undetectable version still patches.
Patchers run in ascending ``priority`` and must be idempotent.
"""

import importlib
import importlib.metadata
Expand All @@ -16,7 +22,13 @@


class BasePatcher(ABC):
"""Base class for one concrete integration patch strategy."""
"""One patch strategy.

``priority`` runs ascending from a default of 100. ``rescan_on_setup``
re-runs ``patch`` on later ``setup()`` calls instead of treating an existing
marker as success. A callback or ``patch_class`` that returns ``False``
leaves the target unmarked so the next ``setup()`` retries.
"""

name: ClassVar[str]
patch_id: ClassVar[str | None] = None
Expand Down Expand Up @@ -132,7 +144,12 @@ def patch(cls, module: Any | None, version: str | None, *, target: Any | None =


class ClassScanPatcher(BasePatcher):
"""Base patcher for rescanning and patching discovered class hierarchies."""
"""Rescan a class hierarchy on every ``setup()``.

Subclasses imported later are still found. A scan that finds nothing new
returns ``False``. Abstract classes are skipped unless
``include_abstract_classes`` is set.
"""

rescan_on_setup: ClassVar[bool] = True
include_abstract_classes: ClassVar[bool] = False
Expand Down Expand Up @@ -564,7 +581,14 @@ def wrap_target(cls, target: Any) -> Any:


class BaseIntegration(ABC):
"""Base class for an instrumentable third-party integration."""
"""Base class for an instrumentable third-party integration.

``import_names`` are tried in order; the first importable module wins.
``distribution_names`` are the PyPI names tried before
:func:`detect_module_version`, for packages whose distribution name differs
from the import root. ``min_version`` and ``max_version`` are inclusive.
``patchers`` must not repeat an identifier.
"""

name: ClassVar[str]
import_names: ClassVar[tuple[str, ...]]
Expand Down Expand Up @@ -597,7 +621,12 @@ def setup(
*,
target: Any | None = None,
) -> bool:
"""Apply all applicable patchers for this integration."""
"""Apply applicable patchers.

Returns ``False`` when the package is missing or outside the version
range. An undetectable version still patches. An already-applied patcher
counts as success unless ``rescan_on_setup`` is set.
"""
module = _import_first_available(cls.import_names)
if module is None:
return False
Expand All @@ -619,7 +648,11 @@ def setup(

@classmethod
def detect_version(cls, module: Any) -> str | None:
"""Return the installed provider version for this integration."""
"""Return the installed provider version for this integration.

``distribution_names`` are tried first. A distribution whose metadata
file is missing falls through to :func:`detect_module_version`.
"""
for distribution_name in cls.distribution_names:
try:
return importlib.metadata.version(distribution_name)
Expand Down
9 changes: 6 additions & 3 deletions py/src/braintrust/integrations/bedrock_runtime/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Public entry points for the boto3 Bedrock Runtime integration."""
"""Braintrust integration for boto3 Bedrock Runtime."""

from .integration import BedrockRuntimeIntegration
from .patchers import wrap_bedrock_client
Expand All @@ -8,10 +8,13 @@


def setup_bedrock() -> bool:
"""Patch botocore client creation to auto-wrap Bedrock Runtime clients."""
"""Set up Braintrust tracing for Bedrock Runtime.

Returns True if patched or already patched, False if botocore is missing or too old.
"""
return BedrockRuntimeIntegration.setup()


def wrap_bedrock(client):
"""Instrument a boto3 Bedrock Runtime client instance in place."""
"""Instrument a Bedrock Runtime client in place."""
return wrap_bedrock_client(client)
Loading
Loading