diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 7fcd14a..78281cd 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "judgment", "displayName": "Judgment", - "description": "Access Judgment traces, sessions, behaviors, judges, prompts, datasets, and tests directly from your AI coding tool, with official skills for tracing, evaluation, and MCP usage best practices.", + "description": "Access Judgment traces, sessions, labels, judges, prompts, datasets, and tests directly from your AI coding tool, with official skills for tracing, evaluation, and MCP usage best practices.", "version": "1.0.1", "author": { "name": "Judgment Labs", diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index 5ccd054..ba8a759 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "judgment", "version": "1.0.1", - "description": "Access Judgment traces, sessions, behaviors, judges, prompts, datasets, and tests directly from Codex, with official skills for tracing, evaluation, and MCP usage best practices.", + "description": "Access Judgment traces, sessions, labels, judges, prompts, datasets, and tests directly from Codex, with official skills for tracing, evaluation, and MCP usage best practices.", "author": { "name": "Judgment Labs", "email": "support@judgmentlabs.ai", @@ -24,7 +24,7 @@ "mcpServers": "./.mcp.json", "interface": { "displayName": "Judgment", - "shortDescription": "Query Judgment traces, sessions, behaviors, judges, and docs.", + "shortDescription": "Query Judgment traces, sessions, labels, judges, and docs.", "longDescription": "Adds the Judgment MCP server and the official Judgment skills so Codex can inspect live agent data and follow Judgment best practices for tracing, evaluations, code judges, datasets, and monitoring.", "developerName": "Judgment Labs", "category": "Productivity", diff --git a/README.md b/README.md index 8876b9b..072677b 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ The official [Judgment](https://judgmentlabs.ai) plugin for AI coding agents. One repo, installable in both Claude Code and OpenAI Codex, bundling: 1. **The Judgment MCP server** (`https://mcp.judgmentlabs.ai`): query traces, - sessions, behaviors, judges, prompts, datasets, tests, automations, and + sessions, labels, judges, prompts, datasets, tests, automations, and agent memory directly from your editor. 2. **The `judgment-tracing` skill**: best-practice guidance for instrumenting apps with tracing, writing evaluations, building code judges, and running diff --git a/skills/judgment-tracing/SKILL.md b/skills/judgment-tracing/SKILL.md index f1e60bf..457d1ec 100644 --- a/skills/judgment-tracing/SKILL.md +++ b/skills/judgment-tracing/SKILL.md @@ -19,7 +19,7 @@ Follow these principles for all Judgment work: 3. **Tracing first for integrations**: When the user asks to add Judgment to an app, start with [references/tracing.md](references/tracing.md) unless they explicitly ask for evaluations or code judges. 4. **Stay on the tracer surface**: In most cases, use Judgment's tracer, wrappers, and documented integrations directly. Do not reach into underlying provider objects or create additional wrapper layers unless the current docs require it or a real instrumentation gap remains after using the supported integration. 5. **Start small**: For evaluations, begin with a focused example set and one scorer before expanding. -6. **Use the right scorer**: Use prompt/hosted scorers for rubric-based judgment and Python code judges for deterministic logic, custom dependencies, or trace inspection. +6. **Use the right scorer**: Use Agent judges for rubric-based judgment and Python code judges for deterministic logic, custom dependencies, or trace inspection. 7. **Keep credentials out of chat**: Ask the user to set `JUDGMENT_API_KEY` and `JUDGMENT_ORG_ID` locally rather than pasting secrets. 8. **Use offline agent tests before production changes**: For model, prompt, tool, or agent config changes, pull a stable dataset, collect fresh `OfflineTracer` traces for each input, then evaluate the generated offline examples in one batch. If the production agent is already traced with Judgment, leave that tracing intact and swap only the test harness initialization to `client.offline_tracer(...)`. diff --git a/skills/judgment-tracing/references/agent-testing.md b/skills/judgment-tracing/references/agent-testing.md index 9398223..03d6d10 100644 --- a/skills/judgment-tracing/references/agent-testing.md +++ b/skills/judgment-tracing/references/agent-testing.md @@ -153,7 +153,7 @@ results = client.evaluation.create().run( The agent entrypoint should be traced as the root span, usually with `@Tracer.observe(span_type="agent")` or the framework/provider integration the application already uses. Hosted scorers can use `offline_trace_id` to inspect -the stored trace server-side. Custom code judges can also use the fields copied +the stored trace server-side. Code judges can also use the fields copied into each generated example. ## Implementation Notes diff --git a/skills/judgment-tracing/references/docs.md b/skills/judgment-tracing/references/docs.md index 8cfdf62..a67251c 100644 --- a/skills/judgment-tracing/references/docs.md +++ b/skills/judgment-tracing/references/docs.md @@ -32,7 +32,7 @@ curl -s https://docs.judgmentlabs.ai/sdk-reference/typescript.md ## Useful Topics - Tracing: `https://docs.judgmentlabs.ai/documentation/performance/tracing` -- Prompt scorers: `https://docs.judgmentlabs.ai/documentation/evaluation/prompt-scorers` +- Agent judges: `https://docs.judgmentlabs.ai/documentation/evaluation/prompt-scorers` - Code judges: `https://docs.judgmentlabs.ai/documentation/evaluation/custom-scorers` - Datasets: `https://docs.judgmentlabs.ai/documentation/evaluation/datasets` - Python SDK reference: `https://docs.judgmentlabs.ai/sdk-reference/python` diff --git a/skills/judgment-tracing/references/evaluations.md b/skills/judgment-tracing/references/evaluations.md index 936e3b2..811f581 100644 --- a/skills/judgment-tracing/references/evaluations.md +++ b/skills/judgment-tracing/references/evaluations.md @@ -17,7 +17,7 @@ Use this reference when creating or debugging evaluations. 4. Pick one scorer first: - - prompt or hosted scorer for rubric-based natural-language judgment + - Agent judge or hosted scorer for rubric-based natural-language judgment - Python code judge for deterministic logic, custom dependencies, or trace inspection 5. Run locally while iterating, then expand the dataset and hosted workflows when the signal is useful. @@ -39,7 +39,7 @@ examples = [ ## Scorer Selection -- Use prompt scorers for qualitative rubrics such as helpfulness, policy adherence, or answer quality. +- Use Agent judges for qualitative rubrics such as helpfulness, policy adherence, or answer quality. - Use code judges for exact checks, business logic, custom library calls, or direct trace inspection. - Use datasets when the user needs repeatable regression testing across many examples. diff --git a/skills/judgment-tracing/references/tracing.md b/skills/judgment-tracing/references/tracing.md index 431f55a..9fcf09d 100644 --- a/skills/judgment-tracing/references/tracing.md +++ b/skills/judgment-tracing/references/tracing.md @@ -150,7 +150,7 @@ when possible, only ask when unclear. | Customer, org, workspace, or tenant identifiers | Multi-tenant app | `customer_id`, `tenant_id`, or plan tier | | Multiple routes, tools, agents, or product features | Multi-feature app | `feature`, `route`, or agent name attribute | | A/B tests, model routing, prompt variants | Experimented app | experiment, prompt, or model-family tag | -| Feedback collection, ratings, thumbs up/down | Has user feedback | capture as scores or behavior signals | +| Feedback collection, ratings, thumbs up/down | Has user feedback | capture as scores | | Stateless HTTP handlers, workers, queues, serverless functions, or RPC calls | Distributed app | distributed tracing with `service.name` | | Environment-specific projects | Needs project routing | active tracers with project names | | Traced functions spread across modules | Cross-file instrumentation | one tracer init, observed functions | @@ -166,8 +166,8 @@ when possible, only ask when unclear. Determines whether project routing is needed - "Does one request cross a stateless HTTP, worker, queue, serverless, or RPC boundary?" -> Determines whether distributed tracing is needed -- "How do you know when a response is good vs bad?" -> Determines scoring or - behavior-monitoring follow-up +- "How do you know when a response is good vs bad?" -> Determines whether to + follow up with a score or a label **Additions and their value:** @@ -181,7 +181,7 @@ when possible, only ask when unclear. | `service.name` | Separates services in OpenTelemetry and distributed flows | https://docs.judgmentlabs.ai/documentation/performance/tracing#opentelemetry-integration | | Distributed trace propagation | Connects stateless downstream service spans to the original request | https://docs.judgmentlabs.ai/documentation/performance/tracing#distributed-tracing | | Agent linked traces | Links delegated agent/subsystem work back to the parent trace | https://docs.judgmentlabs.ai/documentation/performance/tracing#subagent-tracing | -| Behavior or score signals | Enables quality filtering and production monitoring | https://docs.judgmentlabs.ai/documentation/performance/agent-behavior-monitoring | +| Labels or scores | Filter by label or score and track how often each label is applied | https://docs.judgmentlabs.ai/documentation/judges/monitor-judge-results | These are NOT baseline best practices. Only add what's relevant based on inference or user input. @@ -211,9 +211,9 @@ After adding context, point users to relevant UI features: - Traces view: See individual requests, span hierarchy, latency, cost, inputs, outputs, and errors - Sessions view: See grouped conversations if `session_id` is added -- Behaviors: Filter traces or sessions by monitored behavior -- Automations and alerts: Route failures, behaviors, or thresholds into - follow-up workflows +- Labels: Filter traces or sessions by the labels judges applied +- Automations and alerts: Route failures, a "Judge applied label" condition, or + thresholds into follow-up workflows - Judgment Agent search: Ask the in-product agent to search traces, cite spans, and investigate failures diff --git a/skills/mcp-server-best-practices/SKILL.md b/skills/mcp-server-best-practices/SKILL.md index b110d83..669ebba 100644 --- a/skills/mcp-server-best-practices/SKILL.md +++ b/skills/mcp-server-best-practices/SKILL.md @@ -11,7 +11,7 @@ metadata: ## Default to MCP -**Always use MCP tools first** when the user asks about anything related to Judgment data — traces, behaviors, sessions, projects, automations, judges, prompts, datasets, tests, documentation, agent memory, agent threads, or organizations. Do not fall back to reading code or asking for IDs if an MCP tool can fetch the data directly. +**Always use MCP tools first** when the user asks about anything related to Judgment data — traces, labels, sessions, projects, automations, judges, prompts, datasets, tests, documentation, agent memory, agent threads, or organizations. Do not fall back to reading code or asking for IDs if an MCP tool can fetch the data directly. ## Using search_traces Effectively @@ -49,7 +49,7 @@ Each filter is an object with a `field` discriminator: // LLM cost (USD) { "field": "llm_cost", "op": ">", "value": 0.10 } -// Behaviors (any of the listed judge/value pairs) +// Labels (any of the listed judge/value pairs) { "field": "behaviors", "op": "any", "value": [{ "judge_name": "toxicity", "value": "toxic" }] } // Numeric score by name