feat: add voice latency and interruptions scorers

AGENTS.md

@@ -58,6 +58,7 @@ The library maintains parallel implementations in TypeScript (`js/`) and Python
 - `ragas.ts` / `ragas.py` - RAG evaluation metrics (ContextRelevancy, Faithfulness, AnswerRelevancy, etc.)
 - `string.ts` / `string.py` - Text similarity (Levenshtein, EmbeddingSimilarity)
 - `json.ts` / `json.py` - JSON validation and diff
+- `voice.ts` / `voice.py` - Voice call timing (VoiceLatency, VoiceInterruptions)
 - `oai.ts` / `oai.py` - OpenAI client wrapper with caching
 - `score.ts` / `score.py` - Core Score type and Scorer base class
 

README.md

@@ -362,6 +362,11 @@ Eval(
 - Numeric difference
 - JSON diff
 
+### Voice evaluations
+
+- Voice latency
+- Voice interruptions
+
 For detailed documentation on all scorers, including parameters, score ranges, and usage examples, see the [**Scorer Reference**](SCORERS.md).
 
 ## Custom evaluation prompts

SCORERS.md

@@ -9,6 +9,7 @@ Complete reference for all scorers available in Autoevals, including parameters,
 - [Heuristic scorers](#heuristic-scorers)
 - [JSON scorers](#json-scorers)
 - [List scorers](#list-scorers)
+- [Voice scorers](#voice-scorers)
 
 ---
 
@@ -590,6 +591,91 @@ result = scorer.eval(
 
 ---
 
+## Voice scorers
+
+Deterministic scorers for the timing of a voice call. They take `utterances`, a list of `{ speaker, start_unix_ms, end_unix_ms, interrupted }` objects, where `speaker` is `"user"` or `"agent"`, times are Unix epoch milliseconds, and `interrupted` (optional) means the user cut the agent off. Build the list from your own traces.
+
+Both scorers group the user's speech into turns the same way:
+
+- Consecutive user utterances merge into one turn until the agent starts speaking.
+- User speech wholly inside the agent's, like "mm-hmm", isn't a turn.
+- The agent talks over a turn when it starts speaking during the turn and overlaps it by more than `minOverlapMs` (default 300).
+- The agent's reply to a turn is the first agent utterance that starts after the turn starts and before the next turn, and runs past the turn's end.
+
+Utterances with a missing or invalid time, or another speaker, are skipped and counted in `skipped`.
+
+Timestamps should mark audible speech, on one clock: the user's end is when they stopped speaking (not when the transcript was finalized), and the agent's start is when its first audio played (not when text-to-speech was requested). Times taken on the server leave out network delay, so callers hear slightly longer gaps.
+
+Limits:
+
+- If the agent cuts in while the user pauses mid-sentence and the user carries on, it counts as a fast reply plus a barge-in, not as talking over the user.
+- If the agent says nothing and the user speaks again ("hello?"), both utterances merge into one turn.
+- Drop intentional agent backchannels when you build the list, or they count as talking over the user.
+- The defaults (1500 ms, 300 ms) follow [Hamming's published bands](https://hamming.ai/resources/conversational-flow-measurement-voice-agents), not a standard.
+
+### VoiceLatency
+
+Measures how quickly the agent replies after the user finishes a turn.
+
+**Parameters:**
+
+- `utterances` (Utterance[], required): The call's utterances
+- `maxGapMs` / `max_gap_ms` (number, optional): Longest acceptable gap before a reply (default: 1500)
+- `minOverlapMs` / `min_overlap_ms` (number, optional): Overlap needed to count as talking over the user (default: 300)
+
+**Score Range:** 0-1, the share of replies that start within `maxGapMs`. `null` when the call has no replies.
+
+Turns the agent talked over aren't counted. A reply that starts slightly before the user stops counts as a gap of 0. Metadata has `replies`, `unanswered_turns`, `max_gap_ms`, `p50_ms`, `p95_ms`, `gaps_ms` and `skipped`; pool `gaps_ms` to get percentiles across a dataset.
+
+**Example:**
+
+```typescript
+import { VoiceLatency } from "autoevals";
+
+const result = await VoiceLatency({
+  output: null,
+  utterances: [
+    { speaker: "user", start_unix_ms: 0, end_unix_ms: 2000 },
+    { speaker: "agent", start_unix_ms: 2800, end_unix_ms: 5000 },
+    { speaker: "user", start_unix_ms: 6000, end_unix_ms: 7000 },
+    { speaker: "agent", start_unix_ms: 10000, end_unix_ms: 11000 },
+  ],
+});
+// Score: 0.5 (replies after 0.8 s and 3 s)
+```
+
+### VoiceInterruptions
+
+Measures how often the agent talks over the user.
+
+**Parameters:**
+
+- `utterances` (Utterance[], required): The call's utterances
+- `minOverlapMs` / `min_overlap_ms` (number, optional): Overlap needed to count as talking over the user (default: 300)
+
+**Score Range:** 0-1, the share of user turns the agent didn't talk over. `null` unless the call has timed user and agent utterances.
+
+Metadata has `user_turns`, `talk_overs`, `min_overlap_ms`, `skipped`, and `barge_ins`: agent utterances marked `interrupted`, where the user cut the agent off. Barge-ins don't affect the score.
+
+**Example:**
+
+```python
+from autoevals import VoiceInterruptions
+
+result = VoiceInterruptions().eval(
+    output=None,
+    utterances=[
+        {"speaker": "user", "start_unix_ms": 0, "end_unix_ms": 3000},
+        {"speaker": "agent", "start_unix_ms": 2000, "end_unix_ms": 4000},
+        {"speaker": "user", "start_unix_ms": 5000, "end_unix_ms": 6000},
+        {"speaker": "agent", "start_unix_ms": 6500, "end_unix_ms": 8000, "interrupted": False},
+    ],
+)
+# Score: 0.5 (the agent talked over the first user turn)
+```
+
+---
+
 ## Custom scorers
 
 You can create custom scorers for domain-specific evaluation needs. See:

js/index.ts

@@ -39,6 +39,7 @@ export * from "./json";
 export * from "./templates";
 export * from "./ragas";
 export * from "./value";
+export * from "./voice";
 export { Evaluators } from "./manifest";
 export { makePartial, ScorerWithPartial } from "./partial";
 export {

js/manifest.ts

@@ -26,6 +26,7 @@ import { ListContains } from "./list";
 import { ScorerWithPartial } from "./partial";
 import { Moderation } from "./moderation";
 import { ExactMatch } from "./value";
+import { VoiceInterruptions, VoiceLatency } from "./voice";
 import { ModelGradedSpec, templates } from "./templates";
 
 interface AutoevalMethod {
@@ -205,4 +206,19 @@ export const Evaluators: {
       },
     ],
   },
+  {
+    label: "Voice",
+    methods: [
+      {
+        method: VoiceLatency,
+        description:
+          "Test how quickly a voice agent replies after the user finishes speaking. Run it with trace scope.",
+      },
+      {
+        method: VoiceInterruptions,
+        description:
+          "Test how often a voice agent talks over the user. Run it with trace scope.",
+      },
+    ],
+  },
 ];

js/voice.test.ts

@@ -0,0 +1,122 @@
+import { expect, test } from "vitest";
+import { type Utterance, VoiceInterruptions, VoiceLatency } from "./voice";
+
+const user = (start: number, end: number): Utterance => ({
+  speaker: "user",
+  start_unix_ms: start,
+  end_unix_ms: end,
+});
+const agent = (
+  start: number,
+  end: number,
+  interrupted?: boolean,
+): Utterance => ({
+  speaker: "agent",
+  start_unix_ms: start,
+  end_unix_ms: end,
+  interrupted,
+});
+
+const latency = (utterances: Utterance[]) =>
+  VoiceLatency({ output: null, utterances });
+const interruptions = (utterances: Utterance[]) =>
+  VoiceInterruptions({ output: null, utterances });
+
+test("VoiceLatency scores the share of fast replies", async () => {
+  const result = await latency([
+    user(0, 2000),
+    agent(2800, 5000),
+    user(6000, 7000),
+    agent(10000, 11000),
+  ]);
+  expect(result.score).toBe(0.5);
+  expect(result.metadata).toMatchObject({
+    replies: 2,
+    unanswered_turns: 0,
+    p50_ms: 800,
+    p95_ms: 3000,
+    gaps_ms: [800, 3000],
+  });
+});
+
+test("VoiceLatency leaves out turns the agent talked over", async () => {
+  const result = await latency([
+    user(0, 3000),
+    agent(2000, 4000),
+    user(5000, 6000),
+    agent(6500, 8000),
+  ]);
+  expect(result.score).toBe(1);
+  expect(result.metadata?.gaps_ms).toEqual([500]);
+});
+
+test("VoiceLatency counts a slightly early reply as instant", async () => {
+  const result = await latency([user(0, 3000), agent(2800, 5000)]);
+  expect(result.metadata?.gaps_ms).toEqual([0]);
+});
+
+test("VoiceLatency doesn't take a short cut-in as the reply", async () => {
+  const result = await latency([
+    user(0, 3000),
+    agent(2000, 2200),
+    agent(3500, 5000),
+  ]);
+  expect(result.metadata?.gaps_ms).toEqual([500]);
+});
+
+test("voice scorers merge user fragments and skip backchannels", async () => {
+  const fragmented = [user(0, 1000), user(1200, 3000), agent(3500, 5000)];
+  expect((await latency(fragmented)).metadata?.gaps_ms).toEqual([500]);
+  expect((await interruptions(fragmented)).metadata?.user_turns).toBe(1);
+
+  const backchannel = [
+    user(0, 1000),
+    agent(1500, 6000),
+    user(3000, 3300),
+    agent(6200, 8000),
+  ];
+  expect((await latency(backchannel)).metadata?.gaps_ms).toEqual([500]);
+  expect((await interruptions(backchannel)).metadata?.user_turns).toBe(1);
+});
+
+test("VoiceInterruptions scores talk-over, not barge-ins", async () => {
+  const result = await interruptions([
+    user(0, 3000),
+    agent(2000, 4000),
+    user(5000, 6000),
+    agent(6500, 8000, false),
+  ]);
+  expect(result.score).toBe(0.5);
+  expect(result.metadata).toMatchObject({
+    talk_overs: 1,
+    user_turns: 2,
+    barge_ins: 0,
+  });
+
+  const bargeIn = await interruptions([agent(0, 3000, true), user(2000, 4000)]);
+  expect(bargeIn.score).toBe(1);
+  expect(bargeIn.metadata?.barge_ins).toBe(1);
+});
+
+test("VoiceInterruptions ignores overlaps under minOverlapMs", async () => {
+  const utterances = [user(0, 3000), agent(2900, 4000)];
+  expect((await interruptions(utterances)).score).toBe(1);
+  expect(
+    (await VoiceInterruptions({ output: null, utterances, minOverlapMs: 50 }))
+      .score,
+  ).toBe(0);
+});
+
+test("voice scorers skip invalid utterances and return null without data", async () => {
+  const utterances = [
+    { speaker: "user" },
+    { speaker: "agent", start_unix_ms: NaN, end_unix_ms: 1000 },
+    { speaker: "agent", start_unix_ms: 2000, end_unix_ms: 1000 },
+    { speaker: "assistant", start_unix_ms: 0, end_unix_ms: 1000 },
+  ] as Utterance[];
+  const result = await latency(utterances);
+  expect(result.score).toBeNull();
+  expect(result.metadata?.skipped).toBe(4);
+  expect((await interruptions(utterances)).score).toBeNull();
+  expect((await interruptions([user(0, 1000)])).score).toBeNull();
+});