Repository navigation
refactor(livekit): Record audio through shared audio pipeline - #2657
Merged
Merged
Conversation
David Elner (delner)
force-pushed
the
delner/audio-recorder-pipeline
branch
from
October 27, 2026 03:13
302df9e to
8f0d11b
Compare
David Elner (delner)
changed the title
David Elner (delner)
force-pushed
the
delner/audio-recorder-pipeline
branch
from
October 27, 2026 03:32
8f0d11b to
efc93db
Compare
Luca Forstner (lforst)
approved these changes
Oct 27, 2026
David Elner (delner)
force-pushed
the
delner/audio-recorder-pipeline
branch
from
October 27, 2026 16:16
f7364f3 to
d57d162
Compare
David Elner (delner)
marked this pull request as ready for review
October 27, 2026 16:29
David Elner (delner)
added a commit
that referenced
this pull request
Oct 27, 2026
) Based on #2657 To provide more efficient, out-of-the-box encoding of audio, this adds `opus` and `ogg` plugins as middleware. ## What it does By default, voice recordings upload as standard Ogg Opus files instead of raw PCM. Install the package to activate: ```sh npm install @braintrust/audio ``` Encoder settings are optional and declarative: ```ts import { configureInstrumentation } from "braintrust/instrumentation"; configureInstrumentation({ integrations: { livekit: { audio: { codec: { name: "opus", options: { bitrate: 32_000 } }, container: { name: "ogg" }, }, }, }, }); ``` - **Optional.** `braintrust` doesn't depend on this package. Without it, recordings fall back to raw PCM, with a warning. - **Per integration.** Each integration owns its defaults: LiveKit records Opus in Ogg. - **Opt out in code.** Set `codec: { name: "pcm" }` to always record raw PCM. ## How it works ``` livekit.audio ─► encoding config ─► loads @braintrust/audio ─► createStack(config) ─► AudioRecorder (+ defaults) (Node only, once) opus → ogg ``` - **Opus** encodes speech with libopus (WebAssembly, via `@audio/encode-opus`) 20 ms at a time, pausing for the event loop every second of audio, so the agent stays responsive. - **Ogg** wraps the packets in a standard file with no extra dependencies. The file plays back at exactly the recorded length. - **Declarative config, middleware underneath.** The SDK turns `codec` and `container` into the internal middleware pipeline; users never write middleware. ### Performance Measured on one 44-second recording segment: about 0.4 s of CPU in total, with the longest pause to the event loop around 10 ms, and 4 MB of PCM uploaded as 344 KB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To facilitate transforms on recorded audio, for the purposes of compression, encoding, or filtering, this adds a configurable hook+middleware stack to the audio recording pipeline.
What it does
Moves recording out of the LiveKit plugin into a reusable
AudioRecorderwith an internal middleware pipeline that can transform each segment before upload.No outward changes to data or UX.
Opus compression and Ogg plugins leverage this pipeline in a separate, optional package.
Design
Internal pipeline:
AudioRecorderplaces PCM frames from each source on a shared timeline and cuts bounded segments. It runs each segment through the middleware and hands the result to the integration.(segment, meta, next), transform on the way in, observe on the way out. Each recording builds its own stack, so state is never shared between calls. Middleware can change the format or channel layout and record facts inmeta.attributes, published with the file.