Skip to content

feat(api): support audio fixture playback in Live sessions - #1185

Merged
yoni-oai merged 1 commit into
mainfrom
castiron/promotions/pr-269
Oct 16, 2026
Merged

yoni-oai merged 1 commit into
mainfrom
castiron/promotions/pr-269

Conversation

@yoni-oai

Copy link
Copy Markdown
Contributor

Summary

Send an audio fixture in a Live session and handle the events for fixture acceptance and server playback completion.

Changes

  • Send up to 30 seconds of Base64-encoded raw mono PCM16 little-endian audio at 24 kHz, with no WAV or other container header with session.output_audio.fixture.play.
  • Handle session.output_audio.fixture.playing when the server accepts the audio, and correlate it with the play event using client_event_id.
  • Handle session.output_audio.fixture.played when the server mixer drains. Audio buffered by the transport may still be playing.
  • Send a fixture from an attached session with LiveSidebandConnection.send(OutputAudioFixturePlayEvent).

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Castiron custom code

Evaluated main: 75b46c0b774cadd323b9b31d24ce150a56700d91.

✅ No new custom-code files detected.

113 mixed files remain; 0 existing customizations changed; 3 generated baselines changed.

Compared 75b46c0b774c → 8dae95545d93. Generated baselines verified.

File Result Current custom patch
openai-java-core/src/test/kotlin/com/openai/models/live/ClientEventTest.kt Generated baseline changed +8 / −2
openai-java-core/src/test/kotlin/com/openai/models/live/ServerEventTest.kt Generated baseline changed +12 / −6
openai-java-core/src/test/kotlin/com/openai/models/live/forks/ForkServerEventTest.kt Generated baseline changed +13 / −6
110 existing customizations unchanged
  • openai-java-core/src/main/kotlin/com/openai/models/audio/AudioResponseFormat.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/agents/environments/EnvironmentCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/agents/vaults/credentials/CredentialAuth.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/agents/vaults/credentials/CredentialAuthCreateParam.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/agents/vaults/credentials/CredentialAuthRotateParam.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/responses/BetaResponse.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/responses/BetaResponseError.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/responses/BetaResponseInjectEvent.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/responses/BetaResponseInjectFailedEvent.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/responses/BetaResponseStreamEvent.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/responses/BetaResponseToolSearchOutputItemParam.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/responses/BetaResponsesServerEvent.kt
  • openai-java-core/src/main/kotlin/com/openai/models/chat/completions/ChatCompletionCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/chat/completions/ChatCompletionMessageFunctionToolCall.kt
  • openai-java-core/src/main/kotlin/com/openai/models/chat/completions/ChatCompletionToolMessageParam.kt
  • openai-java-core/src/main/kotlin/com/openai/models/conversations/ConversationCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/conversations/items/ItemCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/embeddings/Embedding.kt
  • openai-java-core/src/main/kotlin/com/openai/models/embeddings/EmbeddingCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/live/ResponseItemCreateEvent.kt
  • openai-java-core/src/main/kotlin/com/openai/models/realtime/RealtimeResponseStatus.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/Response.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseError.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseFunctionToolCall.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseFunctionWebSearch.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseInputItem.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseStreamEvent.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseTextConfig.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseToolSearchOutputItemParam.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponsesServerEvent.kt
  • openai-java-core/src/main/kotlin/com/openai/models/videos/Video.kt
  • openai-java-core/src/main/kotlin/com/openai/models/videos/VideoCreateError.kt
  • openai-java-core/src/main/kotlin/com/openai/models/webhooks/UnwrapWebhookEvent.kt
  • openai-java-core/src/main/kotlin/com/openai/models/webhooks/WebhookEndpointWithSecret.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/BetaServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/BetaServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/ImageServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/ResponseServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/ResponseServiceAsyncImpl.kt

70 more in the full report.

A changed generated baseline means this report cannot reliably identify which handwritten lines changed.

Inspect the custom-code diff

Download the exact patch produced by this run (requires repository access):

gh run download 38821281267 --repo openai/openai-java \
  --name castiron-custom-code-38821281267-1 --dir /tmp/castiron-custom-code-38821281267-1
git apply --stat /tmp/castiron-custom-code-38821281267-1/custom-code.patch
cat /tmp/castiron-custom-code-38821281267-1/custom-code.patch

Or reproduce it from an SDK checkout containing the vendored reporter:

git fetch --no-tags origin 75b46c0b774cadd323b9b31d24ce150a56700d91 8dae95545d93d5d77cee16d81235fbb10bc1cae0
python3 scripts/castiron/custom_code_report.py report \
  --base 75b46c0b774cadd323b9b31d24ce150a56700d91 \
  --head 8dae95545d93d5d77cee16d81235fbb10bc1cae0 --fetch --require-head-hash --public \
  --out /tmp/castiron-custom-code-8dae95545d93
cat /tmp/castiron-custom-code-8dae95545d93/custom-code.patch

This is the current full custom patch for mixed files, not an attribution of only the handwritten lines changed by this PR.

Full report and patch

@jiayuhuang-openai
jiayuhuang-openai marked this pull request as ready for review October 15, 2026 04:24
@jiayuhuang-openai
jiayuhuang-openai requested a review from a team as a code owner October 15, 2026 04:24
@openai-sdks

openai-sdks Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

OkTest Summary

✅ 236/236 SDK tests passed in 16.871s for Java SDK PR #1185.

Test results — 42 files
Test Result Time
tests/chat-completions-complex-body.test.ts ✅ Passed 230ms
tests/chat-completions-create.test.ts ✅ Passed 900ms
tests/chat-completions-stream.test.ts ✅ Passed 837ms
tests/files-content-binary.test.ts ✅ Passed 265ms
tests/files-create-multipart.test.ts ✅ Passed 717ms
tests/files-list-pagination.test.ts ✅ Passed 450ms
tests/initialize-config.test.ts ✅ Passed 212ms
tests/instance-isolation.test.ts ✅ Passed 219ms
tests/models-list.test.ts ✅ Passed 415ms
tests/responses-background-lifecycle.test.ts ✅ Passed 341ms
tests/responses-body-method-errors.test.ts ✅ Passed 508ms
tests/responses-cancel-timeout.test.ts ✅ Passed 246ms
tests/responses-cancel.test.ts ✅ Passed 271ms
tests/responses-compact-retries.test.ts ✅ Passed 492ms
tests/responses-compact.test.ts ✅ Passed 378ms
tests/responses-create-advanced-stream.test.ts ✅ Passed 385ms
tests/responses-create-advanced.test.ts ✅ Passed 1.468s
tests/responses-create-disconnect.test.ts ✅ Passed 1.236s
tests/responses-create-errors.test.ts ✅ Passed 361ms
tests/responses-create-malformed-api-responses.test.ts ✅ Passed 281ms
tests/responses-create-retries.test.ts ✅ Passed 379ms
tests/responses-create-stream-failures.test.ts ✅ Passed 1.537s
tests/responses-create-stream-timeout.test.ts ✅ Passed 264ms
tests/responses-create-stream-wire.test.ts ✅ Passed 5.659s
tests/responses-create-stream.test.ts ✅ Passed 321ms
tests/responses-create-terminal-states.test.ts ✅ Passed 426ms
tests/responses-create-timeout.test.ts ✅ Passed 288ms
tests/responses-create.test.ts ✅ Passed 435ms
tests/responses-delete.test.ts ✅ Passed 263ms
tests/responses-input-items-errors.test.ts ✅ Passed 396ms
tests/responses-input-items-list.test.ts ✅ Passed 293ms
tests/responses-input-items-options.test.ts ✅ Passed 367ms
tests/responses-input-tokens-count-timeout.test.ts ✅ Passed 314ms
tests/responses-input-tokens-count.test.ts ✅ Passed 399ms
tests/responses-malformed-inputs.test.ts ✅ Passed 4.352s
tests/responses-not-found-errors.test.ts ✅ Passed 397ms
tests/responses-parse.test.ts ✅ Passed 656ms
tests/responses-retrieve-retries.test.ts ✅ Passed 311ms
tests/responses-retrieve.test.ts ✅ Passed 426ms
tests/responses-stored-method-errors.test.ts ✅ Passed 1.015s
tests/retry-behavior.test.ts ✅ Passed 3.328s
tests/sdk-error-shape.test.ts ✅ Passed 500ms

View OkTest run #38821238486

SDK merge (6c532e6ce1ba) · head (8dae95545d93) · base (75b46c0b774c) · OkTest (63b471b22245)

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T23:06:13.752274Z 8dae955 New commits
🔒 Security Review ✅ Completed 2026-10-15T23:08:17.337486Z 8dae955 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@yoni-oai
yoni-oai force-pushed the castiron/promotions/pr-269 branch from 775ac8f to 7eb7e69 Compare October 15, 2026 21:59

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7eb7e69d68

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread openai-java-core/src/main/kotlin/com/openai/models/live/ServerEvent.kt Outdated

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full 18-file diff at 7eb7e69, including the generated models/unions, sideband send path, and all changed tests.

Requesting changes for the existing visitor compatibility finding, which I independently confirmed. The new fixture methods in ServerEvent.Visitor, ClientEvent.Visitor, and ForkClientEvent.Visitor are abstract. Existing implementations therefore fail to recompile, and an already-compiled visitor can throw AbstractMethodError when accept() dispatches a newly typed fixture event. Please give these added methods a default that delegates to unknown(...), as ForkServerEvent.Visitor already does, and cover an old-style visitor consumer. No duplicate inline comment added.

The maximum-size outbound fixture regression suggested in the other existing comment would also be useful, but I did not establish a production size-rejection bug: the default message limit is Int.MAX_VALUE. The blocking issue above is the visitor API break.

Static source review only; no local code/tests/builds were executed. All current non-skipped hosted checks are green. I inspected the successful test job and the API-compatibility log; the latter reports structured-output API probes, not an old Live visitor consumer test. No additional security finding in the reviewed diff.

@markstuart-oai markstuart-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete 18-file change at 7eb7e69d6851e40c0f3c29e20267632c5a44af65.

The existing visitor compatibility finding still needs correction. ClientEvent.Visitor, ServerEvent.Visitor, and ForkClientEvent.Visitor add abstract methods. Existing implementations fail to recompile, and an already compiled visitor can throw AbstractMethodError when a new fixture event is dispatched instead of reaching its unknown(...) handler. Generate default methods delegating to unknown(JsonValue.from(event)), as ForkServerEvent.Visitor already does, and add an old-style visitor consumer regression. The project enables -Xjvm-default=all. I independently confirmed the existing finding and have not duplicated its inline thread.

The fixture models, union mappings, and sideband command guard otherwise look consistent with the schema. The maximum-size fixture test suggested in the other thread would strengthen coverage; I did not establish a production size-rejection defect because the default message limit is Int.MAX_VALUE.

All 14 non-skipped current-head hosted checks passed; one queue signal was skipped. Review covered source, checked-in tests, and transport/serialization trust boundaries. No local tests, builds, generation, or live API calls were run.

Castiron-Internal-PR: openai/openai-java-internal#269
Castiron-Source-SHA: 5efa70326247e2eaedff12cafbcc5067257e6f2a
Castiron-Public-Base-SHA: 75b46c0
@yoni-oai
yoni-oai force-pushed the castiron/promotions/pr-269 branch from 7eb7e69 to 8dae955 Compare October 15, 2026 23:03

@markstuart-oai markstuart-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete 20-file change at 8dae95545d93d5d77cee16d81235fbb10bc1cae0. No actionable findings remain.

The new fixture visitor methods now delegate to unknown(JsonValue.from(event)). Together with the existing -Xjvm-default=all setting, this resolves the earlier abstract-method compatibility defect. Existing overrides retain their dispatch behavior. The whole-union setting also makes 44 existing methods optional: new visitors lose compile-time exhaustiveness, while the default unknown still throws unless the consumer overrides it. That scope is deliberate and consistent across the three selected unions.

The compatibility test exercises a visitor with the old method surface and checks that all four new methods are JVM defaults. The sequential payload regression sends all 1,920,000 Base64 characters for a synthetic 30-second PCM fixture through the public sideband connection to a local WebSocket peer, without raising the default limit. These address the two existing review threads. They do not execute a separately compiled old consumer or verify remote audio playback.

At the latest check, hosted API compatibility and every other completed, non-skipped check passed; the test job was still running. Review covered source, existing tests, and transport/serialization trust boundaries. No local tests, builds, generation, or live API calls were run.

@yoni-oai
yoni-oai disabled auto-merge October 15, 2026 23:24

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 8dae95545d93d5d77cee16d81235fbb10bc1cae0. The visitor compatibility blocker from my previous review is fixed; approving this revision.

I reviewed the complete six-file update against my prior full-diff review. The four added fixture visitor methods now delegate to unknown(JsonValue.from(event)), and the existing -Xjvm-default=all setting makes them actual JVM default methods. Existing implementations keep their overrides and no longer need to implement the new methods. The whole-union configuration also makes 44 pre-existing methods optional: this intentionally relaxes compile-time exhaustiveness, while the default unknown still throws unless the caller overrides it.

The new compatibility test exercises an old-shaped visitor and verifies all four new methods are JVM defaults. The added public-entrypoint sideband test sends the full 1,920,000-character Base64 fixture through OkHttp to a local WebSocket peer without raising the default limit. These tests cover the reported regression; they do not constitute a separately compiled old-consumer test or remote playback validation.

All 14 current-head non-skipped hosted checks now pass, including tests, API compatibility, Jackson compatibility, Java 8/25 runtime compatibility, and the trusted custom-code checks. I also reviewed the companion configuration Delta #2543303; it has an exact-head peer approval, so I am not duplicating that approval. Its CI is still running.

Static source/security-focused review only; no local code, tests, builds, generation, or live calls were executed. No new findings. No budget, ownership, workflow, or publication policy changes are included.

@yoni-oai
yoni-oai added this pull request to the merge queue Oct 16, 2026
Merged via the queue into main with commit c36bc50 Oct 16, 2026
15 checks passed
@openai-sdks openai-sdks Bot mentioned this pull request Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants