Skip to content

Add support for camera talkback (two-way) audio#494

Open
LSalami wants to merge 1 commit into
ikalchev:devfrom
LSalami:talkback-audio-support
Open

Add support for camera talkback (two-way) audio#494
LSalami wants to merge 1 commit into
ikalchev:devfrom
LSalami:talkback-audio-support

Conversation

@LSalami

@LSalami LSalami commented Jul 15, 2026

Copy link
Copy Markdown

Add support for camera talkback (two-way) audio

Fixes #446

The problem

SetupEndpoints currently echoes the controller's own audio port back in
the Accessory Address TLV of its response, instead of reporting a real
local port the accessory listens on:

res_address_tlv = tlv.encode(
    ...
    SETUP_ADDR_INFO["AUDIO_RTP_PORT"], struct.pack("<H", target_audio_port),
)

Per the HAP spec (section 11, "IP Cameras" — the Accessory Address field of
the SetupEndpoints response, distinct from the Controller Address the
accessory receives in the request), this field is meant to tell the
controller where it should send the talkback audio during a live view
session. Because this library has always echoed the controller's own port
back at it instead, no Camera accessory built with HAP-python — regardless of
what the application code does — could ever receive that audio: the
controller had nowhere valid to send it.

This is a real, long-standing gap, not just a theoretical reading of the spec.
#446 describes exactly this symptom two years ago: "I do see that my phone
sends an RTP stream back to HAP-python when the talk button is pressed but I
don't know how I can get the properties of that stream in advance, mainly
which port, codec and encryption is used."
— because the port it needs was
never a real one to begin with. It's also why two-way audio has never worked
in Home Assistant's HomeKit Bridge integration (which uses this library),
tracked there as an open feature request with no implementation in several
years either.

The fix

An opt-in talkback option on Camera.__init__. When enabled:

  • A Speaker service is added (mirroring the Microphone service this class
    already adds unconditionally), so HAP clients know the accessory supports
    receiving two-way audio and show the corresponding UI (e.g. the microphone
    button during a live view in the iOS Home app).

  • set_endpoints opens a local UDP receiver (new pyhap.camera_talkback
    module) bound to an OS-assigned port, and reports that port in the
    response instead of echoing the controller's.

  • The receiver decrypts incoming SRTP packets using the same key already
    negotiated earlier in the same set_endpoints call (not renegotiated), and
    parses the RTP framing.

  • Each received packet's payload — still encoded with whichever audio codec
    was negotiated (typically Opus), decoding is intentionally left to the
    caller — is dispatched to a new overridable method:

    def talkback_audio_received(self, session_id, payload: bytes) -> None:
        """Override to consume the received audio."""

Leaving codec decoding to the caller mirrors how start_stream already
leaves the outgoing video/audio pipeline (ffmpeg or otherwise) to the
implementation rather than pyhap. It also avoids adding a hard dependency on
an Opus decoder to this library for a codec choice the negotiation makes at
runtime (AAC-ELD is also possible per the spec).

The new pylibsrtp dependency is optional (pip install HAP-python[Talkback]), only required when talkback=True is actually used.

Backward compatibility

talkback defaults to False. With it disabled (the default, i.e. every
existing accessory), behavior is unchanged — verified by running the full
existing test suite unmodified: all pre-existing tests pass.

Testing

  • Added unit tests in tests/test_camera.py:
    • talkback=True adds a muted-by-default Speaker service.
    • SetupEndpoints reports a real, distinct local port when talkback is
      enabled (vs. the pre-existing echo behavior when it isn't).
    • An end-to-end test that builds a real SRTP-encrypted RTP packet (using
      the same key negotiated in the SetupEndpoints request), sends it over
      an actual UDP socket to the reported port, and asserts
      talkback_audio_received is called with the correct decrypted payload.
  • Beyond the unit tests in this PR: this fix has been verified against a real
    iOS Home app and a real HomeKit pairing (not just synthetic packets in a
    test) via a Home Assistant custom component built on top of an earlier,
    in-tree version of this same fix. Logs from that session show a sustained,
    correctly-timed stream of decrypted/decoded audio packets while
    speaking into the Home app's microphone button during a live camera view.
    Happy to share more detail on that setup if useful for review.

Notes for reviewers / open questions

  • I called the option talkback — happy to rename if you'd prefer something
    else (two_way_audio, support_talkback, etc.)
  • Payload delivery is raw/undecoded by design (see above) — flagging in case
    you'd rather this library also handled Opus decoding for the common case.
  • Camera._talkback_payload_received hops onto self.driver.loop via
    call_soon_threadsafe before calling the public (synchronous)
    talkback_audio_received, since the receiver's socket loop runs in its own
    background thread. Let me know if you'd rather this be a coroutine
    (await self.talkback_audio_received(...)) instead of a plain callback —
    went with synchronous since most consumers will just want to hand the bytes
    off to a queue/pipe rather than do async work per-packet, but no strong
    opinion.

SetupEndpoints previously echoed the controller's own audio port back in
the "Accessory Address" TLV of its response, instead of reporting a real
local port the accessory listens on. Per the HAP spec (section 11, "IP
Cameras"), that field is meant to tell the controller where to send the
talkback audio during a live view session -- so no Camera accessory built
with this library could ever receive it, regardless of application code,
since the controller had nowhere valid to send it to.

Adds an opt-in `talkback` option to `Camera.__init__` that, when enabled:
- adds a "Speaker" service so HAP clients know two-way audio is supported
- opens a local UDP receiver (new `pyhap.camera_talkback` module) that
  decrypts incoming SRTP with the already-negotiated key and parses the
  RTP framing
- reports that receiver's real port in the SetupEndpoints response
- dispatches each received packet's payload (still codec-encoded, decoding
  is left to the caller same as the existing outbound path) to a new
  overridable `talkback_audio_received(session_id, payload)` method

Off by default; existing accessories are unaffected (verified: all
existing tests pass unchanged).

Requires the optional `pylibsrtp` dependency (`pip install
HAP-python[Talkback]`) only when `talkback=True` is used.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Support For Camera 2-way Audio

1 participant