Add support for camera talkback (two-way) audio#494
Open
LSalami wants to merge 1 commit into
Open
Conversation
SetupEndpoints previously echoed the controller's own audio port back in the "Accessory Address" TLV of its response, instead of reporting a real local port the accessory listens on. Per the HAP spec (section 11, "IP Cameras"), that field is meant to tell the controller where to send the talkback audio during a live view session -- so no Camera accessory built with this library could ever receive it, regardless of application code, since the controller had nowhere valid to send it to. Adds an opt-in `talkback` option to `Camera.__init__` that, when enabled: - adds a "Speaker" service so HAP clients know two-way audio is supported - opens a local UDP receiver (new `pyhap.camera_talkback` module) that decrypts incoming SRTP with the already-negotiated key and parses the RTP framing - reports that receiver's real port in the SetupEndpoints response - dispatches each received packet's payload (still codec-encoded, decoding is left to the caller same as the existing outbound path) to a new overridable `talkback_audio_received(session_id, payload)` method Off by default; existing accessories are unaffected (verified: all existing tests pass unchanged). Requires the optional `pylibsrtp` dependency (`pip install HAP-python[Talkback]`) only when `talkback=True` is used.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add support for camera talkback (two-way) audio
Fixes #446
The problem
SetupEndpointscurrently echoes the controller's own audio port back inthe
Accessory AddressTLV of its response, instead of reporting a reallocal port the accessory listens on:
Per the HAP spec (section 11, "IP Cameras" — the
Accessory Addressfield ofthe
SetupEndpointsresponse, distinct from theController Addresstheaccessory receives in the request), this field is meant to tell the
controller where it should send the talkback audio during a live view
session. Because this library has always echoed the controller's own port
back at it instead, no Camera accessory built with HAP-python — regardless of
what the application code does — could ever receive that audio: the
controller had nowhere valid to send it.
This is a real, long-standing gap, not just a theoretical reading of the spec.
#446 describes exactly this symptom two years ago: "I do see that my phone
sends an RTP stream back to HAP-python when the talk button is pressed but I
don't know how I can get the properties of that stream in advance, mainly
which port, codec and encryption is used." — because the port it needs was
never a real one to begin with. It's also why two-way audio has never worked
in Home Assistant's HomeKit Bridge integration (which uses this library),
tracked there as an open feature request with no implementation in several
years either.
The fix
An opt-in
talkbackoption onCamera.__init__. When enabled:A
Speakerservice is added (mirroring theMicrophoneservice this classalready adds unconditionally), so HAP clients know the accessory supports
receiving two-way audio and show the corresponding UI (e.g. the microphone
button during a live view in the iOS Home app).
set_endpointsopens a local UDP receiver (newpyhap.camera_talkbackmodule) bound to an OS-assigned port, and reports that port in the
response instead of echoing the controller's.
The receiver decrypts incoming SRTP packets using the same key already
negotiated earlier in the same
set_endpointscall (not renegotiated), andparses the RTP framing.
Each received packet's payload — still encoded with whichever audio codec
was negotiated (typically Opus), decoding is intentionally left to the
caller — is dispatched to a new overridable method:
Leaving codec decoding to the caller mirrors how
start_streamalreadyleaves the outgoing video/audio pipeline (ffmpeg or otherwise) to the
implementation rather than pyhap. It also avoids adding a hard dependency on
an Opus decoder to this library for a codec choice the negotiation makes at
runtime (AAC-ELD is also possible per the spec).
The new
pylibsrtpdependency is optional (pip install HAP-python[Talkback]), only required whentalkback=Trueis actually used.Backward compatibility
talkbackdefaults toFalse. With it disabled (the default, i.e. everyexisting accessory), behavior is unchanged — verified by running the full
existing test suite unmodified: all pre-existing tests pass.
Testing
tests/test_camera.py:talkback=Trueadds a muted-by-defaultSpeakerservice.SetupEndpointsreports a real, distinct local port when talkback isenabled (vs. the pre-existing echo behavior when it isn't).
the same key negotiated in the
SetupEndpointsrequest), sends it overan actual UDP socket to the reported port, and asserts
talkback_audio_receivedis called with the correct decrypted payload.iOS Home app and a real HomeKit pairing (not just synthetic packets in a
test) via a Home Assistant custom component built on top of an earlier,
in-tree version of this same fix. Logs from that session show a sustained,
correctly-timed stream of decrypted/decoded audio packets while
speaking into the Home app's microphone button during a live camera view.
Happy to share more detail on that setup if useful for review.
Notes for reviewers / open questions
talkback— happy to rename if you'd prefer somethingelse (
two_way_audio,support_talkback, etc.)you'd rather this library also handled Opus decoding for the common case.
Camera._talkback_payload_receivedhops ontoself.driver.loopviacall_soon_threadsafebefore calling the public (synchronous)talkback_audio_received, since the receiver's socket loop runs in its ownbackground thread. Let me know if you'd rather this be a coroutine
(
await self.talkback_audio_received(...)) instead of a plain callback —went with synchronous since most consumers will just want to hand the bytes
off to a queue/pipe rather than do async work per-packet, but no strong
opinion.