feat: microphone redirection (MS-RDPEAI) — the client's mic as a real macOS input device - #191
Merged
Merged
Conversation
… 0 protocol gate The RDP client redirects its microphone (a standalone mic, or a webcam's built-in mic) over the AUDIO_INPUT dynamic virtual channel; macrdp — the server — receives it. Phase 0 is the go/no-go gate: negotiate the channel and log the client streaming its mic, before any macOS virtual-audio-device work. Unlike MS-RDPECAM/MS-RDPEUSB, audio input is a SINGLE channel (client-push, no enumerator + per-device split), so the vendored processor collapses to one DvcProcessor with no ServerEventSender and no create_channel path. The server speaks first: Version -> Formats (permissive PCM set) -> Open (client's first format) -> consume inbound Data PDUs. AUDIO_FORMAT/WAVEFORMATEX is reused from ironrdp-rdpsnd; only the MS-RDPEAI framing is new. process() tolerates every decode so a malformed PDU can't tear down the session for an opt-in feature. - vendored ironrdp-server: new src/audin.rs (AudinServer + AudinServerFactory + the AudinSampleSink Phase-2 seam). Wired into the post-#174 factory architecture (divergence 24): Rc-stored audin_factory attached via attach_channels_impl on both the normal AND candidate paths, and carried in NegotiationContext so a preempting candidate advertises AUDIO_INPUT too (safe: stateless per-connection, unlike the process-wide multitransport state that candidates skip). - macrdp: src/audin/mod.rs (MacAudin factory), --enable-microphone-redirection flag + ENABLE_MICROPHONE_REDIRECTION config key. Inert/byte-identical when off. Rebased onto current main (#174/#175/#176). Builds green: fmt + clippy -D warnings + 198 tests. NOT yet live-verified against a real client — next step is FreeRDP /microphone.
…c PCM Wire the AudinSampleSink seam (Phase-0 stubbed it) to a WavDumpSink that writes the client's inbound microphone PCM to a canonical WAV under $TMPDIR, gated by MACRDP_MIC_DUMP=1 (the audio analogue of MACRDP_CAMERA_DUMP). Off by default → the factory hands the processor None and this is never constructed (byte-identical to Phase 0). The point is to verify — by playing back the WAV — that the negotiated PCM the client streams is real, correctly-framed audio BEFORE Phase 2 (the AudioServerPlugIn virtual mic) is built on the same sink. The two WAV length fields are patched from the byte count on drop, so a disconnect still leaves a playable file. build + clippy -D warnings + stable/nightly fmt clean.
…able without Ctrl-C WavDumpSink only finalized the RIFF/data size fields in Drop, which under a foreground macrdp run doesn't fire until the process exits — so a live dump had zeroed sizes and read as silence in players that honor them. Re-patch the two size fields every FINALIZE_EVERY_PACKETS (~0.5s), seeking back to the end to keep appending; Drop reuses the same helper. Data was always intact; this just keeps the header current.
… tone) The macOS Phase-2 side of MS-RDPEAI mic redirection: a from-scratch CoreAudio HAL plug-in (route A — no entitlement, installs like the IFD handler) presenting a single virtual audio INPUT device 'macrdp Microphone'. P2a delivers an internal 440 Hz test tone in DoIOOperation so the whole novel path can be verified — builds without Xcode, coreaudiod loads it, the device appears in input pickers — before the shared-memory feed (P2b) is wired behind the AudinSampleSink seam. - audioplugin/macrdp_mic.c: the CFPlugIn factory + AudioServerPlugInDriverInterface (IUnknown, property system for plug-in/device/stream objects, StartIO/StopIO, a host-clock-driven GetZeroTimeStamp, DoIOOperation). Stereo Float32 44100 Hz. - packaging/make-audio-plugin.sh: clang -bundle -> universal .driver, monotonic CFBundleVersion, Developer-ID sign, hardened runtime, NO entitlements. - packaging/install-audio-plugin.sh: privileged copy to /Library/Audio/Plug-Ins/HAL + coreaudiod restart (one GUI admin prompt); --uninstall. Compiles clean (-Wall -Wextra), factory symbol exported, links CoreAudio + CoreFoundation. coreaudiod-load + device-appears live-verify is the next step.
coreaudiod queries kAudioDevicePropertyStreams per scope; returning the input stream for the OUTPUT scope made 'macrdp Microphone' report 2 phantom output channels (apps could try to play into it). Return an empty stream list for the output scope so it's a clean input-only device. Verified: system_profiler now shows Input Channels: 2 with no Output Channels.
…ss-user The one real unknown in P2b was whether a CoreAudio HAL plug-in inside coreaudiod (user _coreaudiod, sandboxed) can read shared memory written by macrdp (the logged-in user). It can — verified through the audio itself. - audioplugin/macrdp_mic_ring.h: the SPSC ring layout (POSIX shm /macrdp_mic_ring, 0666, Float32 stereo, 65536-frame power-of-two ring, atomic write/read cursors). The Rust writer (P2b-1) mirrors this. - macrdp_mic.c: DoIOOperation drains the ring when mapped+valid (overrun drops to half a ring; underrun fills silence), else the 440 Hz fallback tone. Mapped at StartIO / unmapped at StopIO (off the real-time thread). - micfeed_test.c: throwaway bring-up writer feeding a 220 Hz real-time tone. PROOF: install the plug-in, run micfeed_test, record the device — recorded 220 Hz @ -20 dB (the feed), NOT the 440 Hz @ -26 dB fallback. So coreaudiod's sandbox permits shm_open of an external, different-user 0666 segment. macrdp -> shm -> coreaudiod plug-in -> app is a working path. Next: the real Rust SharedMemSink.
macrdp's side of the shared-memory feed: SharedMemSink : AudinSampleSink maps the same POSIX ring the plug-in reads (byte-for-byte mirror of macrdp_mic_ring.h), converts received 16-bit PCM (mono upmixed to stereo, or stereo) to Float32, and publishes it with release ordering. Wired into src/audin/mod.rs build_processor: MACRDP_MIC_DUMP still wins (WAV debug), else (macOS) the feed, else None — a feed that can't map falls back to None so a mic setup problem never kills the session. C reader now acquire-loads the magic (arm64 release/acquire pair with the writer). Verified client-free against the installed plug-in: a layout test pins the Rust struct to the C offsets, and feeding a 330 Hz tone through the REAL SharedMemSink recorded 330 Hz @ -20 dB (= amplitude 0.1) off macrdp Microphone — so the layout, the atomic ordering, and the PCM16->f32 stereo conversion are all correct. Next: the full RDP chain (real client mic -> audin -> this sink -> plug-in -> app).
…lable) A stale MACRDP_MIC_DUMP in the shell silently routes the mic to the WAV dumper instead of the shared-ring feed, so the virtual mic plays its fallback tone and the failure is invisible. build_processor now logs the choice: 'MACRDP_MIC_DUMP set — dumping ... NOT feeding the virtual mic', 'feeding the macrdp Microphone shared ring', or a warn when the ring can't be mapped (plug-in not installed).
…on reuse) macOS permits ftruncate on a POSIX shm object only ONCE — when it first sizes a freshly-created 0-length segment. On reconnect (or any reuse), SharedMemSink::new re-opened the existing segment and called ftruncate again, which fails EINVAL (errno 22), so new() returned None and the mic silently fell back to the plug-in's tone. fstat first and ftruncate only when st_size == 0; reuse an already-sized segment as-is (page-rounded size >= struct size is fine). LIVE-VERIFIED end-to-end: real Win11 client mic -> RDP AUDIO_INPUT -> macrdp audin -> shared ring -> macrdp Microphone HAL plug-in -> recorded voice (crest ~19 dB = speech). Reconnect path confirmed (reused=true maps successfully). The client-free tests missed this because each shm_unlink'd first, always hitting the create path.
Two DoIOOperation changes for a real installed mic: - No-feed fallback is now SILENCE, not the 440 Hz bring-up tone (a system-wide mic must be quiet when no client is connected). The tone stays behind a compile-time MACRDP_MIC_FALLBACK_TONE=0/1 for feed-path diagnosis. - Bound latency: the reader maps the ring mid-stream with the writer already ahead, so it sat up to ~0.75 s behind; now skip ahead to keep ~100 ms of the freshest audio, dropping only past a ~250 ms backlog so RDP network jitter doesn't over-drop. Target clamped to the ring size. Silence verified: idle recording is Peak/RMS -inf (digital zeros) vs the old -26 dB tone. Latency bound pending a live client re-confirm (voice natural + more responsive + no dropouts).
…dp does it) Prompted by macrdp's own MS-RDPEAI feature — the mistake (claiming a mic first) would have been easy since USB/UDP/camera above ARE firsts. Verified by direct source read 2026-09-01: xrdp presents the client mic as a real recordable OS input device (a PulseAudio source), so it precedes macrdp end-to-end. - pulseaudio-module-xrdp builds module-xrdp-source.so (audio input), not just sink. - xrdp/sesman/chansrv/sound.c: sound_start_source_listener() + PA_CMD_START_REC/ SEND_DATA/STOP_REC + audin_start()/audin_stop() (MS-RDPEAI), FIFO-buffered. New Part-1 section 5 (+ verdicts row + re-verify step); the only defensible framing left is the macOS-Core-Audio angle, and only with the 'as far as is known' hedge.
The clap doc comment for --enable-camera-redirection has said "Phase 0 protocol gate only ... does NOT present a camera yet (no per-device channel, no stream, no macOS code)" since the feasibility spike. Camera redirection SHIPPED in v0.9.0 — it presents "macrdp Camera" as a real macOS camera, live-verified on mstsc at 1080p/~30 fps — so that text has been wrong in every released binary's --help since. Replaced with what the feature actually does, matching the docs/cli.md entry: the pipeline, the client-side opt-in, the one-time system-extension requirement (and what happens without it), the MACRDP_CAMERA_DUMP debug knob, and a pointer to docs/camera-extension-setup.md instead of the long-superseded feasibility doc. Also dropped the EXPERIMENTAL prefix, which docs/cli.md already does not carry for this flag. Help text only — no behavior change.
…0 only" The clap doc comment still described the flag as a "Phase 0 protocol gate only" that "does NOT present a macOS microphone yet (no virtual audio device)". That stopped being true at P2b: the received PCM is fed to the "macrdp Microphone" AudioServerPlugIn over a shared-memory ring, and the full chain is live- and ear-verified against a real Win11 client (2026-09-01) — natural voice at normal pitch through QuickTime. Rewritten to describe the shipped behavior: what appears on the Mac and where, the shared-ring feed, the P2c guarantees (idle = digital silence, feed bounded at ~100 ms), the client-side opt-in for both mstsc and FreeRDP, the one-time plug-in install via packaging/install-audio-plugin.sh and what happens without it, and the MACRDP_MIC_DUMP debug knob. Kept EXPERIMENTAL — unlike camera this has not shipped in a release yet — and corrected "Cross-platform (pure protocol)", which was true of the Phase 0 gate but not of the device half. Help text only — no behavior change. Embedding the .driver in macrdp.app (so the install is not a manual copy) remains a P3 item.
…o (25) PR #182 (@antonmos, "bound accept_finalize") claims vendored-server divergence (24) for its FINALIZE_TIMEOUT. This branch already claims (24) for MS-RDPEAI microphone redirection. Neither number exists on main, so both branches saw it as free. #182 is review-ready while this branch still needs P3, so #182 lands first and keeps (24); the mic divergence becomes (25). Recorded at the divergence heading so the rename isn't missed when this branch rebases onto a main that carries #182 — two (24)s in the log is the same bookkeeping slip that produced #179 at pin-bump time. Docs only.
vendor/ironrdp-server/CLAUDE.md, divergence (24) - Un-split the heading: the (25) renumber marker had been inserted mid-sentence; it's now its own paragraph. - Status no longer says "Phase 0 (protocol gate) only" or "Phase 0 passes None": the processor feeds a real sink (P1 WAV dump, P2 shared-memory ring into the AudioServerPlugIn), live- and ear-verified 2026-09-01. - "Next: Phase 2" is now done; P3 remains. - Replaces "cleanly upstreamable as the server counterpart to a (nonexistent-upstream) client": upstream already ships ironrdp-rdpeai (IronRDP#1645) with RdpeaiServer, and #1946 wires it into ironrdp-server, so the question at the next bump is adopting it rather than upstreaming this. TODO.md - Replaces the one-line P3 entry with the real remaining checklist: CLI help (done), driver embedding, cleanup, missing docs and stale source wording, the unverified 2026-09-12 review findings (security items first), the (24)->(25) renumber, the upstream overlap, and the rebase. Docs only.
A third branch claims vendored divergence (24): PR #183's per-served-connection input-reset handle, alongside PR #182's FINALIZE_TIMEOUT and this branch's MS-RDPEAI processor. Decided by expected merge order: #183 keeps (24), #182 takes (25), and this divergence becomes (26). Updates the collision marker at the divergence heading and the P3 checklist item, and records the rule if the order changes: take the next free number on main at merge time. Docs only.
#182 was held for the pin bump on 2026-09-17 rather than landing its vendored divergence — the identical bound is already upstream in IronRDP#1890, so macrdp harvests it. That leaves #183 keeping (24) and frees (25) for this divergence. Updates the collision marker and the P3 checklist item. Docs only.
…ped) Verifies and fixes the Mac-side findings from the 2026-09-12 review. Security: - The shared ring was created 0666 without O_EXCL and never unlinked, so any local account could record or inject the redirected mic. It is now created only once the client negotiates the mic (never for an unauthenticated connection: macrdp always uses NLA), exclusively (a stale segment is replaced, one that can't be is refused), mode 0644, and wiped + unlinked when the session ends. The plug-in opens it read-only and keeps its read position privately. - The plug-in trusted ring_frames/channels from the segment (out-of-bounds read inside coreaudiod). It now validates the header against its own constants, rejects group/other-writable segments, and indexes only with compile-time sizes. - StopIO unmapped the ring while the real-time IO thread could read it. A 1 s background timer now maps, follows and releases segments; a replaced mapping is unmapped at least 500 ms later, never from StopIO. This also fixes a device opened before the feed started staying silent. Two macOS facts behind the design, verified by probe: shm_open applies its mode exactly (no umask) and fchmod on shm fails with EINVAL (the old fchmod was silently failing); fstat reports st_ino 0 for shm objects, so segments are told apart by a random session id in the header. Also: MACRDP_MIC_DUMP uses the camera's =1/true rule and is bridged from config.env; the WAV dump finalizes and resets on renegotiation; the Rust tests parse the C header's #defines and the header _Static_asserts the offsets; micfeed_test.c speaks v2; an fstat failure no longer logs an uninitialized size; stale "Phase 0" / "440 Hz test tone" wording removed. Trust boundary documented in docs/macos-gotchas.md (5): other local accounts can still read a live stream. Live-verified on a real Windows 11 mstsc mic: 0644 v2 segment created on negotiation, voice recorded off "macrdp Microphone", segment wiped and unlinked on disconnect; a device already recording picks up a late feed within ~1 s; a 0666 segment with a valid header is ignored. 254 tests pass.
…l from the controller - make-app.sh builds macrdp-mic.driver (signed with the app's identity) into macrdp.app/Contents/Resources next to install-audio-plugin.sh, the way the smart-card IFD handler ships. - install-audio-plugin.sh finds the driver next to itself first, so the copy embedded in the app installs from there. - The controller's Redirection tab gains a Microphone section: the ENABLE_MICROPHONE_REDIRECTION toggle, Install / Update / Remove buttons (running the embedded installer: one admin prompt, a coreaudiod restart), and a status line comparing the installed driver's build number with the bundled one. - make-audio-plugin.sh skips the secure timestamp for the self-signed macrdp-dev identity, as make-app.sh does. - packaging/README.md documents it. Verified: a signed make-app.sh build embeds the driver (Developer ID, secure timestamp) and passes codesign --verify --deep --strict; the installer run from inside the app resolves the embedded copy. End to end in the controller: it reported the bundled driver as newer, Update installed it (root:wheel, valid signature, loaded by coreaudiod) and the status changed to installed; the device still carries a fed test tone at the expected level.
- docs/audio.md: a user guide section — controller install, client settings (mstsc "Record from this computer", FreeRDP /microphone), what to expect, the limits (other accounts can listen to a live stream; 44.1 kHz only for now), and MIC_DUMP. Linked from the README's Audio row. - docs/features.md, docs/cli.md, docs/configuration.md: the flag, pipeline, setup and caveats. No "first" claim — xrdp already presents a client mic. - docs/architecture.md: src/audin/ and audioplugin/ entries. - packaging/config.env.example: ENABLE_MICROPHONE_REDIRECTION and MIC_DUMP. - --help: points at the controller install; notes the 44.1 kHz limit and the privacy boundary. - TODO.md: packaging and docs marked done. Numbers checked against the source: the vendored protocol layer advertises 44.1 and 48 kHz (mono + stereo); the latency bounds are 4410 / 11025 frames (100 / 250 ms).
The protocol layer advertised 48 kHz too and opened whatever the client picked, so a 48 kHz client played at the wrong pitch. It now advertises and opens only 16-bit PCM at 44.1 kHz (the first acceptable entry in the client's list, by its index), follows a Format Change only to an acceptable format (dropping data otherwise), and serves a newer client at version 1. A client with no acceptable format gets no mic, and that is logged. Causal tests in src/audin/mod.rs (4 of 5 fail on the old code). Divergence (24) is reserved for #183, so the mic is (25). It ships ahead of the IronRDP pin bump; upstream ironrdp-rdpeai replaces it at the bump. Docs and help text no longer describe the old wrong-pitch behaviour.
clintcan
added a commit
that referenced
this pull request
Sep 30, 2026
The client's microphone presents as "macrdp Microphone", a real macOS input device (#191). Opt-in and EXPERIMENTAL (--enable-microphone-redirection, or the GUI controller); the Core Audio driver ships in macrdp.app and installs from the controller. 16-bit PCM 44.1 kHz only. Protocol layer is vendored divergence (25); upstream ironrdp-rdpeai replaces it at the next pin bump. Also #190: --lock-on-disconnect holds while a reconnect is handshaking. Pre-tag gates: fmt clean (stable + nightly), clippy -D warnings clean, 259 tests passing. Docs: release-history, README status, CLAUDE.md status.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
EXPERIMENTAL, opt-in (
--enable-microphone-redirection, configENABLE_MICROPHONE_REDIRECTION=1, default OFF). The connecting client redirects its microphone and macrdp presents it as "macrdp Microphone", a real macOS input device (System Settings → Sound → Input, QuickTime, Zoom, Teams).AUDIO_INPUTDVC — vendoredironrdp-serverdivergence (25) ((24) is reserved for fix(input): held modifiers on mouse events + Ctrl+click→Cmd+click remap #183). Only 16-bit PCM at 44.1 kHz is advertised and accepted (no resampler); the client's matching format is opened by its index, a Format Change is followed only to an acceptable format, and a newer client version is served at version 1. A client that can't offer 44.1 kHz gets no mic, and that's logged.audioplugin/) insidecoreaudiod. Ring v2: created lazily and exclusively at mode 0644, read-only reader with a private cursor, header validated, compile-time indexing, wiped and unlinked on session end.macrdp.app; the menu-bar controller installs/removes it (Settings → Redirection → Microphone), orinstall-audio-plugin.sh. One admin prompt, no entitlement.Ships ahead of the IronRDP pin bump; at the bump, upstream
ironrdp-rdpeai(#1645/#1946) replaces the vendored protocol layer and the Mac side carries over.Verification
src/audin/mod.rsdrives the protocol server with client PDUs; 4 of 5 fail on the pre-fix code. Full suite, clippy-D warnings, fmt (stable + nightly), plug-in-Wall -Werror.Not covered
/microphone) not live-tested.