Skip to content

Dictation: transcribe with a local whisper.cpp model on this computer - #881

Open
Dkm0315 wants to merge 2 commits into
mainfrom
feat/840-local-dictation
Open

Dkm0315 wants to merge 2 commits into
mainfrom
feat/840-local-dictation

Conversation

@Dkm0315

@Dkm0315 Dkm0315 commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Dictation with a local model: whisper.cpp transcribes the composer recording on this computer.

What the user can now do

  • Choose Local model under Settings › Chat › Dictation. With whisper-cli installed (brew install whisper-cpp) and a ggml model, the mic and the hold-to-talk shortcut work as they do with Cloud, but the audio never leaves the computer.
  • Muster looks for models in the same places as the muster CLI: ~/.muster/models, ~/.cache/whisper and ~/.cache/whisper.cpp. It also checks Homebrew's share/whisper-cpp/models folder. A model downloaded for the CLI works here as is. You can also paste a path to any .bin model file.
  • When something is missing, the row says so: either "install whisper.cpp" or "put a model in ~/.muster/models", with a download hint.

How it works (same approach as the CLI's voice.ts whisper-cpp provider, ported because the repos share no package)

  • Renderer: when the backend is Local, Chromium decodes the MediaRecorder recording and an OfflineAudioContext mixes it down to 16 kHz mono. The result is encoded as PCM WAV in memory, so ffmpeg isn't needed.
  • Runtime: whisper-cli is found on PATH and in Homebrew's bin folder; the latter matters because an app opened from the Dock doesn't see Homebrew's PATH. It runs from an argv array (no shell) as -m <model> -f <wav> -nt -np, adding -l auto for multilingual models.
    • The WAV goes into a private mkdtemp folder that is removed as soon as whisper answers. The base64 buffer is zeroed.
    • Timestamps and [BLANK_AUDIO]-style markers are stripped from the output, as the CLI does (cleanTranscript).
    • Calls time out after 120 s, and the process is killed on dispose.
  • Only the model path is added to dictation.json. Cloud behaviour is unchanged.

Spec cases (MUS-SPEC-01, Dictation)

  • 840L: Local transcribes with whisper.cpp; no network; temp audio deleted; no audio in the data folder or logs.
  • 840M: names what is missing; finds the CLI's and Homebrew's models; refuses a non-model path; refuses non-WAV audio without running anything.
  • 840N (DOM): with Local, the composer sends 16 kHz mono 16-bit WAV and inserts the words.
  • 840O (DOM): Settings › Chat lists the found models and saves the chosen path.

Red, at b01be6f (tests and cases only)

  • node --experimental-transform-types --test --test-concurrency=1 tests/dictation-840.test.ts: exit 1, 3 passed and 2 failed. 840L failed on "A local speech model is not in this version yet"; 840M failed on "Local can be chosen".
  • composer-dictation-840-dom (built with esbuild as scripts/test-renderer.mjs does): 840A to 840E passed, then the suite stopped at not ok - with the local backend the composer sends a 16 kHz mono WAV recording…, because the mime was audio/webm. The suite stops at its first failure, so 840O's red wasn't reached.

Green, at 32f681e

  • dictation-840.test.ts: exit 0, 5 passed, 0 failed.
  • composer-dictation-840-dom: all 9 checks ok.
  • microphone-840: 2 passed.
  • resource-errors and device-noun: 4 passed each.
  • check-spec: OK.
  • check-test-coverage: no orphan files.
  • npm run typecheck and npx tsc --noEmit: clean.
  • One harness-only change in dictation-840.test.ts: call() now turns a synchronous throw from a handler into a rejection, so assert.rejects can see it. No assertion changed.

Evidence: real app, isolated instance (own --user-data-dir, --use-mock-keychain, captured over CDP, then quit)

  • Settings › Chat with Local model chosen shows the found models, the model path field and the program in use.
  • Real end to end: Homebrew's whisper-cli with ggml-tiny.en.bin, on a WAV made with say. dictation.transcribe in the app returned "Please add a unit test for the passer." in about 9 s. "passer" for "parser" comes from the tiny model. No muster-dictation-* temp folder was left behind.
  • Screenshots: https://gist.github.com/Dkm0315/d1c9f1568e667162f5f88f70100c8d4d (local copies in ~/.muster-work/review/840/)

Not verified

  • Recording from a real microphone in the app, including Chromium's decodeAudioData on a real MediaRecorder webm. The DOM suite fakes Web Audio. This needs a check by hand: Local, click the mic, speak, click again.
  • Windows and Linux paths: the program is looked up on PATH there too, but this wasn't run.
  • On-device macOS speech is still listed as "not in this version yet". It needs a native helper with its own speech usage string and authorization, so I left it for a follow-up PR.

Part of #840

MUS-SPEC-01-840L to 840O: the Local backend runs whisper.cpp with the chosen model on a WAV recording and keeps nothing; it says what is missing and finds models in ~/.muster/models (shared with the muster CLI) and Homebrew; the composer sends 16 kHz mono WAV for Local; Settings › Chat picks the model.
- Runtime: the local backend finds whisper-cli on PATH and Homebrew, models in ~/.muster/models (shared with the muster CLI), ~/.cache/whisper* and Homebrew's whisper-cpp folder; runs it with argv, no shell, on a WAV in a private temp folder removed as soon as it answers; cleans timestamps and markers as the CLI does.
- Renderer: with Local, the recording is decoded and resampled to 16 kHz mono WAV in memory before it is sent.
- Settings › Chat: Local model row with the found models, a model path field and the setup hint.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant