Repository navigation
Conversation
MUS-SPEC-01-840L to 840O: the Local backend runs whisper.cpp with the chosen model on a WAV recording and keeps nothing; it says what is missing and finds models in ~/.muster/models (shared with the muster CLI) and Homebrew; the composer sends 16 kHz mono WAV for Local; Settings › Chat picks the model.
- Runtime: the local backend finds whisper-cli on PATH and Homebrew, models in ~/.muster/models (shared with the muster CLI), ~/.cache/whisper* and Homebrew's whisper-cpp folder; runs it with argv, no shell, on a WAV in a private temp folder removed as soon as it answers; cleans timestamps and markers as the CLI does. - Renderer: with Local, the recording is decoded and resampled to 16 kHz mono WAV in memory before it is sent. - Settings › Chat: Local model row with the found models, a model path field and the setup hint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dictation with a local model: whisper.cpp transcribes the composer recording on this computer.
What the user can now do
whisper-cliinstalled (brew install whisper-cpp) and a ggml model, the mic and the hold-to-talk shortcut work as they do with Cloud, but the audio never leaves the computer.~/.muster/models,~/.cache/whisperand~/.cache/whisper.cpp. It also checks Homebrew'sshare/whisper-cpp/modelsfolder. A model downloaded for the CLI works here as is. You can also paste a path to any.binmodel file.How it works (same approach as the CLI's
voice.tswhisper-cpp provider, ported because the repos share no package)whisper-cliis found on PATH and in Homebrew's bin folder; the latter matters because an app opened from the Dock doesn't see Homebrew's PATH. It runs from an argv array (no shell) as-m <model> -f <wav> -nt -np, adding-l autofor multilingual models.mkdtempfolder that is removed as soon as whisper answers. The base64 buffer is zeroed.[BLANK_AUDIO]-style markers are stripped from the output, as the CLI does (cleanTranscript).dictation.json. Cloud behaviour is unchanged.Spec cases (MUS-SPEC-01, Dictation)
Red, at b01be6f (tests and cases only)
node --experimental-transform-types --test --test-concurrency=1 tests/dictation-840.test.ts: exit 1, 3 passed and 2 failed. 840L failed on "A local speech model is not in this version yet"; 840M failed on "Local can be chosen".composer-dictation-840-dom(built with esbuild asscripts/test-renderer.mjsdoes): 840A to 840E passed, then the suite stopped atnot ok - with the local backend the composer sends a 16 kHz mono WAV recording…, because the mime wasaudio/webm. The suite stops at its first failure, so 840O's red wasn't reached.Green, at 32f681e
dictation-840.test.ts: exit 0, 5 passed, 0 failed.composer-dictation-840-dom: all 9 checks ok.microphone-840: 2 passed.resource-errorsanddevice-noun: 4 passed each.check-spec: OK.check-test-coverage: no orphan files.npm run typecheckandnpx tsc --noEmit: clean.dictation-840.test.ts:call()now turns a synchronous throw from a handler into a rejection, soassert.rejectscan see it. No assertion changed.Evidence: real app, isolated instance (own
--user-data-dir,--use-mock-keychain, captured over CDP, then quit)whisper-cliwithggml-tiny.en.bin, on a WAV made withsay.dictation.transcribein the app returned "Please add a unit test for the passer." in about 9 s. "passer" for "parser" comes from the tiny model. Nomuster-dictation-*temp folder was left behind.Not verified
decodeAudioDataon a real MediaRecorder webm. The DOM suite fakes Web Audio. This needs a check by hand: Local, click the mic, speak, click again.Part of #840