diff --git a/docs/spec/clauses/01-chat-and-composer.md b/docs/spec/clauses/01-chat-and-composer.md index c201b998..260b75e9 100644 --- a/docs/spec/clauses/01-chat-and-composer.md +++ b/docs/spec/clauses/01-chat-and-composer.md @@ -48,6 +48,10 @@ Boundaries: | [MUS-SPEC-01-840I](01-chat-and-composer.md#mus-spec-01-840i) | Cloud dictation through a saved OpenAI-compatible connection uses that connection's endpoint and stored key. | [`the cloud backend can use a saved provider connection and its key`](../../../packages/agent-app/tests/dictation-840.test.ts) | | [MUS-SPEC-01-840J](01-chat-and-composer.md#mus-spec-01-840j) | On macOS, microphone access asks the system once when undecided and returns a System Settings message when denied, without asking. | [`microphone access asks macOS when undecided and explains a denial`](../../../packages/agent-app/tests/microphone-840.test.ts) | | [MUS-SPEC-01-840K](01-chat-and-composer.md#mus-spec-01-840k) | The window allows a media request only for audio from the app's own page; video, other permissions and other pages are refused. | [`only the app page may open the microphone, and only for audio`](../../../packages/agent-app/tests/microphone-840.test.ts) | +| [MUS-SPEC-01-840L](01-chat-and-composer.md#mus-spec-01-840l) | With Local chosen, the WAV recording goes to the whisper.cpp program with the chosen model; its timestamps and silence markers are removed from the words; no network request is made; the temporary audio is gone afterwards and no audio is in the data folder or the logs. | [`the local backend transcribes the recording with whisper.cpp on this computer and keeps nothing`](../../../packages/agent-app/tests/dictation-840.test.ts) | +| [MUS-SPEC-01-840M](01-chat-and-composer.md#mus-spec-01-840m) | Without whisper.cpp the view says how to install it; without a model it says where to put one; models in `~/.muster/models` and Homebrew's folder are listed; a model path that is not a model file is refused; a recording that is not WAV is refused without running anything. | [`the local backend says what is missing and finds the models the muster CLI uses`](../../../packages/agent-app/tests/dictation-840.test.ts) | +| [MUS-SPEC-01-840N](01-chat-and-composer.md#mus-spec-01-840n) | With Local chosen, the recording is decoded and sent to `dictation.transcribe` as `audio/wav`: a RIFF header for 16 kHz, one channel, 16-bit samples, then the samples; the words land in the composer. | [`with the local backend the composer sends a 16 kHz mono WAV recording and inserts the words`](../../../packages/agent-app/tests/composer-dictation-840-dom.tsx) | +| [MUS-SPEC-01-840O](01-chat-and-composer.md#mus-spec-01-840o) | Choosing Local in Settings › Chat shows the found models and the setup hint; picking a model calls `dictation.config.set` with its path. | [`Settings › Chat chooses the local model and shows what is missing`](../../../packages/agent-app/tests/composer-dictation-840-dom.tsx) | | [MUS-SPEC-01-401A](01-chat-and-composer.md#mus-spec-01-401a) | With the chat idle and the composer empty, the last turn's suggestion shows as ghost text in the composer (the textarea placeholder steps aside), and Tab puts exactly that text in the composer without sending. | [`the suggested follow-up shows as ghost text in an empty composer and Tab fills it`](../../../packages/agent-app/tests/follow-up-401-dom.tsx) | | [MUS-SPEC-01-401B](01-chat-and-composer.md#mus-spec-01-401b) | Clicking the suggestion fills the composer and sends nothing; a Cmd/Ctrl-click sends the suggestion as the next message. | [`clicking the suggestion fills the composer, and a modifier-click sends it`](../../../packages/agent-app/tests/follow-up-401-dom.tsx) | | [MUS-SPEC-01-401C](01-chat-and-composer.md#mus-spec-01-401c) | The suggestion is gone once anything is typed, and is not shown while a turn runs. | [`the suggestion hides once you type and while a turn runs`](../../../packages/agent-app/tests/follow-up-401-dom.tsx) | @@ -436,6 +440,50 @@ Boundaries: - Known gaps: None known. - Evidence: UNVERIFIED + +#### MUS-SPEC-01-840L: A local model transcribes on this computer + +- Precondition: Dictation is set to Local, whisper.cpp (`whisper-cli`) is installed and a ggml model file is chosen. +- Action: Transcribe a WAV recording. +- Expected result: The whisper.cpp program runs once with the chosen model and the recording (no shell), and the words come back without timestamps or markers such as `[BLANK_AUDIO]`. No network request is made. When the call ends the temporary copy of the recording is deleted, and no audio is left in the data folder or written to the logs. +- Test: `packages/agent-app/tests/dictation-840.test.ts::the local backend transcribes the recording with whisper.cpp on this computer and keeps nothing` +- Environment: Node test runner; the real dictation domain with a stand-in `whisper-cli` script on PATH that records its arguments and checks the WAV it is given; fake fetch; no microphone. +- Known gaps: A real whisper.cpp build and model are not run in CI; accuracy and speed depend on the model the person chooses. +- Evidence: UNVERIFIED + + +#### MUS-SPEC-01-840M: The local backend says what is missing + +- Precondition: Dictation is set to Local. First no whisper.cpp program and no model; then whisper.cpp and model files in `~/.muster/models` (the folder the muster CLI uses) and in Homebrew's whisper-cpp folder. +- Action: Read the dictation settings, choose a model, try a path that is not a model, and transcribe a recording that is not WAV. +- Expected result: Without whisper.cpp the settings are not ready and say to install it (`brew install whisper-cpp`). With it but no model, they say where to put a model. Found models are listed, and choosing one makes the backend ready. A path that is not an existing `.bin` file is refused. A recording that is not WAV is refused with a clear message and nothing is run. +- Test: `packages/agent-app/tests/dictation-840.test.ts::the local backend says what is missing and finds the models the muster CLI uses` +- Environment: Node test runner; the real dictation domain with a temp home folder, a temp PATH and a stand-in `whisper-cli`. +- Known gaps: Muster does not download models; the person gets one themselves (the hint says from where). +- Evidence: UNVERIFIED + + +#### MUS-SPEC-01-840N: With Local the composer sends WAV + +- Precondition: Dictation is set to Local and ready. The microphone is allowed. +- Action: Click the mic, then click it again. +- Expected result: The recording is decoded and resampled to 16 kHz mono and sent to `dictation.transcribe` as `audio/wav` with a valid RIFF header (16,000 Hz, 1 channel, 16-bit) followed by the samples. The transcript fills the composer. +- Test: `packages/agent-app/tests/composer-dictation-840-dom.tsx::with the local backend the composer sends a 16 kHz mono WAV recording and inserts the words` +- Environment: Real components in a DOM suite (linkedom); fake MediaRecorder, getUserMedia and Web Audio decoding; fake app bridge; no microphone. +- Known gaps: Chromium's real decoding of a MediaRecorder recording is not run here; it is checked by hand in the app. +- Evidence: UNVERIFIED + + +#### MUS-SPEC-01-840O: Settings › Chat sets up the local model + +- Precondition: Settings › Chat is open. The local backend reports two found models and that no model is chosen yet. +- Action: Choose Local, then pick a model. +- Expected result: Local is offered in the Dictation choice. Once chosen, a row lists the found models and shows the setup hint. Picking one calls `dictation.config.set` with that model's path. +- Test: `packages/agent-app/tests/composer-dictation-840-dom.tsx::Settings › Chat chooses the local model and shows what is missing` +- Environment: Real components in a DOM suite (linkedom); fake app bridge. +- Known gaps: None known. +- Evidence: UNVERIFIED + ### Suggested follow-up and model markers (#401) diff --git a/packages/agent-app/CHANGELOG.md b/packages/agent-app/CHANGELOG.md index a6b0f7f3..03e90db7 100644 --- a/packages/agent-app/CHANGELOG.md +++ b/packages/agent-app/CHANGELOG.md @@ -26,6 +26,7 @@ All notable changes to Muster Agent. Each `## ` section becomes the not - A suggested follow-up after a turn, as ghost text in the empty message field: Tab or click puts it there, Cmd/Ctrl-click sends it. Off by default (Settings › Chat › Suggest a follow-up) since it is one small extra request per turn: Claude Code chats use Claude Code's own suggestion, API and gateway chats ask the same model once; Codex chats offer none. The transcript also marks where the model or effort changed between turns: "Switched to GPT-5.5 · high effort" (#401). - Commits in the History graph have a menu (right-click, or Shift+F10 on a row): copy the SHA or message, check the commit out (detached), create a branch there, cherry-pick it onto the current branch or revert it. A picker shows another branch’s commits to pick from. Each change confirms first, saying what will happen; it is refused while tracked files have uncommitted changes, warns when a chat is running in the folder, and a conflicting pick or revert opens the conflict view. History and Changes refresh afterwards (#409). - Automations can run once at a set date and time: pick **Once at a date and time** under Repeat. The time is in the automation's time zone and is shown with it. After it runs the automation stays in the list as Done with its run linked; a time that has already passed is refused when you save. If Muster was closed or asleep at that time, it runs once at the next start and its history says it ran late (or records it as missed when catch-up is off). Before it runs you can move it to another chat with Edit (#845). +- **Dictation with a local model.** Settings › Chat › Dictation now offers Local model: whisper.cpp (`brew install whisper-cpp`) transcribes your recording on this computer, so the audio never leaves it. Muster finds models in `~/.muster/models` (the same folder the muster CLI uses) and in Homebrew's whisper-cpp folder, or you can give it a model file path. ## 0.3.18 diff --git a/packages/agent-app/src/renderer/components/PreferencesScreen.tsx b/packages/agent-app/src/renderer/components/PreferencesScreen.tsx index 44c6c554..0c83f7d5 100644 --- a/packages/agent-app/src/renderer/components/PreferencesScreen.tsx +++ b/packages/agent-app/src/renderer/components/PreferencesScreen.tsx @@ -280,6 +280,7 @@ function DictationSection():React.ReactElement { const [endpoint,setEndpoint]=useState(''); const [model,setModel]=useState(''); const [key,setKey]=useState(''); + const [modelPath,setModelPath]=useState(''); const [busy,setBusy]=useState(false); const providers=useStoreSelector(state=>state.providers); const connections=(('value' in providers?providers.value:undefined)??[]).filter(provider=>provider.custom&&provider.endpoint); @@ -312,6 +313,18 @@ function DictationSection():React.ReactElement { } + {config.backend==='local'&& + + + setModelPath(event.target.value)}/> + + + + + } ; } diff --git a/packages/agent-app/src/renderer/dictation.ts b/packages/agent-app/src/renderer/dictation.ts index 8bc83773..542628df 100644 --- a/packages/agent-app/src/renderer/dictation.ts +++ b/packages/agent-app/src/renderer/dictation.ts @@ -9,6 +9,7 @@ */ import { useEffect, useRef, useState } from 'react'; import { DICTATION_HOLD_MS, DICTATION_MAX_SECONDS, MICROPHONE_SETTINGS_HINT } from '../shared/domains/dictation-protocol'; +import { recordingAsWav } from './wavRecording'; import { invoke } from './bridge'; import { runtimeMessage as errorText } from './composerBridge'; import { openAppSettings, pushNotice } from './store'; @@ -50,7 +51,7 @@ function base64(bytes: Uint8Array): string { const pickMime = (Class: RecorderClass) => MIME_PREFERENCE.find(type => { try { return Class.isTypeSupported?.(type) ?? false; } catch { return false; } }); const release = (stream: MediaStream | undefined) => { for (const track of stream?.getTracks() ?? []) { try { track.stop(); } catch { /* already stopped */ } } }; -interface Session { stream: MediaStream; recorder: Recorder; chunks: Blob[]; mime: string; cancelled: boolean; timer: ReturnType } +interface Session { stream: MediaStream; recorder: Recorder; chunks: Blob[]; mime: string; cancelled: boolean; timer: ReturnType; /** The backend reads WAV only (local). */ wav: boolean } export interface DictationOptions { /** Final words to insert at the caret. */ @@ -91,14 +92,16 @@ export function useDictation(options: DictationOptions) { }; const transcribe = async (current: Session) => { - const blob = new Blob(current.chunks, { type: current.mime }); + const recorded = new Blob(current.chunks, { type: current.mime }); current.chunks.length = 0; - if (!blob.size) { latest.current.flash('No audio was recorded'); move('idle'); return; } + if (!recorded.size) { latest.current.flash('No audio was recorded'); move('idle'); return; } move('transcribing'); const mine = generation.current; try { + const blob = current.wav ? await recordingAsWav(recorded) : recorded; + if (mine !== generation.current) return; const audio = base64(new Uint8Array(await blob.arrayBuffer())); - const result = await invoke('dictation.transcribe', { audio, mime: current.mime }); + const result = await invoke('dictation.transcribe', { audio, mime: current.wav ? 'audio/wav' : current.mime }); if (mine !== generation.current) return; const words = result?.text?.trim() ?? ''; if (words) latest.current.onText(words); else latest.current.flash('No speech heard'); @@ -128,7 +131,7 @@ export function useDictation(options: DictationOptions) { if (mine !== generation.current) { release(stream); return; } const Class = recorderClass()!, wanted = pickMime(Class); const recorder = new Class(stream, wanted ? { mimeType: wanted } : undefined); - const current: Session = { stream, recorder, chunks: [], mime: recorder.mimeType || wanted || 'audio/webm', cancelled: false, + const current: Session = { stream, recorder, chunks: [], mime: recorder.mimeType || wanted || 'audio/webm', cancelled: false, wav: config.recording === 'wav', timer: setTimeout(() => { if (session.current === current) finish(); }, DICTATION_MAX_SECONDS * 1000) }; recorder.ondataavailable = event => { if (event.data && event.data.size) current.chunks.push(event.data); }; recorder.onerror = () => { if (session.current === current) cancel(); latest.current.flash('Recording stopped unexpectedly'); }; diff --git a/packages/agent-app/src/renderer/wavRecording.ts b/packages/agent-app/src/renderer/wavRecording.ts new file mode 100644 index 00000000..1ce20cb1 --- /dev/null +++ b/packages/agent-app/src/renderer/wavRecording.ts @@ -0,0 +1,43 @@ +/** + * #840: the local dictation backend (whisper.cpp) reads 16 kHz mono 16-bit WAV, and MediaRecorder makes webm/opus. + * Chromium decodes the recording, an OfflineAudioContext mixes it to one channel at 16 kHz, and the samples are + * written as PCM WAV. Everything stays in memory. + */ +import { DICTATION_WAV_RATE } from '../shared/domains/dictation-protocol'; + +type AudioScope = { AudioContext?: typeof AudioContext; webkitAudioContext?: typeof AudioContext; OfflineAudioContext?: typeof OfflineAudioContext }; + +/** PCM 16-bit mono WAV of the given samples (-1..1). */ +export function encodeWav(samples: Float32Array, rate = DICTATION_WAV_RATE): Uint8Array { + const out = new Uint8Array(44 + samples.length * 2), view = new DataView(out.buffer); + const text = (at: number, value: string) => { for (let i = 0; i < value.length; i++) out[at + i] = value.charCodeAt(i); }; + text(0, 'RIFF'); view.setUint32(4, 36 + samples.length * 2, true); text(8, 'WAVE'); text(12, 'fmt '); + view.setUint32(16, 16, true); view.setUint16(20, 1, true); view.setUint16(22, 1, true); view.setUint32(24, rate, true); + view.setUint32(28, rate * 2, true); view.setUint16(32, 2, true); view.setUint16(34, 16, true); + text(36, 'data'); view.setUint32(40, samples.length * 2, true); + for (let i = 0; i < samples.length; i++) { + const sample = Math.max(-1, Math.min(1, samples[i]!)); + view.setInt16(44 + i * 2, sample < 0 ? Math.round(sample * 0x8000) : Math.round(sample * 0x7fff), true); + } + return out; +} + +/** The recording as 16 kHz mono WAV. Throws a plain message when this window cannot decode it. */ +export async function recordingAsWav(blob: Blob): Promise { + const scope = globalThis as unknown as AudioScope; + const Context = scope.AudioContext ?? scope.webkitAudioContext, Offline = scope.OfflineAudioContext; + if (!Context || !Offline) throw new Error('This window cannot convert the recording for the local model.'); + const context = new Context(); + let decoded: AudioBuffer; + try { decoded = await context.decodeAudioData(await blob.arrayBuffer()); } + catch { throw new Error('The recording could not be read. Try again.'); } + finally { void context.close().catch(() => undefined); } + const frames = Math.max(1, Math.ceil(decoded.duration * DICTATION_WAV_RATE)); + const offline = new Offline(1, frames, DICTATION_WAV_RATE); + const source = offline.createBufferSource(); + source.buffer = decoded; + source.connect(offline.destination); + source.start(); + const rendered = await offline.startRendering(); + return new Blob([encodeWav(rendered.getChannelData(0))], { type: 'audio/wav' }); +} diff --git a/packages/agent-app/src/runtime/domains/dictation.ts b/packages/agent-app/src/runtime/domains/dictation.ts index 9e1be3cf..0ff48967 100644 --- a/packages/agent-app/src/runtime/domains/dictation.ts +++ b/packages/agent-app/src/runtime/domains/dictation.ts @@ -4,14 +4,20 @@ * - `cloud`: an OpenAI-compatible `POST /audio/transcriptions` with the person's own key, either a saved * provider connection (its endpoint and key) or an endpoint and key entered for dictation (the key is encrypted in * the secret store). This is the only backend that sends audio anywhere. - * - `macos` (on-device speech) and `local` (a local model) are listed so the choice is visible, but this build does - * not ship them: they are reported unavailable and never transcribe. + * - `local`: whisper.cpp (`whisper-cli`) with a ggml model file, on this computer. The composer sends 16 kHz mono WAV, + * which is written to a private temp folder for the program to read and deleted as soon as it answers. Models are + * found where the muster CLI keeps them (~/.muster/models, ~/.cache/whisper*) and in Homebrew's whisper-cpp folder. + * - `macos` (on-device speech) is listed so the choice is visible, but this build does not ship it: it is reported + * unavailable and never transcribes. * - * The recording arrives as base64, is posted once and dropped: it is never written to disk or logged, and errors - * never quote it. Only the choice (backend, connection, endpoint, model) is stored, in dataDir/dictation.json. + * The recording arrives as base64 and is dropped after one use: it is never kept on disk or logged, and errors never + * quote it. Only the choice (backend, connection, endpoint, model, local model) is stored, in dataDir/dictation.json. */ -import { chmodSync, mkdirSync, readFileSync, renameSync, writeFileSync } from 'node:fs'; -import { join } from 'node:path'; +import { execFile, type ChildProcess } from 'node:child_process'; +import { accessSync, chmodSync, constants, mkdirSync, readdirSync, readFileSync, renameSync, statSync, writeFileSync } from 'node:fs'; +import { mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { delimiter, isAbsolute, join } from 'node:path'; import { DICTATION_BACKENDS, DICTATION_MAX_AUDIO_BYTES, type DictationBackendId, type DictationBackendView, type DictationConfigInput, type DictationConfigView, type MicrophoneAccess, @@ -25,13 +31,28 @@ const DEFAULT_MODEL = 'whisper-1'; const CHOOSE = 'Choose a dictation service in Settings › Chat.'; const LABELS: Record = { off: 'Off', cloud: 'Cloud', macos: 'On-device (macOS)', local: 'Local model' }; const TIMEOUT_MS = 120_000; +const WHISPER_PROGRAMS = ['whisper-cli', 'whisper-cpp']; +const INSTALL_WHISPER = 'Install whisper.cpp to use a local model (brew install whisper-cpp), then try again.'; +const GET_MODEL = 'Choose a whisper model in Settings › Chat, or put one in ~/.muster/models (for example ggml-base.en.bin from huggingface.co/ggerganov/whisper.cpp).'; +const MODEL_FILE = /^ggml-[A-Za-z0-9._-]+\.bin$/; + +/** whisper.cpp output -> plain words: no timestamps, no [BLANK_AUDIO]-style markers (as the muster CLI does). */ +export function cleanTranscript(raw: string): string { + return raw.split('\n') + .map(line => line + .replace(/^\s*\[\d{2}:\d{2}(?::\d{2})?[.,]\d{3}\s*-->\s*\d{2}:\d{2}(?::\d{2})?[.,]\d{3}\]\s*/, '') + .replace(/\[(?:BLANK_AUDIO|MUSIC|NOISE|SILENCE|INAUDIBLE)\]|\((?:silence|music|noise)\)/gi, '') + .trim()) + .filter(Boolean).join(' ').replace(/\s+/g, ' ').trim(); +} +const isWav = (audio: Buffer) => audio.byteLength > 44 && audio.toString('latin1', 0, 4) === 'RIFF' && audio.toString('latin1', 8, 12) === 'WAVE'; const MAX_ANSWER_BYTES = 1024 * 1024; const MIME = /^audio\/[a-z0-9.+-]{1,40}(?:;\s*codecs="?[a-z0-9.,-]{1,40}"?)?$/i; const MODEL = /^[A-Za-z0-9._:/-]{1,128}$/; const CONNECTION = /^[A-Za-z0-9_-]{1,128}$/; -interface Stored { version: 1; backend: DictationBackendId; connectionId: string | null; endpoint: string; model: string } -const DEFAULTS: Stored = { version: 1, backend: 'off', connectionId: null, endpoint: '', model: DEFAULT_MODEL }; +interface Stored { version: 1; backend: DictationBackendId; connectionId: string | null; endpoint: string; model: string; localModel: string } +const DEFAULTS: Stored = { version: 1, backend: 'off', connectionId: null, endpoint: '', model: DEFAULT_MODEL, localModel: '' }; export interface DictationDomainOptions { fetch?: typeof fetch; @@ -63,6 +84,31 @@ export function createDictationDomain(context: Pick, o const env = options.env ?? process.env; const platform = options.platform ?? process.platform; const inflight = new Set(); + const children = new Set(); + const home = env.HOME || env.USERPROFILE || ''; + /** Homebrew's prefix: HOMEBREW_PREFIX when set, else both usual ones. The app's PATH misses them when opened from the Dock. */ + const brewPrefixes = () => env.HOMEBREW_PREFIX ? [env.HOMEBREW_PREFIX] : ['/opt/homebrew', '/usr/local']; + const whisperProgram = (): string | null => { + const dirs = [...(env.PATH ?? '').split(delimiter).filter(Boolean), ...brewPrefixes().map(prefix => join(prefix, 'bin'))]; + for (const name of WHISPER_PROGRAMS) for (const dir of dirs) { + const candidate = join(dir, name); + try { accessSync(candidate, constants.X_OK); if (statSync(candidate).isFile()) return candidate; } catch { /* keep looking */ } + } + return null; + }; + /** ggml model files where the muster CLI and Homebrew keep them, in that order. */ + const localModels = (): string[] => { + const dirs = home ? [join(home, '.muster', 'models'), join(home, '.cache', 'whisper'), join(home, '.cache', 'whisper.cpp')] : []; + dirs.push(...brewPrefixes().map(prefix => join(prefix, 'share', 'whisper-cpp', 'models'))); + const found: string[] = []; + for (const dir of dirs) { + let names: string[]; + try { names = readdirSync(dir).filter(name => MODEL_FILE.test(name)).sort(); } catch { continue; } + for (const name of names) { const path = join(dir, name); try { if (statSync(path).isFile() && !found.includes(path)) found.push(path); } catch { /* gone */ } } + } + return found; + }; + const modelFile = (path: string) => { try { return isAbsolute(path) && /\.bin$/i.test(path) && statSync(path).isFile(); } catch { return false; } }; const read = (): Stored => { try { @@ -73,6 +119,7 @@ export function createDictationDomain(context: Pick, o connectionId: typeof raw.connectionId === 'string' && CONNECTION.test(raw.connectionId) ? raw.connectionId : null, endpoint: typeof raw.endpoint === 'string' ? raw.endpoint : '', model: typeof raw.model === 'string' && MODEL.test(raw.model) ? raw.model : DEFAULT_MODEL, + localModel: typeof raw.localModel === 'string' ? raw.localModel : '', }; } catch { return { ...DEFAULTS }; } }; @@ -84,10 +131,18 @@ export function createDictationDomain(context: Pick, o try { chmodSync(file, 0o600); } catch { /* filesystems without modes */ } }; const backends = (): DictationBackendView[] => DICTATION_BACKENDS.map(id => { - if (id === 'off' || id === 'cloud') return { id, label: LABELS[id], available: true }; if (id === 'macos') return { id, label: LABELS[id], available: false, reason: platform === 'darwin' ? 'On-device macOS speech is not in this version yet.' : 'On-device speech is available on macOS only.' }; - return { id, label: LABELS[id], available: false, reason: 'A local speech model is not in this version yet.' }; + return { id, label: LABELS[id], available: true }; }); + /** The program and model a local transcription uses; a reason instead when one is missing. */ + const localTarget = (stored: Stored): { program: string; model: string } | { reason: string } => { + const program = whisperProgram(); + if (!program) return { reason: INSTALL_WHISPER }; + const model = stored.localModel || localModels()[0] || ''; + if (!model) return { reason: GET_MODEL }; + if (!modelFile(model)) return { reason: `The whisper model ${model} is missing. ${GET_MODEL}` }; + return { program, model }; + }; /** Where the cloud request goes and with which key; a reason instead when it cannot go anywhere. */ const target = (stored: Stored): { endpoint: string; key?: string } | { reason: string } => { if (stored.connectionId) { @@ -104,12 +159,14 @@ export function createDictationDomain(context: Pick, o const view = (stored = read()): DictationConfigView => { const store = secrets(); const cloud = { connectionId: stored.connectionId, endpoint: stored.endpoint, model: stored.model, keyStored: Boolean(store?.status(DICTATION_SECRET_ID).stored), secureStorage: Boolean(store?.secureStorage()) }; + const program = whisperProgram(), models = localModels(); + const local = { program, model: stored.localModel || models[0] || '', models }; const option = backends().find(entry => entry.id === stored.backend)!; let reason: string | undefined; if (stored.backend === 'off') reason = CHOOSE; else if (!option.available) reason = `${option.reason} ${CHOOSE}`; - else { const where = target(stored); if ('reason' in where) reason = where.reason; } - return { backend: stored.backend, ready: !reason, ...(reason ? { reason } : {}), backends: backends(), cloud }; + else { const where = stored.backend === 'local' ? localTarget(stored) : target(stored); if ('reason' in where) reason = where.reason; } + return { backend: stored.backend, ready: !reason, ...(reason ? { reason } : {}), backends: backends(), cloud, local, ...(stored.backend === 'local' ? { recording: 'wav' as const } : {}) }; }; const set = (raw: Record): DictationConfigView => { const input = raw as DictationConfigInput; @@ -132,6 +189,11 @@ export function createDictationDomain(context: Pick, o if (!MODEL.test(model)) throw new Error('Enter a transcription model name, such as whisper-1.'); next.model = model; } + if (input.localModel !== undefined) { + const path = typeof input.localModel === 'string' ? input.localModel.trim() : ''; + if (path && !modelFile(path)) throw new Error('Choose a whisper model file (a ggml .bin file) that exists on this computer.'); + next.localModel = path; + } if (input.apiKey !== undefined) { const store = secrets(); if (input.apiKey === null || input.apiKey === '') store?.clear(DICTATION_SECRET_ID); @@ -140,17 +202,49 @@ export function createDictationDomain(context: Pick, o write(next); return view(next); }; + /** whisper.cpp reads the WAV from a private temp folder that is removed as soon as it answers. */ + const transcribeLocally = async (stored: Stored, raw: Record): Promise<{ text: string }> => { + const where = localTarget(stored); + if ('reason' in where) throw new Error(where.reason); + const audio = recording(raw); + if (!isWav(audio)) { audio.fill(0); throw new Error('The local model needs the recording as WAV. Update Muster, then try again.'); } + const dir = await mkdtemp(join(tmpdir(), 'muster-dictation-')); + try { + const file = join(dir, 'dictation.wav'); + await writeFile(file, audio, { mode: 0o600 }); + audio.fill(0); + // A multilingual model listens for any language; an English-only (.en) model is English already. + const args = ['-m', where.model, '-f', file, '-nt', '-np', ...(/\.en\.bin$/i.test(where.model) ? [] : ['-l', 'auto'])]; + const output = await new Promise((resolve, reject) => { + const child = execFile(where.program, args, { timeout: TIMEOUT_MS, maxBuffer: MAX_ANSWER_BYTES, windowsHide: true }, (error, stdout, stderr) => { + children.delete(child); + if (!error) { resolve(String(stdout)); return; } + const failure = error as NodeJS.ErrnoException & { killed?: boolean }; + const detail = String(stderr).trim().split('\n').slice(-2).join(' ').slice(0, 300); + reject(new Error(failure.killed ? 'The local model took too long to transcribe. Try a shorter recording or a smaller model.' : `The local model could not transcribe the recording${detail ? `: ${detail}` : '.'}`)); + }); + children.add(child); + }); + return { text: cleanTranscript(output) }; + } finally { await rm(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 200 }); } + }; + /** The recording from the composer, checked for size. */ + const recording = (raw: Record): Buffer => { + if (typeof raw.audio !== 'string' || !raw.audio || raw.audio.length > Math.ceil(DICTATION_MAX_AUDIO_BYTES / 3) * 4 + 4) throw new Error('The recording is empty or too long to transcribe.'); + const audio = Buffer.from(raw.audio, 'base64'); + if (!audio.byteLength || audio.byteLength > DICTATION_MAX_AUDIO_BYTES) throw new Error('The recording is empty or too long to transcribe.'); + return audio; + }; const transcribe = async (raw: Record): Promise<{ text: string }> => { const stored = read(), current = view(stored); + if (stored.backend === 'local' && current.ready) return transcribeLocally(stored, raw); // Nothing is sent anywhere unless the chosen backend is a cloud one that is ready. if (stored.backend !== 'cloud' || !current.ready) throw new Error(current.reason ?? CHOOSE); const where = target(stored); if ('reason' in where) throw new Error(where.reason); const mime = typeof raw.mime === 'string' && MIME.test(raw.mime) ? raw.mime : ''; if (!mime) throw new Error('The recording is not in an audio format Muster can send.'); - if (typeof raw.audio !== 'string' || !raw.audio || raw.audio.length > Math.ceil(DICTATION_MAX_AUDIO_BYTES / 3) * 4 + 4) throw new Error('The recording is empty or too long to transcribe.'); - const audio = Buffer.from(raw.audio, 'base64'); - if (!audio.byteLength || audio.byteLength > DICTATION_MAX_AUDIO_BYTES) throw new Error('The recording is empty or too long to transcribe.'); + const audio = recording(raw); const form = new FormData(); form.append('file', new Blob([audio], { type: mime.split(';')[0]! }), `dictation.${extension(mime)}`); form.append('model', stored.model); @@ -178,7 +272,7 @@ export function createDictationDomain(context: Pick, o } finally { clearTimeout(timer); inflight.delete(controller); } }; return { - dispose: () => { for (const controller of inflight) controller.abort(); inflight.clear(); }, + dispose: () => { for (const controller of inflight) controller.abort(); inflight.clear(); for (const child of children) child.kill(); children.clear(); }, handlers: { 'dictation.config.get': () => view(), 'dictation.config.set': input => set(input), diff --git a/packages/agent-app/src/shared/domains/dictation-protocol.ts b/packages/agent-app/src/shared/domains/dictation-protocol.ts index 52ec1784..7e2c1bf5 100644 --- a/packages/agent-app/src/shared/domains/dictation-protocol.ts +++ b/packages/agent-app/src/shared/domains/dictation-protocol.ts @@ -4,8 +4,8 @@ * and nothing is kept after transcription (no file, no log line). */ /** `off`: no backend (a browser host may still use its own Web Speech). `cloud`: an OpenAI-compatible - * `/audio/transcriptions` endpoint with the person's own key. `macos` (on-device speech) and `local` (a local model) - * are listed for the choice but not available in this build. */ + * `/audio/transcriptions` endpoint with the person's own key. `local`: whisper.cpp with a model file on this computer. + * `macos` (on-device speech) is listed for the choice but not available in this build. */ export type DictationBackendId = 'off' | 'cloud' | 'macos' | 'local'; export const DICTATION_BACKENDS: readonly DictationBackendId[] = ['off', 'cloud', 'macos', 'local']; export interface DictationBackendView { id: DictationBackendId; label: string; available: boolean; reason?: string } @@ -18,6 +18,14 @@ export interface DictationCloudView { keyStored: boolean; secureStorage: boolean; } +export interface DictationLocalView { + /** The whisper.cpp program found (`whisper-cli`), or null when it is not installed. */ + program: string | null; + /** The chosen ggml model file; empty when none is chosen. */ + model: string; + /** Model files found where the muster CLI and Homebrew keep them. */ + models: string[]; +} export interface DictationConfigView { backend: DictationBackendId; /** The chosen backend can transcribe now. */ @@ -26,6 +34,9 @@ export interface DictationConfigView { reason?: string; backends: DictationBackendView[]; cloud: DictationCloudView; + local: DictationLocalView; + /** `wav`: the composer sends 16 kHz mono 16-bit WAV (the local backend reads nothing else); otherwise the recording as made. */ + recording?: 'wav'; } export interface DictationConfigInput { backend?: DictationBackendId; @@ -34,6 +45,8 @@ export interface DictationConfigInput { model?: string; /** A new key for the dictation endpoint; null removes the saved one. */ apiKey?: string | null; + /** The ggml model file for the local backend. */ + localModel?: string; } export type MicrophoneStatus = 'granted' | 'denied' | 'restricted' | 'not-determined' | 'unknown'; export interface MicrophoneAccess { granted: boolean; status: MicrophoneStatus; message?: string } @@ -45,6 +58,8 @@ export const DICTATION_MAX_AUDIO_BYTES = 24 * 1024 * 1024; export const DICTATION_MAX_SECONDS = 300; /** Holding the shortcut longer than this is push-to-talk (release transcribes); a shorter tap toggles recording on. */ export const DICTATION_HOLD_MS = 350; +/** The sample rate whisper.cpp reads. */ +export const DICTATION_WAV_RATE = 16_000; export const MICROPHONE_SETTINGS_HINT = 'Turn it on in System Settings › Privacy & Security › Microphone, then try again.'; export interface DictationCommands { diff --git a/packages/agent-app/tests/composer-dictation-840-dom.tsx b/packages/agent-app/tests/composer-dictation-840-dom.tsx index 13f3bd6b..b4b8362b 100644 --- a/packages/agent-app/tests/composer-dictation-840-dom.tsx +++ b/packages/agent-app/tests/composer-dictation-840-dom.tsx @@ -30,6 +30,13 @@ class FakeRecorder{ Object.defineProperty(globalThis,'navigator',{configurable:true,value:{userAgent:'Mozilla/5.0 Muster Electron/39.0.0',language:'en-US',platform:'MacIntel', mediaDevices:{async getUserMedia(constraints:any){assert.deepEqual(Object.keys(constraints),['audio'],'audio only');microphoneOpens++;const track={stopped:false,stop(){this.stopped=true;}};tracks.push(track);return {getTracks:()=>[track]};}}}}); Object.assign(globalThis,{MediaRecorder:FakeRecorder});(window as any).MediaRecorder=FakeRecorder; +// Fake Web Audio: decoding gives 10 ms of audio; rendering at 16 kHz mono gives 160 samples alternating +0.5 and -0.5. +const audioUse={decoded:0,rendered:[] as {channels:number;length:number;rate:number}[]}; +class FakeAudioContext{async decodeAudioData(data:ArrayBuffer){assert.ok(data.byteLength>0);audioUse.decoded++;return {duration:0.01,numberOfChannels:2,sampleRate:48000,length:480};}async close(){}} +class FakeOfflineAudioContext{destination={};constructor(public channels:number,public length:number,public rate:number){} + createBufferSource(){return {buffer:null,connect(){},start(){}};} + async startRendering(){audioUse.rendered.push({channels:this.channels,length:this.length,rate:this.rate});const samples=new Float32Array(this.length).map((_,i)=>i%2?-0.5:0.5);return {getChannelData:()=>samples,length:this.length,sampleRate:this.rate,numberOfChannels:1};}} +Object.assign(globalThis,{AudioContext:FakeAudioContext,OfflineAudioContext:FakeOfflineAudioContext});Object.assign(window,{AudioContext:FakeAudioContext,OfflineAudioContext:FakeOfflineAudioContext}); const twoHoursAgo=new Date(Date.now()-2*3600_000).toISOString(); let chat:any={id:'one',title:'one',folderId:'folder',status:'completed',updatedAt:twoHoursAgo,pinned:false,archived:false,draft:'',model:'fixture',mode:'agent',providerId:'hybrow'}; @@ -37,9 +44,10 @@ const listeners=new Set<(event:any)=>void>(); const snapshot=()=>JSON.parse(JSON.stringify({version:1,chats:[chat],folders:[{id:'folder',name:'Folder',path:'/fixture'}],projects:[],activeChatId:'one'})); const emit=()=>{for(const fn of listeners)fn({type:'snapshot',snapshot:snapshot()});}; const calls:{command:string;input:any}[]=[]; -const view=(backend:string,extra:Record={})=>({backend,ready:backend==='cloud',...(backend==='cloud'?{}:{reason:'Choose a dictation service in Settings › Chat.'}), - backends:[{id:'off',label:'Off',available:true},{id:'cloud',label:'Cloud',available:true},{id:'macos',label:'On-device',available:false,reason:'Coming in a later version.'}], - cloud:{connectionId:null,endpoint:'',model:'whisper-1',keyStored:false,secureStorage:true},...extra}); +const MODELS=['/Users/me/.muster/models/ggml-small.en.bin','/opt/homebrew/share/whisper-cpp/models/ggml-base.bin']; +const view=(backend:string,extra:Record={})=>({backend,ready:backend==='cloud',...(backend==='cloud'?{}:{reason:backend==='local'?'Choose a whisper model in Settings › Chat, or put one in ~/.muster/models.':'Choose a dictation service in Settings › Chat.'}), + backends:[{id:'off',label:'Off',available:true},{id:'cloud',label:'Cloud',available:true},{id:'macos',label:'On-device',available:false,reason:'Coming in a later version.'},{id:'local',label:'Local model',available:true}], + cloud:{connectionId:null,endpoint:'',model:'whisper-1',keyStored:false,secureStorage:true},local:{program:'/opt/homebrew/bin/whisper-cli',model:'',models:MODELS},...extra}); let config:any=view('cloud',{cloud:{connectionId:null,endpoint:'https://speech.example/v1',model:'whisper-1',keyStored:true,secureStorage:true}}); let microphone:any={granted:true,status:'granted'}; let transcript='ship the release notes';let transcribeGate:Promise=Promise.resolve(); @@ -60,7 +68,8 @@ window.muster={subscribe(fn:any){listeners.add(fn);return()=>{listeners.delete(f if(command==='dictation.config.get')return config; if(command==='dictation.config.set'){ const cloud={...config.cloud,...(input.endpoint!==undefined?{endpoint:input.endpoint}:{}),...(input.model!==undefined?{model:input.model}:{}),...(input.connectionId!==undefined?{connectionId:input.connectionId}:{}),...(input.apiKey?{keyStored:true}:{})}; - config=view(input.backend??config.backend,{cloud});return config; + const local={...config.local,...(input.localModel!==undefined?{model:input.localModel}:{})}; + config=view(input.backend??config.backend,{cloud,local,...(local.model&&(input.backend??config.backend)==='local'?{ready:true,reason:undefined,recording:'wav'}:{})});return config; } if(command==='dictation.microphone')return microphone; if(command==='dictation.transcribe'){await transcribeGate;return {text:transcript};} @@ -166,6 +175,31 @@ await check('dictation works in the steer composer while a chat runs',async()=>{ chat={...chat,status:'completed'};emit();await delay(40); }); +// Spec: MUS-SPEC-01-840N (docs/spec/clauses/01-chat-and-composer.md#mus-spec-01-840n) +await check('with the local backend the composer sends a 16 kHz mono WAV recording and inserts the words',async()=>{ + const cloud=config; + config=view('local',{ready:true,reason:undefined,recording:'wav',local:{program:'/opt/homebrew/bin/whisper-cli',model:MODELS[0],models:MODELS}}); + transcript='run the local tests'; + const before=count('dictation.transcribe'); + mic()!.click(); + await until(()=>mic()?.getAttribute('aria-pressed')==='true','recording'); + mic()!.click(); + await until(()=>count('dictation.transcribe')===before+1,'the recording is sent'); + const sent=calls.filter(call=>call.command==='dictation.transcribe').at(-1)!.input; + assert.equal(sent.mime,'audio/wav'); + assert.equal(audioUse.decoded,1,'the recording was decoded'); + assert.deepEqual(audioUse.rendered,[{channels:1,length:160,rate:16000}],'resampled to 16 kHz mono'); + const bytes=Buffer.from(sent.audio,'base64'); + assert.equal(bytes.toString('latin1',0,4),'RIFF');assert.equal(bytes.toString('latin1',8,16),'WAVEfmt '); + assert.equal(bytes.readUInt16LE(20),1,'PCM');assert.equal(bytes.readUInt16LE(22),1,'one channel'); + assert.equal(bytes.readUInt32LE(24),16000,'16 kHz');assert.equal(bytes.readUInt16LE(34),16,'16-bit'); + assert.equal(bytes.toString('latin1',36,40),'data');assert.equal(bytes.readUInt32LE(40),320);assert.equal(bytes.length,44+320); + assert.equal(bytes.readInt16LE(44),Math.round(0.5*32767));assert.equal(bytes.readInt16LE(46),-Math.round(0.5*32768)); + await until(()=>field().value==='run the local tests','the words fill the composer'); + config=cloud; +}); +await setDraft(''); + // Spec: MUS-SPEC-01-840F (docs/spec/clauses/01-chat-and-composer.md#mus-spec-01-840f) await check('Settings › Chat chooses the dictation service and saves a cloud key',async()=>{ config=view('off'); @@ -190,5 +224,19 @@ await check('Settings › Chat chooses the dictation service and saves a cloud k assert.match(document.body.textContent??'',/Key saved/); assert.equal((document.body.innerHTML).includes('sk-typed-key'),false,'the key is not shown back'); }); + +// Spec: MUS-SPEC-01-840O (docs/spec/clauses/01-chat-and-composer.md#mus-spec-01-840o) +await check('Settings › Chat chooses the local model and shows what is missing',async()=>{ + const segment=(label:string)=>[...document.querySelectorAll('[aria-label="Dictation"] button')].find(button=>button.textContent===label) as HTMLButtonElement|undefined; + await until(()=>segment('Local model'),'Local is offered'); + segment('Local model')!.click(); + await until(()=>$('[aria-label="Local whisper model"]'),'the local model row appears'); + assert.match(document.body.textContent??'',/~\/\.muster\/models/,'the setup hint shows'); + const picker=$('[aria-label="Local whisper model"]') as HTMLSelectElement; + assert.deepEqual([...picker.querySelectorAll('option')].map(option=>(option as HTMLOptionElement).value).filter(Boolean),MODELS,'the found models are listed'); + for(const option of picker.querySelectorAll('option'))option.toggleAttribute('selected',(option as HTMLOptionElement).value===MODELS[1]); + picker.dispatchEvent(new window.Event('change',{bubbles:true})); + await until(()=>calls.some(call=>call.command==='dictation.config.set'&&call.input.localModel===MODELS[1]),'picking a model saves its path'); +}); assert.deepEqual(errors,[]); process.exit(0); diff --git a/packages/agent-app/tests/dictation-840.test.ts b/packages/agent-app/tests/dictation-840.test.ts index f77ab63a..40335fd5 100644 --- a/packages/agent-app/tests/dictation-840.test.ts +++ b/packages/agent-app/tests/dictation-840.test.ts @@ -1,7 +1,8 @@ /** #840: dictation transcription backends in the runtime. Fake fetch and an in-memory secret box: no network, no microphone. */ import assert from 'node:assert/strict'; import {test, type TestContext} from 'node:test'; -import {mkdtemp, readdir, readFile, rm, stat} from 'node:fs/promises'; +import {chmod, mkdir, mkdtemp, readdir, readFile, rm, stat, writeFile} from 'node:fs/promises'; +import {existsSync} from 'node:fs'; import {tmpdir} from 'node:os'; import {join} from 'node:path'; import {SecretStore} from '../src/runtime/secret-store.ts'; @@ -16,13 +17,13 @@ function fakeFetch(answer: () => Response) { const fetch = (async (url: string | URL, init: RequestInit = {}) => { calls.push({url: String(url), init}); return answer(); }) as typeof globalThis.fetch; return {calls, fetch}; } -async function domain(t: TestContext, options: {fetch: typeof globalThis.fetch; connections?: () => unknown[]}) { +async function domain(t: TestContext, options: {fetch: typeof globalThis.fetch; connections?: () => unknown[]; env?: NodeJS.ProcessEnv}) { const dataDir = await directory(t); const secrets = new SecretStore(dataDir, () => box); t.after(() => secrets.close()); const {createDictationDomain} = await import('../src/runtime/domains/dictation.ts'); - const module = createDictationDomain({dataDir} as never, {fetch: options.fetch, secrets: () => secrets, connections: options.connections as never, platform: 'darwin'}); + const module = createDictationDomain({dataDir} as never, {fetch: options.fetch, secrets: () => secrets, connections: options.connections as never, platform: 'darwin', ...(options.env ? {env: options.env} : {})}); t.after(() => module.dispose?.()); - const call = (command: string, input: Record = {}) => Promise.resolve(module.handlers[command]!(input)) as Promise; + const call = (command: string, input: Record = {}) => Promise.resolve().then(() => module.handlers[command]!(input)) as Promise; return {dataDir, secrets, call}; } async function filesUnder(dir: string): Promise { @@ -103,3 +104,100 @@ test('the cloud backend can use a saved provider connection and its key', async assert.equal(net.calls[0]!.url, 'https://router.example/v1/audio/transcriptions'); assert.equal(new Headers(net.calls[0]!.init.headers).get('authorization'), 'Bearer sk-router-key'); }); + +/** A 16 kHz mono 16-bit WAV (what the composer sends for Local), with a recognisable sample pattern. */ +function wav(samples = 1600): Buffer { + const data = Buffer.alloc(samples * 2); + for (let i = 0; i < samples; i++) data.writeInt16LE(((i * 37) % 2000) - 1000, i * 2); + const head = Buffer.alloc(44); + head.write('RIFF', 0); head.writeUInt32LE(36 + data.length, 4); head.write('WAVE', 8); head.write('fmt ', 12); + head.writeUInt32LE(16, 16); head.writeUInt16LE(1, 20); head.writeUInt16LE(1, 22); head.writeUInt32LE(16000, 24); head.writeUInt32LE(32000, 28); head.writeUInt16LE(2, 32); head.writeUInt16LE(16, 34); + head.write('data', 36); head.writeUInt32LE(data.length, 40); + return Buffer.concat([head, data]); +} +/** A temp home, a Homebrew prefix and a bin folder; `whisper()` puts a stand-in whisper-cli there that logs each run. */ +async function localSetup(t: TestContext) { + const root = await directory(t); + const home = join(root, 'home'), brew = join(root, 'brew'), bin = join(root, 'bin'), log = join(root, 'whisper.log'); + await mkdir(home); await mkdir(bin); await mkdir(join(brew, 'bin'), {recursive: true}); + const env = {PATH: bin, HOME: home, HOMEBREW_PREFIX: brew} as NodeJS.ProcessEnv; + return { + root, home, brew, bin, log, env, + async whisper() { + const file = join(bin, 'whisper-cli'); + // Logs its arguments and the first bytes of the file it was given, then prints whisper.cpp-style output. + await writeFile(file, `#!/bin/sh\nwhile [ $# -gt 0 ]; do printf '%s\\n' "$1" >> '${log}'; if [ "$1" = -f ]; then shift; printf 'arg:%s\\n' "$1" >> '${log}'; printf 'head:%s\\n' "$(head -c 4 "$1")" >> '${log}'; fi; shift; done\nprintf '[00:00:00.000 --> 00:00:02.000] Hello there.\\n[00:00:02.000 --> 00:00:03.000] [BLANK_AUDIO]\\n[00:00:03.000 --> 00:00:04.500] Ship it.\\n'\n`); + await chmod(file, 0o755); + }, + async model(dir: string, name: string) { await mkdir(dir, {recursive: true}); const path = join(dir, name); await writeFile(path, 'ggml model'); return path; }, + }; +} + +// Spec: MUS-SPEC-01-840L (docs/spec/clauses/01-chat-and-composer.md#mus-spec-01-840l) +test('the local backend transcribes the recording with whisper.cpp on this computer and keeps nothing', async t => { + const net = fakeFetch(() => new Response('{"text":"leak"}')); + const local = await localSetup(t); + await local.whisper(); + const model = await local.model(join(local.home, '.muster', 'models'), 'ggml-base.en.bin'); + const {dataDir, call} = await domain(t, {fetch: net.fetch, env: local.env}); + const logged: string[] = []; + for (const method of ['log', 'info', 'warn', 'error', 'debug'] as const) { + const original = console[method]; + console[method] = (...args: unknown[]) => { logged.push(args.map(String).join(' ')); }; + t.after(() => { console[method] = original; }); + } + const view = await call('dictation.config.set', {backend: 'local', localModel: model}); + assert.equal(view.ready, true, view.reason); + assert.equal(view.recording, 'wav', 'the composer is asked for WAV'); + const audio = wav(); + const result = await call('dictation.transcribe', {audio: audio.toString('base64'), mime: 'audio/wav'}); + assert.deepEqual(result, {text: 'Hello there. Ship it.'}, 'timestamps and markers are removed'); + assert.equal(net.calls.length, 0, 'no network request'); + const lines = (await readFile(local.log, 'utf8')).trim().split('\n'); + assert.deepEqual(lines.slice(0, 2), ['-m', model], 'the chosen model'); + assert.ok(lines.includes('-nt') && lines.includes('-np'), 'plain text output'); + assert.ok(lines.includes('head:RIFF'), 'whisper.cpp got the WAV recording'); + const given = lines.find(line => line.startsWith('arg:'))!.slice(4); + assert.equal(existsSync(given), false, 'the temporary recording is deleted'); + const pattern = audio.subarray(44, 84).toString('latin1'), base64 = audio.toString('base64').slice(60, 100); + for (const path of await filesUnder(dataDir)) { + if (!(await stat(path)).isFile()) continue; + const text = (await readFile(path)).toString('latin1'); + assert.equal(text.includes(pattern) || text.includes(base64), false, `${path} holds no audio`); + } + assert.equal(logged.some(line => line.includes(base64)), false, 'no audio in logs'); +}); + +// Spec: MUS-SPEC-01-840M (docs/spec/clauses/01-chat-and-composer.md#mus-spec-01-840m) +test('the local backend says what is missing and finds the models the muster CLI uses', async t => { + const net = fakeFetch(() => new Response('{"text":"leak"}')); + const local = await localSetup(t); + const {call} = await domain(t, {fetch: net.fetch, env: local.env}); + const listed = (await call('dictation.config.get')).backends.find((entry: {id: string}) => entry.id === 'local'); + assert.equal(listed.available, true, 'Local can be chosen'); + let view = await call('dictation.config.set', {backend: 'local'}); + assert.equal(view.ready, false); + assert.match(view.reason, /brew install whisper-cpp/, 'says how to install whisper.cpp'); + assert.equal(view.local.program, null); + + await local.whisper(); + view = await call('dictation.config.get'); + assert.equal(view.ready, false); + assert.match(view.reason, /~\/\.muster\/models/, 'says where to put a model'); + assert.equal(view.local.program, join(local.bin, 'whisper-cli')); + + const own = await local.model(join(local.home, '.muster', 'models'), 'ggml-small.bin'); + const brewed = await local.model(join(local.brew, 'share', 'whisper-cpp', 'models'), 'ggml-base.en.bin'); + await local.model(join(local.home, '.muster', 'models'), 'notes.txt'); + view = await call('dictation.config.get'); + assert.deepEqual(view.local.models, [own, brewed], 'models from ~/.muster/models and Homebrew'); + await assert.rejects(call('dictation.config.set', {localModel: join(local.home, 'missing.bin')}), /model file/); + await assert.rejects(call('dictation.config.set', {localModel: join(local.home, '.muster', 'models', 'notes.txt')}), /model file/); + view = await call('dictation.config.set', {localModel: brewed}); + assert.equal(view.ready, true, view.reason); + assert.equal(view.local.model, brewed); + + await assert.rejects(call('dictation.transcribe', {audio: AUDIO.toString('base64'), mime: 'audio/webm'}), /WAV/); + assert.equal(existsSync(local.log), false, 'nothing was run'); + assert.equal(net.calls.length, 0); +});