Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions docs/spec/clauses/01-chat-and-composer.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ Boundaries:
| [MUS-SPEC-01-840I](01-chat-and-composer.md#mus-spec-01-840i) | Cloud dictation through a saved OpenAI-compatible connection uses that connection's endpoint and stored key. | [`the cloud backend can use a saved provider connection and its key`](../../../packages/agent-app/tests/dictation-840.test.ts) |
| [MUS-SPEC-01-840J](01-chat-and-composer.md#mus-spec-01-840j) | On macOS, microphone access asks the system once when undecided and returns a System Settings message when denied, without asking. | [`microphone access asks macOS when undecided and explains a denial`](../../../packages/agent-app/tests/microphone-840.test.ts) |
| [MUS-SPEC-01-840K](01-chat-and-composer.md#mus-spec-01-840k) | The window allows a media request only for audio from the app's own page; video, other permissions and other pages are refused. | [`only the app page may open the microphone, and only for audio`](../../../packages/agent-app/tests/microphone-840.test.ts) |
| [MUS-SPEC-01-840L](01-chat-and-composer.md#mus-spec-01-840l) | With Local chosen, the WAV recording goes to the whisper.cpp program with the chosen model; its timestamps and silence markers are removed from the words; no network request is made; the temporary audio is gone afterwards and no audio is in the data folder or the logs. | [`the local backend transcribes the recording with whisper.cpp on this computer and keeps nothing`](../../../packages/agent-app/tests/dictation-840.test.ts) |
| [MUS-SPEC-01-840M](01-chat-and-composer.md#mus-spec-01-840m) | Without whisper.cpp the view says how to install it; without a model it says where to put one; models in `~/.muster/models` and Homebrew's folder are listed; a model path that is not a model file is refused; a recording that is not WAV is refused without running anything. | [`the local backend says what is missing and finds the models the muster CLI uses`](../../../packages/agent-app/tests/dictation-840.test.ts) |
| [MUS-SPEC-01-840N](01-chat-and-composer.md#mus-spec-01-840n) | With Local chosen, the recording is decoded and sent to `dictation.transcribe` as `audio/wav`: a RIFF header for 16 kHz, one channel, 16-bit samples, then the samples; the words land in the composer. | [`with the local backend the composer sends a 16 kHz mono WAV recording and inserts the words`](../../../packages/agent-app/tests/composer-dictation-840-dom.tsx) |
| [MUS-SPEC-01-840O](01-chat-and-composer.md#mus-spec-01-840o) | Choosing Local in Settings › Chat shows the found models and the setup hint; picking a model calls `dictation.config.set` with its path. | [`Settings › Chat chooses the local model and shows what is missing`](../../../packages/agent-app/tests/composer-dictation-840-dom.tsx) |
| [MUS-SPEC-01-401A](01-chat-and-composer.md#mus-spec-01-401a) | With the chat idle and the composer empty, the last turn's suggestion shows as ghost text in the composer (the textarea placeholder steps aside), and Tab puts exactly that text in the composer without sending. | [`the suggested follow-up shows as ghost text in an empty composer and Tab fills it`](../../../packages/agent-app/tests/follow-up-401-dom.tsx) |
| [MUS-SPEC-01-401B](01-chat-and-composer.md#mus-spec-01-401b) | Clicking the suggestion fills the composer and sends nothing; a Cmd/Ctrl-click sends the suggestion as the next message. | [`clicking the suggestion fills the composer, and a modifier-click sends it`](../../../packages/agent-app/tests/follow-up-401-dom.tsx) |
| [MUS-SPEC-01-401C](01-chat-and-composer.md#mus-spec-01-401c) | The suggestion is gone once anything is typed, and is not shown while a turn runs. | [`the suggestion hides once you type and while a turn runs`](../../../packages/agent-app/tests/follow-up-401-dom.tsx) |
Expand Down Expand Up @@ -436,6 +440,50 @@ Boundaries:
- Known gaps: None known.
- Evidence: UNVERIFIED

<a id="mus-spec-01-840l"></a>
#### MUS-SPEC-01-840L: A local model transcribes on this computer

- Precondition: Dictation is set to Local, whisper.cpp (`whisper-cli`) is installed and a ggml model file is chosen.
- Action: Transcribe a WAV recording.
- Expected result: The whisper.cpp program runs once with the chosen model and the recording (no shell), and the words come back without timestamps or markers such as `[BLANK_AUDIO]`. No network request is made. When the call ends the temporary copy of the recording is deleted, and no audio is left in the data folder or written to the logs.
- Test: `packages/agent-app/tests/dictation-840.test.ts::the local backend transcribes the recording with whisper.cpp on this computer and keeps nothing`
- Environment: Node test runner; the real dictation domain with a stand-in `whisper-cli` script on PATH that records its arguments and checks the WAV it is given; fake fetch; no microphone.
- Known gaps: A real whisper.cpp build and model are not run in CI; accuracy and speed depend on the model the person chooses.
- Evidence: UNVERIFIED

<a id="mus-spec-01-840m"></a>
#### MUS-SPEC-01-840M: The local backend says what is missing

- Precondition: Dictation is set to Local. First no whisper.cpp program and no model; then whisper.cpp and model files in `~/.muster/models` (the folder the muster CLI uses) and in Homebrew's whisper-cpp folder.
- Action: Read the dictation settings, choose a model, try a path that is not a model, and transcribe a recording that is not WAV.
- Expected result: Without whisper.cpp the settings are not ready and say to install it (`brew install whisper-cpp`). With it but no model, they say where to put a model. Found models are listed, and choosing one makes the backend ready. A path that is not an existing `.bin` file is refused. A recording that is not WAV is refused with a clear message and nothing is run.
- Test: `packages/agent-app/tests/dictation-840.test.ts::the local backend says what is missing and finds the models the muster CLI uses`
- Environment: Node test runner; the real dictation domain with a temp home folder, a temp PATH and a stand-in `whisper-cli`.
- Known gaps: Muster does not download models; the person gets one themselves (the hint says from where).
- Evidence: UNVERIFIED

<a id="mus-spec-01-840n"></a>
#### MUS-SPEC-01-840N: With Local the composer sends WAV

- Precondition: Dictation is set to Local and ready. The microphone is allowed.
- Action: Click the mic, then click it again.
- Expected result: The recording is decoded and resampled to 16 kHz mono and sent to `dictation.transcribe` as `audio/wav` with a valid RIFF header (16,000 Hz, 1 channel, 16-bit) followed by the samples. The transcript fills the composer.
- Test: `packages/agent-app/tests/composer-dictation-840-dom.tsx::with the local backend the composer sends a 16 kHz mono WAV recording and inserts the words`
- Environment: Real components in a DOM suite (linkedom); fake MediaRecorder, getUserMedia and Web Audio decoding; fake app bridge; no microphone.
- Known gaps: Chromium's real decoding of a MediaRecorder recording is not run here; it is checked by hand in the app.
- Evidence: UNVERIFIED

<a id="mus-spec-01-840o"></a>
#### MUS-SPEC-01-840O: Settings › Chat sets up the local model

- Precondition: Settings › Chat is open. The local backend reports two found models and that no model is chosen yet.
- Action: Choose Local, then pick a model.
- Expected result: Local is offered in the Dictation choice. Once chosen, a row lists the found models and shows the setup hint. Picking one calls `dictation.config.set` with that model's path.
- Test: `packages/agent-app/tests/composer-dictation-840-dom.tsx::Settings › Chat chooses the local model and shows what is missing`
- Environment: Real components in a DOM suite (linkedom); fake app bridge.
- Known gaps: None known.
- Evidence: UNVERIFIED

### Suggested follow-up and model markers (#401)

<a id="mus-spec-01-401a"></a>
Expand Down
1 change: 1 addition & 0 deletions packages/agent-app/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ All notable changes to Muster Agent. Each `## <version>` section becomes the not
- A suggested follow-up after a turn, as ghost text in the empty message field: Tab or click puts it there, Cmd/Ctrl-click sends it. Off by default (Settings › Chat › Suggest a follow-up) since it is one small extra request per turn: Claude Code chats use Claude Code's own suggestion, API and gateway chats ask the same model once; Codex chats offer none. The transcript also marks where the model or effort changed between turns: "Switched to GPT-5.5 · high effort" (#401).
- Commits in the History graph have a menu (right-click, or Shift+F10 on a row): copy the SHA or message, check the commit out (detached), create a branch there, cherry-pick it onto the current branch or revert it. A picker shows another branch’s commits to pick from. Each change confirms first, saying what will happen; it is refused while tracked files have uncommitted changes, warns when a chat is running in the folder, and a conflicting pick or revert opens the conflict view. History and Changes refresh afterwards (#409).
- Automations can run once at a set date and time: pick **Once at a date and time** under Repeat. The time is in the automation's time zone and is shown with it. After it runs the automation stays in the list as Done with its run linked; a time that has already passed is refused when you save. If Muster was closed or asleep at that time, it runs once at the next start and its history says it ran late (or records it as missed when catch-up is off). Before it runs you can move it to another chat with Edit (#845).
- **Dictation with a local model.** Settings › Chat › Dictation now offers Local model: whisper.cpp (`brew install whisper-cpp`) transcribes your recording on this computer, so the audio never leaves it. Muster finds models in `~/.muster/models` (the same folder the muster CLI uses) and in Homebrew's whisper-cpp folder, or you can give it a model file path.

## 0.3.18

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,7 @@ function DictationSection():React.ReactElement {
const [endpoint,setEndpoint]=useState('');
const [model,setModel]=useState('');
const [key,setKey]=useState('');
const [modelPath,setModelPath]=useState('');
const [busy,setBusy]=useState(false);
const providers=useStoreSelector(state=>state.providers);
const connections=(('value' in providers?providers.value:undefined)??[]).filter(provider=>provider.custom&&provider.endpoint);
Expand Down Expand Up @@ -312,6 +313,18 @@ function DictationSection():React.ReactElement {
</span>
</span>
</Row>}
{config.backend==='local'&&<Row title="Local model" scope={device().title+" · audio stays on this computer"} description={config.ready?`whisper.cpp transcribes on this computer${config.local.program?` (${config.local.program})`:''}. Models in ~/.muster/models are shared with the muster CLI.`:config.reason}>
<span className="dictation-settings">
<select aria-label="Local whisper model" value={config.local.model} disabled={busy||!config.local.models.length&&!config.local.model} onChange={event=>void save({localModel:event.target.value})}>
{!config.local.model&&<option value="">{config.local.models.length?'Choose a model':'No model found'}</option>}
{[...new Set([...(config.local.model?[config.local.model]:[]),...config.local.models])].map(path=><option key={path} value={path}>{path.split('/').pop()} ({path})</option>)}
</select>
<input type="text" aria-label="Whisper model path" placeholder="/path/to/ggml-base.en.bin" spellCheck={false} value={modelPath} disabled={busy} onChange={event=>setModelPath(event.target.value)}/>
<span className="dictation-settings-actions">
<button type="button" className="settings-button" disabled={busy||!modelPath.trim()} onClick={()=>void save({localModel:modelPath.trim()}).then(ok=>{if(ok)setModelPath('');})}>Use this model</button>
</span>
</span>
</Row>}
</>;
}

Expand Down
13 changes: 8 additions & 5 deletions packages/agent-app/src/renderer/dictation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
*/
import { useEffect, useRef, useState } from 'react';
import { DICTATION_HOLD_MS, DICTATION_MAX_SECONDS, MICROPHONE_SETTINGS_HINT } from '../shared/domains/dictation-protocol';
import { recordingAsWav } from './wavRecording';
import { invoke } from './bridge';
import { runtimeMessage as errorText } from './composerBridge';
import { openAppSettings, pushNotice } from './store';
Expand Down Expand Up @@ -50,7 +51,7 @@ function base64(bytes: Uint8Array): string {
const pickMime = (Class: RecorderClass) => MIME_PREFERENCE.find(type => { try { return Class.isTypeSupported?.(type) ?? false; } catch { return false; } });
const release = (stream: MediaStream | undefined) => { for (const track of stream?.getTracks() ?? []) { try { track.stop(); } catch { /* already stopped */ } } };

interface Session { stream: MediaStream; recorder: Recorder; chunks: Blob[]; mime: string; cancelled: boolean; timer: ReturnType<typeof setTimeout> }
interface Session { stream: MediaStream; recorder: Recorder; chunks: Blob[]; mime: string; cancelled: boolean; timer: ReturnType<typeof setTimeout>; /** The backend reads WAV only (local). */ wav: boolean }

export interface DictationOptions {
/** Final words to insert at the caret. */
Expand Down Expand Up @@ -91,14 +92,16 @@ export function useDictation(options: DictationOptions) {
};

const transcribe = async (current: Session) => {
const blob = new Blob(current.chunks, { type: current.mime });
const recorded = new Blob(current.chunks, { type: current.mime });
current.chunks.length = 0;
if (!blob.size) { latest.current.flash('No audio was recorded'); move('idle'); return; }
if (!recorded.size) { latest.current.flash('No audio was recorded'); move('idle'); return; }
move('transcribing');
const mine = generation.current;
try {
const blob = current.wav ? await recordingAsWav(recorded) : recorded;
if (mine !== generation.current) return;
const audio = base64(new Uint8Array(await blob.arrayBuffer()));
const result = await invoke('dictation.transcribe', { audio, mime: current.mime });
const result = await invoke('dictation.transcribe', { audio, mime: current.wav ? 'audio/wav' : current.mime });
if (mine !== generation.current) return;
const words = result?.text?.trim() ?? '';
if (words) latest.current.onText(words); else latest.current.flash('No speech heard');
Expand Down Expand Up @@ -128,7 +131,7 @@ export function useDictation(options: DictationOptions) {
if (mine !== generation.current) { release(stream); return; }
const Class = recorderClass()!, wanted = pickMime(Class);
const recorder = new Class(stream, wanted ? { mimeType: wanted } : undefined);
const current: Session = { stream, recorder, chunks: [], mime: recorder.mimeType || wanted || 'audio/webm', cancelled: false,
const current: Session = { stream, recorder, chunks: [], mime: recorder.mimeType || wanted || 'audio/webm', cancelled: false, wav: config.recording === 'wav',
timer: setTimeout(() => { if (session.current === current) finish(); }, DICTATION_MAX_SECONDS * 1000) };
recorder.ondataavailable = event => { if (event.data && event.data.size) current.chunks.push(event.data); };
recorder.onerror = () => { if (session.current === current) cancel(); latest.current.flash('Recording stopped unexpectedly'); };
Expand Down
43 changes: 43 additions & 0 deletions packages/agent-app/src/renderer/wavRecording.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
/**
* #840: the local dictation backend (whisper.cpp) reads 16 kHz mono 16-bit WAV, and MediaRecorder makes webm/opus.
* Chromium decodes the recording, an OfflineAudioContext mixes it to one channel at 16 kHz, and the samples are
* written as PCM WAV. Everything stays in memory.
*/
import { DICTATION_WAV_RATE } from '../shared/domains/dictation-protocol';

type AudioScope = { AudioContext?: typeof AudioContext; webkitAudioContext?: typeof AudioContext; OfflineAudioContext?: typeof OfflineAudioContext };

/** PCM 16-bit mono WAV of the given samples (-1..1). */
export function encodeWav(samples: Float32Array, rate = DICTATION_WAV_RATE): Uint8Array<ArrayBuffer> {
const out = new Uint8Array(44 + samples.length * 2), view = new DataView(out.buffer);
const text = (at: number, value: string) => { for (let i = 0; i < value.length; i++) out[at + i] = value.charCodeAt(i); };
text(0, 'RIFF'); view.setUint32(4, 36 + samples.length * 2, true); text(8, 'WAVE'); text(12, 'fmt ');
view.setUint32(16, 16, true); view.setUint16(20, 1, true); view.setUint16(22, 1, true); view.setUint32(24, rate, true);
view.setUint32(28, rate * 2, true); view.setUint16(32, 2, true); view.setUint16(34, 16, true);
text(36, 'data'); view.setUint32(40, samples.length * 2, true);
for (let i = 0; i < samples.length; i++) {
const sample = Math.max(-1, Math.min(1, samples[i]!));
view.setInt16(44 + i * 2, sample < 0 ? Math.round(sample * 0x8000) : Math.round(sample * 0x7fff), true);
}
return out;
}

/** The recording as 16 kHz mono WAV. Throws a plain message when this window cannot decode it. */
export async function recordingAsWav(blob: Blob): Promise<Blob> {
const scope = globalThis as unknown as AudioScope;
const Context = scope.AudioContext ?? scope.webkitAudioContext, Offline = scope.OfflineAudioContext;
if (!Context || !Offline) throw new Error('This window cannot convert the recording for the local model.');
const context = new Context();
let decoded: AudioBuffer;
try { decoded = await context.decodeAudioData(await blob.arrayBuffer()); }
catch { throw new Error('The recording could not be read. Try again.'); }
finally { void context.close().catch(() => undefined); }
const frames = Math.max(1, Math.ceil(decoded.duration * DICTATION_WAV_RATE));
const offline = new Offline(1, frames, DICTATION_WAV_RATE);
const source = offline.createBufferSource();
source.buffer = decoded;
source.connect(offline.destination);
source.start();
const rendered = await offline.startRendering();
return new Blob([encodeWav(rendered.getChannelData(0))], { type: 'audio/wav' });
}
Loading
Loading