Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
a8473a8
Document accepted local Voice program
MarcusJRLee Aug 26, 2026
01629a4
Add the macOS Voice M1 history tracer
MarcusJRLee Aug 26, 2026
7943ef7
Test Voice history after insertion failure
MarcusJRLee Aug 26, 2026
3ec59f1
Add macOS Voice hold and latch trigger
MarcusJRLee Aug 26, 2026
21f8427
Add structured Voice formatting styles
MarcusJRLee Aug 26, 2026
24b13b8
Add replayable Voice spoken edits
MarcusJRLee Aug 26, 2026
1186fe0
Preserve Voice target ownership
MarcusJRLee Aug 26, 2026
67c0347
Add reusable local Voice History
MarcusJRLee Aug 26, 2026
32cacc2
Add bounded Voice History audio retention
MarcusJRLee Aug 26, 2026
1a97a3b
Recover interrupted Voice History artifacts
MarcusJRLee Aug 26, 2026
e51e0d4
Enforce local-only Voice provider boundaries
MarcusJRLee Aug 26, 2026
1bd0f6b
Stabilize shared CI verification
MarcusJRLee Aug 27, 2026
6287981
Remove simulated crash test races
MarcusJRLee Aug 27, 2026
8ccda14
Harden Voice History writer convergence
MarcusJRLee Aug 27, 2026
1f953cd
Converge macOS Voice trigger entry points
MarcusJRLee Aug 27, 2026
90fe962
Add portable Voice retention tracer
MarcusJRLee Aug 27, 2026
c60a4fc
Import local audio into Voice History
MarcusJRLee Aug 27, 2026
cf8e402
Document M12 signed validation
MarcusJRLee Aug 27, 2026
9af9a0b
Validate portable Voice model packages
MarcusJRLee Aug 27, 2026
43cd5e5
Document M13 signed validation
MarcusJRLee Aug 27, 2026
9d9664a
Add portable Voice History archives
MarcusJRLee Aug 27, 2026
e6cf1d9
Document M14 signed validation
MarcusJRLee Aug 27, 2026
ce888a4
Link the portable Voice runtime on macOS
MarcusJRLee Aug 27, 2026
246f16f
Document M15 signed validation
MarcusJRLee Aug 27, 2026
c5fd24e
feat: prove local ios keyboard handoff
MarcusJRLee Aug 27, 2026
fd41ea7
feat: add local ios onboarding
MarcusJRLee Aug 27, 2026
a09c301
feat: admit local ios model packages
MarcusJRLee Aug 27, 2026
efa5b6a
feat(ios): add verified local whisper transcription
MarcusJRLee Aug 27, 2026
a5a16e5
fix(ci): scope ASR performance to named hardware
MarcusJRLee Aug 27, 2026
49008b1
fix(voice): serialize startup recovery before finalization
MarcusJRLee Aug 27, 2026
ed0c0c9
feat(ios): add local formatting and durable history
MarcusJRLee Aug 27, 2026
70003ac
feat(ios): qualify keyboard delivery by style
MarcusJRLee Aug 27, 2026
e47879e
feat(ios): guard voice delivery by host field
MarcusJRLee Aug 27, 2026
d43be06
feat(ios): recover lifecycle-interrupted capture
MarcusJRLee Aug 27, 2026
f8b916d
feat(ios): bound stale service recovery
MarcusJRLee Aug 27, 2026
1b80356
feat(ios): recover unconfirmed insertion
MarcusJRLee Aug 27, 2026
fb4e58a
feat(ios): enforce offline storage limits
MarcusJRLee Aug 27, 2026
b5dfd54
feat(ios): finish system-surface capture
MarcusJRLee Aug 27, 2026
c6a5a59
CI: target Swift CodeQL builds (#36)
MarcusJRLee Aug 28, 2026
222f354
Add interactive iOS device installer (#37)
MarcusJRLee Aug 28, 2026
c597c92
Keep installed-build evidence stable (#38)
MarcusJRLee Aug 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ on:
pull_request:
push:
branches:
- dev
- main

permissions:
Expand Down
85 changes: 85 additions & 0 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
name: CodeQL

on:
pull_request:
branches:
- dev
- main
push:
branches:
- dev
- main
schedule:
- cron: "23 7 * * 1"

permissions:
contents: read
security-events: write

concurrency:
group: codeql-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
analyze:
name: Analyze (swift)
runs-on: macos-26
timeout-minutes: 45
steps:
- name: Check out source
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false

- name: Detect Swift build changes
id: changes
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
run: |
analyze=false
if [[ "${{ github.event_name }}" == "schedule" ]]; then
analyze=true
else
while IFS= read -r changed_path; do
case "$changed_path" in
.github/workflows/codeql.yml|Cargo.lock|Cargo.toml|Package.resolved|Package.swift|rust-toolchain.toml|Sources/*|apps/ios/voice_input/*|crates/*|scripts/build_ios_rust_ffi.sh|scripts/build_rust_ffi.sh|scripts/fetch_ios_asr_runtime.sh)
analyze=true
break
;;
esac
done < <(git diff --name-only --diff-filter=ACMRT "$BASE_SHA" "$HEAD_SHA")
fi
echo "analyze=$analyze" >> "$GITHUB_OUTPUT"

- name: Initialize CodeQL
if: steps.changes.outputs.analyze == 'true'
uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
with:
build-mode: manual
languages: swift
queries: security-extended

- name: Build Swift products
if: steps.changes.outputs.analyze == 'true'
run: |
scripts/build_rust_ffi.sh
swift build --arch arm64
rustup target add aarch64-apple-ios aarch64-apple-ios-sim
scripts/build_ios_rust_ffi.sh
scripts/fetch_ios_asr_runtime.sh
xcodebuild build -quiet \
-project apps/ios/voice_input/VoiceInput.xcodeproj \
-scheme VoiceInput \
-configuration Debug \
-destination "generic/platform=iOS Simulator" \
-derivedDataPath "$RUNNER_TEMP/codeql_ios" \
ARCHS=arm64 \
CODE_SIGNING_ALLOWED=NO \
CODE_SIGNING_REQUIRED=NO \
ONLY_ACTIVE_ARCH=YES

- name: Perform CodeQL analysis
if: steps.changes.outputs.analyze == 'true'
uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ xcuserdata/
.swiftpm/
Package.resolved

# Rust
target/
Sources/hardware_controller_voice_ffi/voice_ffi_build_stamp.generated.swift

# Local configuration and diagnostics
.env
.env.*
Expand Down
17 changes: 16 additions & 1 deletion CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,23 @@ identify deep implementation modules without changing that language.
| Speech input | Configuration-leased microphone capture plus an on-device Apple recognition session. It copies callback-owned samples before crossing isolation. |
| Dictation coordinator | Process-wide serial owner that cancels one Dictation workflow before beginning the other. |
| Local Dictation controller | Existing live-composition workflow. It owns recognition, adaptive target delivery, finalization, and recovery without model refinement. |
| Local AI Dictation controller | Final-only workflow that composes the existing recognition controller, starts model preparation during speech, refines immutable text, validates it, and delivers refined or raw fallback once. |
| Local AI Dictation controller | Final-only workflow that composes recognition, starts model preparation during speech, validates refinement, delivers refined or Raw fallback once, and finalizes one Voice session. |
| Voice session History | Searchable local archive whose session owns at most one audio artifact and an append-only graph of immutable Raw, Edited, Formatted, Delivered, and corrected results. |
| Audio artifact recorder | Bounded nonblocking tee from immutable capture buffers to an atomically finalized local CAF. |
| Voice session store | Actor-owned system SQLite connection that serializes local session metadata transactions. |
| Voice History service | Actor that retranscribes retained audio, reformats reusable text, retries delivery, and appends each outcome as a linked immutable result. |
| Voice audio importer | Actor that bounds a user-selected recording, runs local ASR/formatting, streams one app-owned CAF, and commits typed imported-audio History without mutating the source. |
| Voice History archive | Bounded portable directory containing one V1 manifest, one checksum contract, and optional CAF; it preserves immutable evidence without representing a mutable store. |
| Voice archive importer | Actor that privately snapshots, verifies, and transactionally restores one Voice History archive without delivery. |
| Reusable result | Newest nonempty result selected deterministically from one session for copy, correction, retranscription, reformatting, export, or explicit re-delivery. |
| Voice trigger | Input adapter that maps physical, exact-chord, or in-app intent into the shared Voice-session contract. |
| Voice chord | Optional machine-wide exact shortcut dedicated to Voice capture and independent of Binding keyboard fallbacks. |
| Latched capture | Voice capture kept active after a valid double press until the next valid double press. |
| Refinement provider | Typed local text-to-text boundary implemented by Apple Foundation Models or fixed-loopback Ollama. |
| Portable Voice core | Dependency-free Rust domain policy shared through versioned CUJ fixtures; it contains no platform lifecycle or UI behavior. |
| Portable archive verifier | Safe Rust boundary that verifies the exact Voice History inventory, limits, identities, and digests before a platform decodes and restores typed evidence. |
| Voice FFI | Versioned synchronous C ABI over portable Voice crates; callers own every buffer and no pointer survives a call. |
| Apple Voice adapter | Typed, pointer-free Swift values and failures over the statically linked Voice FFI; it adds no portable policy. |
| Target lease | Captured editable element, process, caret/selection, and delivery capability revalidated before mutation. |
| Nearby context | Optional bounded text around the caret from an approved nonsecure multiline target, held only for one Local AI session. |
| Personal dictionary | Machine-wide recognition vocabulary plus deterministic spoken-form replacements. |
Expand Down
17 changes: 11 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ scope and acceptance evidence are explicit.

## Start locally

Requirements: Apple silicon, macOS 15 or later, and Xcode 26 or a compatible
Swift 6 toolchain.
Requirements: Apple silicon, macOS 15 or later, Xcode 26 or a compatible Swift
6 toolchain, and rustup. The repository pins Rust 1.98.

```bash
git clone https://github.com/MarcusJRLee/hardware_controller.git
cd hardware_controller
swift run HardwareController --demo
scripts/run_demo.sh
```

Demo mode is deterministic and requires no Device, Apple signing identity, or
Expand Down Expand Up @@ -41,9 +41,14 @@ test placement, hardware fixtures, and signed Device testing.
- Do not change version or build metadata, create release artifacts, or publish
a Release without explicit approval for that exact version.

GitHub requests review from the repository owner. Automated verification and
CodeQL must pass, review threads must be resolved, and accepted changes are
squash-merged into `main`.
GitHub requests review from the repository owner. Automated verification must
pass, review threads must be resolved, and accepted changes are squash-merged.
The `Analyze (swift)` gate performs an extended CodeQL scan when Swift or its
build inputs change and completes without compiling Swift for unrelated
changes. During the accepted Voice program, focused pull requests target
`dev`; `dev` returns to `main` only after the completed program receives final
user verification. See
[`0029_local_voice_platform_expansion.md`](docs/decisions/0029_local_voice_platform_expansion.md).

## Privacy and hardware evidence

Expand Down
216 changes: 216 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading