Repository navigation
[DRAFT/prototype] Native audio reading experience (#1580) - #1581
openlibrary-bot wants to merge 23 commits into
Conversation
|
Progress update — PocketTTS synthesis now works, and the branch has moved to this repo. Branch location. PocketTTS is producing speech. The reference runtime is Python, so the flow-matching loop is now ported to JS ( The session signatures corroborate the port independently: 20 = 2 real + 18 state tensors, 57 = 1 + 56, exactly the I cannot listen to the output, so I compared its acoustics against the human reference clip: spectral centroid 1560Hz vs 1620Hz, 98.9% vs 99.4% of energy below 4kHz, envelope dynamics 27x vs 21x, RMS 0.030 vs 0.029, and 18 words in 5.28s (3.4 words/sec). That is a speech profile rather than noise. Intelligibility by ear remains unverified. Two pieces of incidental work that may be of wider interest:
Tests: 216 jest + 14 Playwright, lint clean. The generation loop is covered with stand-in ONNX sessions so the suite needs none of the ~146MB of weights; real-weight verification is Still to do: Web Worker wrapper, Cache API model storage, |
|
Milestone: PocketTTS now runs client-side in the browser and produces real speech. This closes the loop on the part of #1580 that was genuinely uncertain. Real audio from the real model, in a real browserA Playwright test drives the demo with 28,800 samples at 24kHz is 1.2s of audio for the landmark segment of the first paragraph, drawn from this book's own OCR. The assertions are deliberately the ones silence would fail: non-zero peak, non-zero RMS, and RMS well below peak (a constant tone or DC offset would not satisfy that). A second test then confirms those samples reach the audio device with a non-zero peak and a The in-browser voice-clone result matches the Node verification exactly — 24kHz, 112 voice frames — which is a useful independent check that the worker path and the direct path agree. How it is put together
Test status216 jest + 17 Playwright, lint clean. New coverage in this milestone:
New screenshots: The PocketTTS browser tests skip rather than fail when the weights are not present locally, so a clean checkout still runs the rest of the suite. They point at a gitignored local copy instead of HuggingFace to keep the run fast; the fetch commands are documented in the spec file. Honest limits
Still a draft — not for review, not for merge. |
|
Milestone: the hybrid preview path is built, which was the last substantive piece the issue describes. What it doesThe issue floats this directly — "a hybrid of PocketTTS and WebSpeechSynthesis (much faster) to provide an instant fallback preview while the higher quality voices are loading" — and it is the answer to PocketTTS's one real weakness in a browser: at ~RTFx 1.7 on a single wasm thread, a paragraph the buffer has not reached yet costs seconds of silence. The rule falls out of when each decision gets made:
So reading straight through sounds like PocketTTS; a cold seek speaks immediately and recovers quality from the next segment. Nothing is thrown away — late PocketTTS audio stays in the queue's cache for a seek back. Verified in a real browser: Two bugs this shook outBoth were mine, both in the hybrid, both would have silently degraded the feature rather than crashing:
An honest note on how the takeover is testedThe preview half tests cleanly with WebSpeech: the utterance starts immediately, with a real voice, speaking the paragraph's landmark segment. The takeover half cannot be observed with WebSpeech as the preview voice, for the environment reason documented earlier in this PR: a WebSpeech sound never fires Rather than assert something weaker, the preview engine is now selectable ( Tests233 jest + 19 Playwright, lint clean. 17 new unit tests pin the hybrid's decisions: quality-wins-inside-grace (and that no preview is even synthesized then), fallback outside it, upgrade-before-play, upgrade counted once across replays, permanent fallback on failure, pause/resume reaching whichever engine is live, stop silencing both mid-handover, and rate changes applying to both so a handover does not change speed. New screenshots: Where this leaves the prototypeEverything issue #1580 asks for is now built and demonstrable: the minimal audio-first view, real OCR read aloud, progressive chunk rendering, the ≤5-paragraph buffer, the spinner, seek throttling, PocketTTS client-side in WASM, and the hybrid preview. Out of scope and untouched: the accessibility word-window mode (the issue calls it future work), mobile packaging, and cross-browser support. Still a draft — not for review, not for merge. |
aaca573 to
a61dc8f
Compare
Core, engine-agnostic pieces of the internetarchive#1580 audio-first prototype, with tests. - textSegments: splits a paragraph into the priority order the issue asks for (first 3 words as a 'landmark', rest of the sentence, then the rest of the paragraph split by sentence so synthesis units stay small). - ParagraphSource: OCR paragraph supply addressed by (leaf, chunk), with page caching, in-flight de-duplication, blank-page skipping, and a lookahead window. - SynthesisQueue: single-worker priority queue so background loading happens in order without overloading the CPU; caches results, evicts what leaves the buffer, and aborts in-flight work when a seek makes it no longer urgent. 56 jest tests.
AudioReaderPlayer ties the source, segmenter and synthesis queue together: 5-paragraph lookahead hydrated in issue-internetarchive#1580 priority order, a loading flag that is on exactly when the segment being waited on is not buffered, and next/prev seeks that stop audio immediately but coalesce a burst of presses into a single buffer rebuild. Generation counter guards against a stale playback loop or hydration overtaking a newer seek. 28 further jest tests (84 total in tests/jest/audioreader).
- IaAudioBook: item metadata + BookReader manifest + Open Library table of contents, exposing only what audio mode needs. Reuses PageChunk's text parser; requests no page images. - AudioReaderView: the minimal Lit view from the issue -- cover, TOC, transport -- with segment styling that makes progressive rendering visible (read / current / buffered / not yet synthesized) plus a debug buffer overlay. - WebSpeechEngine: adapts the read-aloud plugin's WebTTSSound, keeping its browser workarounds, to the player's engine interface. - simulateLatency: optional synthesis delay so the buffering, spinner and seek-cancellation behaviour is observable before PocketTTS exists. - Standalone webpack config + jqueryShim, so the bundle stays jQuery-free (250KB, boots no BookReader) while still importing the real tts modules. - Starts reading at the book proper, not the cover: OCR chunks that are stray marks are skipped, and the first TOC entry chooses the start leaf. demo at /BookReaderDemo/demo-audioreader.html; 112 jest tests, lint clean.
- PcmAudioOutput: Web Audio playback for engines that emit samples, which is what PocketTTS will need. Hand-rolled pause/resume (an AudioBufferSourceNode is single-use) plus sample/peak counters so 'audio happened' is measurable rather than assumed. - Watchdog completion: an AudioBufferSourceNode's 'ended' event is not guaranteed -- an interrupted or non-rendering context never fires it, which would strand the reader on one segment for the rest of the book. Fall back to wall-clock, counted separately so a recovery is never mistaken for playback. - Fix: resuming after a pause started a second playback loop, so both advanced on the same sound and a segment was skipped. Regression test added. - SyntheticPcmEngine (?engine=pcm): renders measurable tones, not speech. Needed because neither speechSynthesis nor Web Audio can witness a sound *finishing* under Playwright here -- the AudioContext clock does not advance without an audio device (0.005s per 2 real seconds, headless and headed alike). 14 Playwright tests green with screenshots; 113 jest tests; lint clean.
…ights Ports the reference Python runtime (pocket_tts_onnx.py) so PocketTTS can run client-side. Runtime-agnostic: handed a session factory, so the same code runs under onnxruntime-web in a worker and under Node for verification. - SentencePieceUnigram: the model ships a sentencepiece ModelProto and there is no JS tokenizer to borrow, so this parses the protobuf and implements Viterbi segmentation with byte fallback. Byte pieces are kept out of the lattice -- their 0.0 score outranks every real piece's log probability. Checked against sentencepiece 0.2.2 token-for-token on ASCII, accents, Japanese, emoji and tabs. - PocketTtsSynthesizer: text conditioner -> flow_lm_main per frame -> flow_lm_flow integration -> mimi decoder, threading all 18 flow and 56 mimi state tensors explicitly. EOS detection, token-derived frame budget, sentence/clause chunking to the 50-token limit, and abort support. - resample: windowed-sinc rational resampling, because the only ungated voice source is a 16kHz reference clip and the encoder wants 24kHz. Naive interpolation colours the timbre we are trying to clone. - scripts/verify-pocket-tts.mjs: runs the whole thing against the real int8 weights and writes a WAV. Verified end to end against the actual models: 5.28s of audio for an 18-word sentence of real Plato OCR (3.4 words/s), RTFx 1.7x on one wasm thread. Its acoustic profile matches the human reference clip closely -- spectral centroid 1560Hz vs 1620Hz, 98.9% vs 99.4% of energy below 4kHz, envelope dynamics 27x vs 21x -- which is what distinguishes speech from noise. I have not listened to it. Voices: the 8 named presets are unusable in a browser (gated repo, 401), so the voice is cloned from ungated reference audio via the mimi encoder. 91 further jest tests (216 total in tests/jest/audioreader).
…eights Closes the loop on the uncertain part of internetarchive#1580: PocketTTS now synthesizes real speech client-side. - pocket-tts-worker: synthesis runs off the main thread. Per-frame ONNX inference on it would stall rendering and make the transport unresponsive, which is the opposite of the experience this issue is about. - modelStore: the ~146MB int8 bundle is cached in the Cache API keyed by bundle name, so it downloads once, not per page load. Fetches are sequential on purpose -- eight parallel downloads that size compete with the book's own text requests and make progress meaningless -- and progress is per byte, since '3 of 5 files' says nothing when one file is 76MB. - PocketTtsEngine: implements the same engine interface as the WebSpeech path, so the existing buffering, spinner and seek-cancellation logic needed no changes. Cancellation reaches into the worker, because a stale frame loop would hold the single synthesis slot against what the patron is waiting for. - build-ort copies onnxruntime's wasm into BookReader/ort/, following the build-bergamot precedent for the translator worker. - The view now shows the model download; 146MB of unexplained silence reads as a broken page. Verified in a real browser, asserting on the samples themselves rather than on playback side effects: 'THE APOLOGY OF' -> 28800 samples at 24kHz (1.2s), peak 0.173, rms 0.022, with rms well below peak so a tone or DC offset would fail. Worker and Node paths agree on the clone (24kHz, 112 voice frames). Also adds a steady-state buffer test: the lookahead holds at exactly 5 paragraphs sampled repeatedly during playback, and the test proves the cursor advanced while measuring so it cannot pass by standing still. PocketTTS browser tests skip rather than fail without local weights.
The assertion about the lookahead holding at 5 was passing; the test's own 'was it actually advancing' guard was failing, because a paragraph of this book is ~15s of audio and the loop sampled for only 12s. Poll for the paragraph crossing instead of waiting a fixed time, and take extra samples on the far side so the rebuilt window is measured too.
The arrangement issue internetarchive#1580 suggests, and the answer to PocketTTS's one real weakness in a browser: at ~RTFx 1.7 on a single wasm thread, a paragraph the buffer has not reached yet costs seconds of silence. The rule falls out of *when* each decision is made: - at synthesis time, give PocketTTS a short grace period; once the lookahead is warm it wins, because synthesis started paragraphs ago - if the grace expires, hand back a WebSpeech preview and leave PocketTTS running - at play time, check again -- if PocketTTS landed meanwhile, play that instead So reading straight through is high quality, while a cold seek speaks at once and recovers quality from the next segment. Nothing is discarded: late PocketTTS audio stays cached for a seek back. Two bugs found and fixed while testing it: - the 'has the quality promise already settled?' check could never win its race against an immediate promise, so the upgrade never fired. Record the result on the holder when it settles instead of re-racing at play time. - a segment whose quality synthesis failed outright was routed to the quality engine for playback, because a bare fallback sound was indistinguishable from a quality one. Wrap it too, so play() always knows which engine owns the audio. Verified in a real browser: counts {quality: 2, preview: 1, upgraded: 2} -- WebSpeech (voice Samantha) covered the long segment while PocketTTS delivered 49920 samples at peak 0.187 for the rest. 17 new jest tests (233 total).
… honest
Two test problems, both mine:
- The hybrid browser test carried both engine=pocket and engine=hybrid in one
query. URLSearchParams.get returns the first value, so it silently ran plain
PocketTTS and the hybrid was never exercised.
- It asserted the preview speaks the paragraph's landmark segment. Which segment
falls back is timing-dependent by design: when PocketTTS keeps up, early
segments are upgraded to quality and only a later, longer one previews. Assert
the invariant -- the preview speaks real text from the current paragraph, with a
real voice -- rather than a specific segment.
The takeover half cannot be watched with WebSpeech previewing, because a
WebSpeech sound never fires 'end' in this browser, so playback parks on the first
preview and PocketTTS never gets a turn. Rather than assert something weaker, the
preview engine is now selectable (&fast=) and the takeover test previews with the
measurable PCM engine so reading advances. The hybrid is engine-agnostic anyway,
and being able to choose a preview voice is useful where speechSynthesis is
missing.
19 Playwright tests green; verified split {quality: 1, preview: 1, upgraded: 1}.
…e worker ## UI I built the first view from the issue's text description and never opened the screenshots, which the brief explicitly said to reproduce. It was wrong in almost every particular: dark navy instead of cream, chapters hidden behind a hamburger instead of listed, no scrub bar, blue accent instead of dark brown. Rebuilt against the screenshot: warm cream page, back arrow and title, centred cover, scrub bar with position labels, one large dark-brown circular play control between circular-arrow skips, and an always-visible white 'Chapters' card with zero-padded numbers and a bullet on the current chapter. Two deliberate departures, both because the design targets LibriVox recordings rather than TTS: - the scrub bar shows position in the book, not elapsed/total time. A recording knows it is 22:58 long; a book synthesized on the fly cannot know its duration without fetching and measuring every page, and inventing a total would be worse than showing pages. - skips move by paragraph, which is what the issue asks for. - the cover keeps its real aspect ratio rather than the design's square: book covers are portrait, and cropping to a square cut away a third of it (measured 180x299 natural rendered as 236x236). The paragraph text is not in the design, so it is now behind ?text=1 -- it stays available because it is how the progressive chunking can be seen and not merely heard. The rate control occupies the design's history-icon slot rather than shipping a dead button. ## Worker serialization Measuring the hybrid showed 10-12 synthesis requests outstanding at once: a worker message handler is async, so every queued request started the moment its message arrived, running a dozen generation loops concurrently against one set of ONNX sessions. That defeats the deliberate one-at-a-time CPU discipline and makes each request slower, so audio lands later than it needs to. Requests are now chained, and a cancel drops a queued request without running it at all -- which is what makes a seek cheap.
The branch added onnxruntime-web (runtime) and @playwright/test (test) without wiring them into renovate.json groups or the matching update:* scripts, so the repo's own tests/jest/dependencies.test.js failed two assertions. Group each dep and add it to its update:* script. The PocketTtsSynthesizer real-tokenizer suite defaulted its model path to a previous session's scratchpad, so it skipped for everyone. Point it at the same BookReaderDemo/pocket-tts-models the Playwright spec serves from (still overridable via POCKET_TTS_MODEL_DIR), so the coverage runs when weights are present.
…t speed Reworks the internetarchive#1580 audio reader prototype from testing feedback: - Buffering: default to pure PocketTTS with a pre-roll gate and worker warm-up, so the good voice plays straight through instead of the hybrid alternating to a WebSpeech preview mid-paragraph. - Text: chunk by whole sentence (the landmark split is now opt-in), and stitch sentences the OCR cut at page breaks so utterances begin and end on sentence boundaries; fuse words a page broke with a hyphen across chunks. - Speed: play PCM through an <audio> element so the rate control time-stretches (preserves pitch) rather than resampling (chipmunk). - Voices: clone kyutai/tts-voices by reference clip; Stuart Bell is the default, with a header picker for Bill, Peter and Caro. - Controls: page/sentence transport (<< < > >>), a draggable page scrubber plus a read-only real-time word-progress bar, and a read-along panel. Tests updated and added across player, segmentation and source; suite green.
- Dark-mode toggle in the header (🌙/☀️), remembered in localStorage and applied by flipping the palette tokens on the host. - Move the voice picker and playback speed into a tucked-away overflow (⋯) menu, decluttering the header. - Sleep timer (🕐): stop playback automatically after a chosen number of pages, with a live countdown badge.
A 👏 control to the right of the transport marks the current spot. Marks are kept per book in localStorage and drawn as faint ticks on both the page scrubber (across the whole book) and the real-time bar (this page), so a patron can see and jump back to the parts they liked. Pressing again at a marked spot removes it.
Clapping is no longer a binary toggle: each press bumps a per-spot count, pops the 👏 glyph, and floats a "+N". The scrubber ticks grow taller and more opaque the more a spot has been clapped, so popular passages stand out. Counts persist per book in localStorage (older single-mark entries read as one clap).
- Claps: no count shown; the glyph pops harder and a clap bursts upward on each press. The control defaults to its old hover contrast (was too faint). - Page bar: clicking jumps to that page, and the knob now holds the clicked spot through the async jump instead of snapping back then forward. The click→page mapping is extracted and unit-tested. - Dark mode: tokenize the highlight, scrubber rails and hairlines so the read-along highlight stays legible (was yellow-on-white) and the bars and rules read as subtle rather than glaring. - Speed: moved out of the overflow menu to the left of the transport (mirrors the clap on the right), since it shapes the whole reading experience. - Read-along: show a small cover beside the title, and give the text panel a fixed height so toggling it or a sentence changing length can't shift the controls. - Back control uses the Internet Archive pantheon mark, not a bare arrow.
An active header toggle filled with the near-black control colour, which read as a black box behind the emoji. Use a soft tinted pill with a stronger border and full-contrast glyph instead.
…ding After the model bytes download, there is still work -- building the ONNX sessions, encoding the voice, a warm-up pass -- during which the progress bar sat parked at 100% and looked stalled. Add a spinner that runs the whole time, switch the label to "Preparing voice…" once the bytes are in, and sweep the bar indeterminately rather than leaving it full.
A very long sentence became one very long utterance (and highlight). Now a
sentence past a length threshold is broken at a clause boundary -- a comma
before a coordinating conjunction (", and", ", but", ...), or a semicolon or
colon. It splits once, then re-checks each half and splits again only if still
too long; a cut that would leave a piece too short is skipped for another (or
left whole); and the most balanced cut is taken so we split as little as
possible. Covered by unit tests.
In read-along (and the reading panel), bring the highlighted sentence back into view in its fixed-height panel whenever the segment being read changes, so the highlight never scrolls off as playback moves forward.
.skip:hover hardcoded color: #000, which vanished against the dark background after a click left the pointer hovering. Use a theme-aware --ink-strong token (black in light, white in dark) instead.
4214b41 to
8c6bc23
Compare
Draft — prototype for discussion, not for merge. Opened by
openlibrary-boton behalf of @mekarpeles' exploration of #1580.Refs #1580.
What this is
A working local prototype of the audio-first reading mode: load a URL, get cover + table of contents + playback controls (no page-image view), press play, and a real archive.org book is read aloud client-side from its OCR text.
Demo:
/BookReaderDemo/demo-audioreader.html?ocaid=theworksofplato01platialaVerified against
theworksofplato01platiala— public, English, 524 leaves, 18 resolvable TOC entries from Open Library.What works
speechSynthesis, sourced fromBookReaderGetTextWrapper.php— the same endpoint the existing read-aloud plugin uses.Shape, and the questions it raises
src/audioreader/is a standalone bundle (250KB) with its own webpack entry. It never bootsBookReader.js, so no plugin registry, no page model, no jQuery, no bergamot. It reuses the read-aloud plugin'sPageChunkparser andWebTTSSound(keeping its hard-won Chrome/Firefox/Safari speech workarounds) via ajqueryShim, so the reuse costs nothing.That is deliberate but not a recommendation: whether audio mode should be a lean standalone bundle or a BookReader mode that disables plugins is exactly the productionization decision this prototype does not settle. Today plugins are wired at the webpack entry level, so "don't load translate" means a different bundle, not a runtime flag.
PocketTTS — synthesis works, browser integration pending
The reference runtime is Python, so this is a port: text conditioner →
flow_lm_mainper frame →flow_lm_flowintegration → mimi decoder, threading all 18 flow and 56 mimi state tensors explicitly.PocketTtsSynthesizeris runtime-agnostic, so the same code runs underonnxruntime-webin a Worker and under Node.Verified against the real int8 weights (
node scripts/verify-pocket-tts.mjs <modelDir>): 5.28s of audio for an 18-word sentence of real Plato OCR — 3.4 words/sec — at RTFx 1.7x on a single wasm thread. Since I cannot listen to it, I compared its acoustics to the human reference clip: spectral centroid 1560Hz vs 1620Hz, 98.9% vs 99.4% of energy below 4kHz, envelope dynamics 27× vs 21×, RMS 0.030 vs 0.029. That profile is speech, not noise. Intelligibility by ear is unverified.Two pieces of unexpected work:
ModelProtoand there is no JS sentencepiece.SentencePieceUnigramparses the protobuf and implements Viterbi with byte fallback, validated token-for-token againstsentencepiece0.2.2 (68 tests: ASCII, accents, Japanese, emoji, tabs, decimals, abbreviations). Worth knowing if anyone else touches this: the 256 byte pieces carry score0.0, higher than every real piece's log probability, so they must be kept out of the lattice or the tokenizer spells everything out byte by byte.kyutai/pocket-tts(fetchingalba.safetensorsreturns 401). The voice is therefore cloned from ungated reference audio through the mimi encoder. Worth deciding deliberately, since "run PocketTTS client-side" implies either cloning from a clip we own or an arrangement with kyutai.It runs in the browser. Synthesis is in a Web Worker (per-frame ONNX on the main thread would stall the transport controls), the ~146MB bundle is cached in the Cache API so it downloads once rather than per page load, and the UI shows the download rather than sitting silent.
build-ortcopies onnxruntime's wasm intoBookReader/ort/, following thebuild-bergamotprecedent.The hybrid preview —
?engine=hybridThe arrangement the issue suggests, and the answer to PocketTTS's one real weakness: at ~RTFx 1.7 on one wasm thread, a paragraph the buffer has not reached costs seconds of silence.
The rule falls out of when each decision is made. At synthesis time, PocketTTS gets a short grace period — once the lookahead is warm it wins easily, because synthesis started paragraphs ago, so no preview is produced at all. If the grace expires, hand back a WebSpeech preview and leave PocketTTS running. At play time, check again: if PocketTTS landed in between, play that instead. Reading straight through is high quality; a cold seek speaks immediately and recovers quality from the next segment. Late PocketTTS audio is still cached for a seek back.
Verified in a browser:
{quality: 1, preview: 1, upgraded: 1}— first segment previewed, then PocketTTS took over with its samples reaching the device.Tests
tests/jest/audioreader/covering segmentation, paragraph walking, queue ordering/abort/eviction, the player state machine, the tokenizer against real sentencepiece, the resampler, and the generation loop (with stand-in ONNX sessions, so the suite needs no model weights).tests/playwright/against the live archive.org endpoints, with screenshots. Playwright is new to this repo (the committed e2e suite is testcafe) and is additive.One environment caveat, stated plainly: Chromium under Playwright on this machine has no audio device, so its AudioContext clock does not advance (0.005s per 2 real seconds) and
speechSynthesisnever firesend. Audio starting is verifiable — utterancestartevents with a real voice, and for the PCM path a captured sample count and non-zero peak amplitude — but a sound finishing is not observable in-browser here. Continuous advancement driven by genuine sound completion is covered by the jest suite instead. This also motivated a real fix:PcmAudioOutputnow has a duration-based watchdog so a context that stops rendering (backgrounded mobile tab, device removed) cannot strand the reader on one segment, and it counts those completions separately so they are never mistaken for playback.Not done / out of scope
Everything #1580 asks for is now built: the minimal view, real OCR read aloud, progressive chunk rendering, the ≤5-paragraph buffer, the spinner, seek throttling, PocketTTS client-side in WASM, and the hybrid preview. Deliberately out of scope: the accessibility word-window mode (the issue calls it future work), mobile packaging, cross-browser testing, production polish.
SyntheticPcmEnginerenders measurable tones, not speech — it exists to verify the pipeline and to exercise the PCM path. Model weights are gitignored and fetched at runtime.