main (e680fc3) attributes more words to the wrong speaker than v0.2.0 in file transcription with diarization #81

In short

We build a Windows meeting transcriber on NeMo-Speech.cpp, pinned to v0.2.0. Before upgrading, we tested main at commit e680fc3. On the public AMI meeting ES2002a, main is slightly less accurate than v0.2.0:

• Speaker-attributed word error rate (cpWER): 0.7 points higher with Nemotron EN, 0.9 with Nemotron 3.5. cpWER also counts a word as wrong when it is given to the wrong speaker.
• Plain word error rate: up only 0.2 points. So the words are almost the same, and the speaker tags moved.
• On longer meetings: our own recordings, 324 minutes, show more dropped words, 3.3 percent more with EN and 1.0 percent more with 3.5.

A copy of v0.2.0 that we built ourselves, with the same compiler and CUDA, matches your release. So the difference comes from main's code, not from our build.

How to reproduce

1. Get ES2002a.Mix-Headset.wav from the AMI corpus, 16 kHz mono, and the AMI manual transcript.
2. Run the same command on the v0.2.0 release and on a build of main at e680fc3:

   nemo-speech transcribe ES2002a.Mix-Headset.wav --model <asr.gguf> --stream --endpointing --asr.streaming.rnnt_right_context 13 --diarize --diar-model Nemotron-3-Diarization.q8_0.gguf --asr.diar.preset v3-offline --device cuda:0 --json --output out.json
   

3. Score both outputs against the manual transcript with meeteval (cpWER) over the annotated span.

Models: the published q8_0 GGUFs of nvidia/nemotron-speech-streaming-en-0.6b (ebe59e5), nvidia/nemotron-3.5-asr-streaming-0.6b (1c8deae) and nvidia/Nemotron-3-Diarization (f667ed7). Machine: Windows 11, RTX 3080 10 GB, CUDA 12.8, MSVC 2022 17.14. Main was built locally for arch 86.

Results on AMI ES2002a (21 minutes, public)

Deleted words stay flat on AMI. The extra cpWER errors are substitutions: 96 rising to 117 for EN, and 113 to 142 for 3.5. That is the pattern of words landing on the wrong speaker.

Results on longer meetings (324 minutes: six private Zoom recordings plus AMI)

These are scored against Zoom's own transcripts, which are machine output, not ground truth. We cannot share the audio, so these are totals.

Repeat runs on each runtime give word-for-word identical output, so these differences are not run-to-run noise. Our own v0.2.0 build differs from the release by 8 to 61 word edits per recording. Main differs from it by 118 to 289 with EN and 135 to 474 with 3.5.

Our guess (not verified)

#46 changes how the first chunk of a stream is aligned. That shifts every later chunk boundary in a file by a few frames. With --endpointing --diarize, this may move where words near a speaker change get their tag. That would fit AMI's pattern: speaker errors up, deletions flat. We have not bisected, so another commit after v0.2.0 could be the cause.

Questions

1. Is this change in speaker tagging expected after fix(asr): correct streaming startup and Q8 pointwise precision #46? If so, is there a setting that keeps v0.2.0's behaviour for file jobs?
2. Would a bisect help? We can run AMI for any commits you name.
3. Is a release planned with fix(diar): keep frames for words not yet tagged #70, the long-file tagging fix for transcribe --diarize assigns wrong speaker tags to the first ~10 min of a 21 min file (standalone diarize on the same file is correct) #67? We want that fix, but would not upgrade while this regression stands.

We can share per-recording tables and our AMI scoring scripts.