[zhangzhenghao]

Summary

• Add native GGML EmbeddingGemma 2 text and audio embeddings for the published Unsloth GGUFs, returning normalized 768D vectors in the joint embedding space.
• Text uses the main gemma-embedding2 model directly. Audio additionally loads the matching mmproj and runs the 16 kHz Gemma4 log-mel frontend, full Conformer/audio projector and shared bidirectional text backbone. Audio is not transcribed to text.
• Preserve F32 activations with F16/BF16/mixed quantized storage, Gemma4 UTF-8 BPE, local/global GQA, RMS/RoPE, merged-input PLE, projection, mean pooling and L2 normalization.
• Add --mmproj and backward-compatible optional projector loading in C++, Python, Rust and Node. Existing constructors and existing audio models remain supported. The HTTP endpoint remains text-only.
• No encode_image. Ignore the mixed mmproj's unused vision tensors, but reject unrecognized audio/projector tensors. Conversion/quantization presets retain their existing scope.

Usage

./build/clap embedding --model embeddinggemma-2-UD-Q4_K_XL.gguf \
  --mmproj mmproj-Q8_0.gguf --audio recording.wav

Audio is downmixed/resampled through the existing audio module. Use the first 30 seconds, matching the published default processor; this yields up to 750 soft tokens, not the processor estimator's stale 280-token cap. Frame and crop-offset APIs are not supported for this joint encoder.

Verification

Current implementation commit: 4309c925bcaecd9d5d5ba6be2c19362ee559b441 (rebased onto current main; only the four READMEs differ from verified 5bd9137, all implementation/build/test files are byte-identical).

• Static build: 34/34 CTests passed.

• Shared build: 36/36 CTests passed, including 799 optional pinned published-tokenizer cases.

• Independent stdlib scalar/DFT audio oracle: F32/F16 storage, SSCP layout, clipped linears, causal relative attention, convolution norms and optional projector RMS weights. Official NumPy frontend comparison: maximum error 4.77e-7.

• Generated end-to-end F32/F16 audio-plus-text scalar oracle verifies public waveform APIs, automatic resampling and WAV CLI output.

• Regressions cover missing projectors, unknown audio/projector tensors, oversized integer metadata, nonfinite/short inputs, mel's additive floor, frame/token boundaries, 30-second truncation equivalence and 750 soft tokens.

• Published Q4 main model plus Q8 mmproj executed on two distinct one-second PCM16 waveforms: finite normalized 768D embeddings, mutual cosine 0.93575437 (smoke evidence, not a quality benchmark).

• Independent review approved exact final tree 224a6fe0b1feff8e2fb2716e6e0f1ac46c861cee; whitespace/security scans passed.

• Prior text validation against pinned unmodified llama.cpp: six inputs including Chinese, retrieval prefixes and 602 tokens; same stored weights decoded to F32 give minimum cosine 0.999999926, maximum absolute error 5.332e-5. Packed-quant comparison is reported separately in source documentation.

• Python installed sdist→wheel: 54 tests passed, plus 54 using tests from the sdist. Node: 9 runtime tests, TypeScript, packaging and standalone npm installation passed. Rust: 5 integration tests, 2 compile-fail doctests, formatting and Clippy passed.

• Installed Python wheel, standalone Node tarball and Rust external consumer each produced 768D embeddings from the published models and a real WAV, matching native output within 2.48e-7 maximum absolute error. Archives include the new audio sources; Linux linkage has no external clap/ggml/ICU/OpenMP dependency. Rust remains intentionally non-publishable (publish=false), not a standalone Cargo shipping claim.

• All 16 remote CI jobs passed on 4309c925bcaecd9d5d5ba6be2c19362ee559b441: native, Python, Rust and Node workflows, covering their configured Linux/macOS/Windows/ARM64 matrix entries. PR is mergeable; not merged.

• Published-weight audio validation is complete, stagewise: on both exact PCM16 fixtures the official pinned NumPy frontend matches native within 4.7684e-7. Supplying those identical features to the unmodified pinned upstream audio tower and full text encoder gives minimum final cosine 0.9999996308 and maximum absolute component difference 1.35035e-4; soft-token minimum cosine is 0.9999980605.

• Precision caveat: upstream SSCP IM2COL packs activations to F16 even with F32-decoded weights. Callback evidence identified this as the first tower divergence (F32 patch error 0.0019512, zero against F16-rounded patches). This is not bit-exact parity, and not every residual was exhaustively attributed to packing. No native architecture correction was identified or needed.

• Frontend caveat: the pinned llama.cpp waveform frontend computes log(max(mel, 0.001)), unlike official/native log(mel + 0.001). Original waveform-level comparisons across those different frontends are not architecture-parity evidence; the reported independent tower/final comparison uses identical official-validated features. These checks are not an audio retrieval-quality benchmark.

Pinned sources, hashes and test contracts are in tests/models/EMBEDDINGGEMMA2_SOURCE.md. GPU inference and Google's original unquantized checkpoint were not exercised.