diff --git a/README.md b/README.md index 812aad1..47d6779 100644 --- a/README.md +++ b/README.md @@ -186,7 +186,7 @@ which provides the `embedding`, `serve` and `quantize` subcommands. | Path | Responsibility | | --- | --- | -| `include/clap.h` | Public, dependency-light C++ API (`clap::Model`, `decode_audio`, `resample_audio`, `log_mel`) | +| `include/clap.h` | Public, dependency-light C++ API (`clap::CLAP`, `decode_audio`, `resample_audio`, `log_mel`) | | `src/model.h`, `src/model.cpp` | Internal architecture interface, dispatch and public API implementation | | `src/model-loader.*` | Shared GGUF metadata, mapping lifetime and tensor validation | | `src/model-ops.*` | Shared ggml matrix operations (CPU/optional Vulkan) and CPU normalization | @@ -385,7 +385,7 @@ mono decoding/resampling to 24 kHz and MERT's whole-recording normalization, five-second windows and all-layer pooling described below. No external MERT model or precomputed feature input is accepted. The public feature-input API and its CLI flag have been removed; old feature-only GGUFs must be reconverted. -`Model::audio(pcm, rate)` is the C++ waveform entry point. +`CLAP::encode_audio(pcm, rate)` is the C++ waveform entry point. The bundle's string metadata contract is: @@ -447,13 +447,13 @@ is required for inference. Two different outputs are intentionally exposed: -- `--frames` / `Model::audio_frames(pcm, rate)` normalize the complete supplied +- `--frames` / `CLAP::audio_frames(pcm, rate)` normalize the complete supplied waveform and return the **last transformer layer**, in time-major order. The published model has 768 channels and frame count `floor((samples - 400) / 320) + 1` for at least 400 samples at 24 kHz. - Without either output flag, `Model::audio()` / the CLI mean-pool these final + Without either output flag, `CLAP::encode_audio()` / the CLI mean-pool these final frames into a non-normalized 768-vector. This is **not** a CLaMP3 input vector. -- `--clamp3-features` / `Model::clamp3_features(pcm, rate)` implement the official +- `--clamp3-features` / `CLAP::clamp3_features(pcm, rate)` implement the official CLaMP3 `preprocessing/audio/extract_mert.py --mean_features` contract at [`sanderwood/clamp3@9016d2b0c8d12d1aa79c2e0ab201e6822bdc83a8`](https://github.com/sanderwood/clamp3/tree/9016d2b0c8d12d1aa79c2e0ab201e6822bdc83a8/preprocessing/audio): mix to mono and resample to 24 kHz, normalize the **whole recording** once as @@ -469,9 +469,9 @@ MERT inspection APIs remain available; CLaMP3 feature encoding itself is private For C++ end-to-end inference: ```cpp -clap::Model clamp3("clamp3-f16.gguf"); +clap::CLAP clamp3("clamp3-f16.gguf"); auto pcm = clap::decode_audio("music.flac"); -auto embedding = clamp3.audio(pcm.samples, pcm.sample_rate); +auto embedding = clamp3.encode_audio(pcm.samples, pcm.sample_rate); ``` CLaMP3 waveform inference includes the leading/trailing zero vectors, chunking @@ -551,10 +551,10 @@ followed by 12 width-1024 Wav2Vec2-Conformer layers. In this architecture RoPE i applied **before Q/K projections**, not after as in MERT2; feed-forward and convolution branches use SiLU. Encoder final LayerNorm is included. -`Model::audio_frames()` and `--frames` return the final unnormalized sequence; +`CLAP::audio_frames()` and `--frames` return the final unnormalized sequence; for `N` native-rate samples its frame count is `ceil(floor(N / 240) / 4)` (about 25 Hz), not MERT2's floor rule. At least 1,025 samples are required. -`Model::audio()` and the default CLI output mean-pool all returned frames into +`CLAP::encode_audio()` and the default CLI output mean-pool all returned frames into one 1,024-vector, without L2 normalization. Pooling is a convenience over MuQ features, **not** a MuQ-MuLan text/audio projection. Text, precomputed features, batching, training/RVQ codes, and intermediate-layer selection are unsupported. @@ -615,7 +615,7 @@ quantized-storage rejection without requiring ML packages. ### MuQ-MuLan-large (native joint audio/text embeddings) `OpenMuQ/MuQ-MuLan-large` has native, single-item text and raw-waveform -inference through `clap::Model` and `clap embedding`. Published embeddings have 512 +inference through `clap::CLAP` and `clap embedding`. Published embeddings have 512 components. No Python, model source execution, or downloads occur at inference. The model revision is `2e01c796b71dca71b45251384c04cd7b237c9020`; the reviewed [official source](https://github.com/tencent-ailab/MuQ/tree/28847ea50cd31ac4b8b6a7dacc051ad7d1c7606a) @@ -661,10 +661,10 @@ artifacts/build/clap embedding --model artifacts/muq-mulan-f16.gguf \ ``` ```cpp -clap::Model model("artifacts/muq-mulan-f16.gguf", 2); -auto text = model.text("hopeful piano music"); +clap::CLAP model("artifacts/muq-mulan-f16.gguf", 2); +auto text = model.encode_text("hopeful piano music"); auto audio = clap::decode_audio("music.flac"); -auto music = model.audio(audio.samples, audio.sample_rate); +auto music = model.encode_audio(audio.samples, audio.sample_rate); // dot(music, text) is the official similarity operation. ``` @@ -690,8 +690,8 @@ additional padding mask for the single-item API, and no final adapter norm. Mean pooling includes special tokens, then the learned 768→512 affine projection and L2 normalization apply. Empty text is valid. Over-context text is rejected, rather than truncated or chunked; the published XLM-R table allows -512 tokenizer IDs including specials. `Model::tokenize(text, max_length)` keeps -its existing explicit truncation API; `Model::text(string)` tokenizes the full +512 tokenizer IDs including specials. `CLAP::tokenize(text, max_length)` keeps +its existing explicit truncation API; `CLAP::encode_text(string)` tokenizes the full input and checks the context. Batch inference and other backbone/use-layer variants are unsupported. @@ -784,8 +784,8 @@ rotary Conformer blocks. No Python or runtime network access is required. Without `--frames`, the CLI mean-pools all final frames and prints one 1,024-dimensional recording embedding, matching the official model-card example. `--frames` instead prints an object with `frames`, `dimension`, and a flat -time-major `values` array. In C++, `Model::audio_frames()` exposes the same -sequence as `clap::AudioFrames`; `Model::audio()` returns its mean. At published +time-major `values` array. In C++, `CLAP::audio_frames()` exposes the same +sequence as `clap::AudioFrames`; `CLAP::encode_audio()` returns its mean. At published settings the frame count is `floor(input_samples / 960)` (25 Hz). FullSong means the complete supplied waveform is processed. There is no implicit diff --git a/include/clap.h b/include/clap.h index d49a025..d9c36b3 100644 --- a/include/clap.h +++ b/include/clap.h @@ -48,30 +48,30 @@ std::vector log_mel(const std::vector &pcm, int sample_rate = 4800 // CLAP returns unit-normalized projections. CLaMP3 returns its native unnormalized // pooled projections. Its mandatory bundled MERT-v1-95M frontend handles // waveform preprocessing internally; no separate feature-input API is exposed. -// MERT-v1 audio_frames() exposes final-layer frames; audio() mean-pools them. +// MERT-v1 audio_frames() exposes final-layer frames; encode_audio() mean-pools them. // Neither is the all-layer CLaMP3 representation: use clamp3_features() for that. -// MERT-v2-FullSong and MuQ audio() mean-pool the complete input's final frames; +// MERT-v2-FullSong and MuQ encode_audio() mean-pool the complete input's final frames; // audio_frames() returns the unpooled sequence (approximately 25 Hz for published weights). -// MuQ-MuLan text() returns a single L2-normalized joint embedding. audio() uses +// MuQ-MuLan encode_text() returns a single L2-normalized joint embedding. encode_audio() uses // successive 10 s clips, one prefix slice for a tail, per-clip normalization and // an unweighted average without final normalization. No amplitude normalization. // MuLan supports neither frame/features APIs nor crop offsets; published width // is 512. Its text string API rejects over-context input without truncation. -class Model { +class CLAP { public: - explicit Model(const std::string &gguf_path, int threads = 0); - ~Model(); - Model(Model &&) noexcept; - Model &operator=(Model &&) noexcept; - Model(const Model &) = delete; - Model &operator=(const Model &) = delete; + explicit CLAP(const std::string &gguf_path, int threads = 0); + ~CLAP(); + CLAP(CLAP &&) noexcept; + CLAP &operator=(CLAP &&) noexcept; + CLAP(const CLAP &) = delete; + CLAP &operator=(const CLAP &) = delete; std::vector tokenize(const std::string &utf8, std::size_t max_length = 512) const; - std::vector text(const std::vector &token_ids); - std::vector text(const std::string &utf8); + std::vector encode_text(const std::vector &token_ids); + std::vector encode_text(const std::string &utf8); // Mono waveform input; automatically resampled for the model. // CLaMP3 GGUFs must contain the bundled MERT-v1-95M frontend. - std::vector audio(const std::vector &pcm, int sample_rate = 48000, + std::vector encode_audio(const std::vector &pcm, int sample_rate = 48000, std::size_t crop_offset = 0); AudioFrames audio_frames(const std::vector &pcm, int sample_rate = 48000); // MERT-v1 CLaMP3 --mean_features preprocessing: normalize the entire mono diff --git a/python/README.md b/python/README.md index abdb957..c74711d 100644 --- a/python/README.md +++ b/python/README.md @@ -21,30 +21,29 @@ Convert a supported checkpoint to GGUF with the repository's conversion tools, then load it once and reuse it: ```python -from clap_cpp import Model, decode_audio +from clap_cpp import CLAP -with Model("model.gguf", threads=4) as model: - text_embedding = model.text("A dog barking") - pcm, sample_rate = decode_audio("dog.wav") - audio_embedding = model.audio(pcm, sample_rate=sample_rate) +with CLAP("model.gguf", threads=4) as model: + text_embedding = model.encode_text("A dog barking") + audio_embedding = model.encode_audio("dog.wav") print(model.projection_dimension, model.audio_sample_rate) ``` -`Model` uses the same GGUF architecture dispatch as the native library (CLAP, +`CLAP` uses the same GGUF architecture dispatch as the native library (CLAP, CLaMP3, MERT-v1, MERT-v2, MuQ and MuQ-MuLan). An operation unsupported by a particular architecture raises a native error; for example audio-only encoders cannot embed text. CLaMP3 requires a GGUF with its bundled MERT frontend. ## API -- `Model(path, threads=0)`: `str`/`pathlib.Path`; zero selects the native thread +- `CLAP(path, threads=0)`: `str`/`pathlib.Path`; zero selects the native thread default. `close()` is idempotent and the context manager closes on exit. -- `model.text(text_or_tokens)`: one UTF-8 string or token-ID sequence to a +- `model.encode_text(text_or_tokens)`: one UTF-8 string or token-ID sequence to a one-dimensional float32 NumPy embedding. - `model.tokenize(text, max_length=512)`: native tokenizer returning a list of integer IDs. Its limit must be compatible with the loaded model; strings may - be truncated by the tokenizer. Prefer `text(str)` for normal use. -- `model.audio(pcm, sample_rate=48000, crop_offset=0)`: mono waveform to a float32 + be truncated by the tokenizer. Prefer `encode_text(str)` for normal use. +- `model.encode_audio(pcm, sample_rate=48000, crop_offset=0)`: mono waveform to a float32 embedding. Automatic model-rate resampling; crop offset uses the native model-rate sample units and is not supported by every architecture. - `model.audio_frames(pcm, sample_rate=48000)`: native frame representations as a @@ -52,10 +51,10 @@ cannot embed text. CLaMP3 requires a GGUF with its bundled MERT frontend. - `model.clamp3_features(pcm, sample_rate=24000)`: MERT-v1 CLaMP3 preprocessing features as a `[windows, dimension]` float32 array (at least one second). - `model.projection_dimension`, `model.audio_sample_rate`: read-only integers. -- `decode_audio(path)`: `(pcm, sample_rate)`; decodes WAV, MP3 or FLAC, averages - channels to mono, preserves sample rate. -- `resample_audio(pcm, source_rate, target_rate=48000)`: mono float32 array using - the native resampler. +- `model.encode_audio(path, crop_offset=0)`: `str`/`pathlib.Path` to a WAV, MP3 + or FLAC file. Decodes internally, averages channels to mono and automatically + resamples from the file's sample rate to the model's rate. Crop offset has the + same model-rate sample units as the PCM overload. PCM accepts one-dimensional numeric arrays or sequences. Noncontiguous arrays and other numeric dtypes are converted to contiguous float32; the binding copies diff --git a/python/bindings.cpp b/python/bindings.cpp index 148b595..306bbf0 100644 --- a/python/bindings.cpp +++ b/python/bindings.cpp @@ -51,19 +51,19 @@ py::array_t matrix(const clap::AudioFrames &frames) { // Always release the GIL BEFORE taking the model mutex; release the mutex // BEFORE reacquiring the GIL. close(), properties and inference share this order. -class Model { +class CLAP { public: - Model(const std::filesystem::path &path, int threads) { + CLAP(const std::filesystem::path &path, int threads) { if (threads < 0) throw std::invalid_argument("threads must be nonnegative"); py::gil_scoped_release release; - model_ = std::make_unique(path.u8string(), threads); + model_ = std::make_unique(path.u8string(), threads); } template auto call(F &&function) { py::gil_scoped_release release; std::lock_guard lock(mutex_); - if (!model_) throw std::runtime_error("Model is closed"); + if (!model_) throw std::runtime_error("CLAP is closed"); return function(*model_); } @@ -74,7 +74,7 @@ class Model { } private: - std::unique_ptr model_; + std::unique_ptr model_; std::mutex mutex_; }; @@ -82,64 +82,53 @@ class Model { PYBIND11_MODULE(_clap_cpp, module) { module.doc() = "Native CPU inference for clap.cpp"; - py::class_(module, "Model") + py::class_(module, "CLAP") .def(py::init(), py::arg("path"), py::arg("threads") = 0) - .def_property_readonly("projection_dimension", [](Model &self) { - return self.call([](clap::Model &model) { return model.projection_dimension(); }); + .def_property_readonly("projection_dimension", [](CLAP &self) { + return self.call([](clap::CLAP &model) { return model.projection_dimension(); }); }) - .def_property_readonly("audio_sample_rate", [](Model &self) { - return self.call([](clap::Model &model) { return model.audio_sample_rate(); }); + .def_property_readonly("audio_sample_rate", [](CLAP &self) { + return self.call([](clap::CLAP &model) { return model.audio_sample_rate(); }); }) - .def("tokenize", [](Model &self, const std::string &text, py::ssize_t max_length) { + .def("tokenize", [](CLAP &self, const std::string &text, py::ssize_t max_length) { if (max_length <= 0) throw std::invalid_argument("max_length must be positive"); - return self.call([&](clap::Model &model) { return model.tokenize(text, max_length); }); + return self.call([&](clap::CLAP &model) { return model.tokenize(text, max_length); }); }, py::arg("text"), py::arg("max_length") = 512) - .def("text", [](Model &self, const std::string &text) { - return array(self.call([&](clap::Model &model) { return model.text(text); })); + .def("encode_text", [](CLAP &self, const std::string &text) { + return array(self.call([&](clap::CLAP &model) { return model.encode_text(text); })); }, py::arg("text")) - .def("text", [](Model &self, const std::vector &tokens) { - return array(self.call([&](clap::Model &model) { return model.text(tokens); })); + .def("encode_text", [](CLAP &self, const std::vector &tokens) { + return array(self.call([&](clap::CLAP &model) { return model.encode_text(tokens); })); }, py::arg("tokens")) - .def("audio", [](Model &self, const PCM &pcm, int sample_rate, py::ssize_t crop_offset) { + .def("encode_audio", [](CLAP &self, const PCM &pcm, int sample_rate, py::ssize_t crop_offset) { validate_rate(sample_rate); if (crop_offset < 0) throw std::invalid_argument("crop_offset must be nonnegative"); auto input = samples(pcm); - return array(self.call([&](clap::Model &model) { return model.audio(input, sample_rate, crop_offset); })); + return array(self.call([&](clap::CLAP &model) { return model.encode_audio(input, sample_rate, crop_offset); })); }, py::arg("pcm"), py::arg("sample_rate") = 48000, py::arg("crop_offset") = 0) - .def("audio_frames", [](Model &self, const PCM &pcm, int sample_rate) { + .def("encode_audio", [](CLAP &self, const std::filesystem::path &path, py::ssize_t crop_offset) { + if (crop_offset < 0) throw std::invalid_argument("crop_offset must be nonnegative"); + const auto filename = path.u8string(); + return array(self.call([&](clap::CLAP &model) { + const auto decoded = clap::decode_audio(filename); + return model.encode_audio(decoded.samples, decoded.sample_rate, crop_offset); + })); + }, py::arg("path"), py::arg("crop_offset") = 0) + .def("audio_frames", [](CLAP &self, const PCM &pcm, int sample_rate) { validate_rate(sample_rate); auto input = samples(pcm); - return matrix(self.call([&](clap::Model &model) { return model.audio_frames(input, sample_rate); })); + return matrix(self.call([&](clap::CLAP &model) { return model.audio_frames(input, sample_rate); })); }, py::arg("pcm"), py::arg("sample_rate") = 48000) - .def("clamp3_features", [](Model &self, const PCM &pcm, int sample_rate) { + .def("clamp3_features", [](CLAP &self, const PCM &pcm, int sample_rate) { validate_rate(sample_rate); auto input = samples(pcm); - return matrix(self.call([&](clap::Model &model) { return model.clamp3_features(input, sample_rate); })); + return matrix(self.call([&](clap::CLAP &model) { return model.clamp3_features(input, sample_rate); })); }, py::arg("pcm"), py::arg("sample_rate") = 24000) - .def("close", &Model::close) - .def("__enter__", [](Model &self) -> Model & { - self.call([](clap::Model &) {}); + .def("close", &CLAP::close) + .def("__enter__", [](CLAP &self) -> CLAP & { + self.call([](clap::CLAP &) {}); return self; }, py::return_value_policy::reference_internal) - .def("__exit__", [](Model &self, py::object, py::object, py::object) { self.close(); }); + .def("__exit__", [](CLAP &self, py::object, py::object, py::object) { self.close(); }); - module.def("decode_audio", [](const std::filesystem::path &path) { - clap::Audio decoded; - { - py::gil_scoped_release release; - decoded = clap::decode_audio(path.u8string()); - } - return py::make_tuple(array(decoded.samples), decoded.sample_rate); - }, py::arg("path")); - module.def("resample_audio", [](const PCM &pcm, int source_rate, int target_rate) { - validate_rate(source_rate); - validate_rate(target_rate); - clap::Audio input{samples(pcm), source_rate}; - clap::Audio output; - { - py::gil_scoped_release release; - output = clap::resample_audio(input, target_rate); - } - return array(output.samples); - }, py::arg("pcm"), py::arg("source_rate"), py::arg("target_rate") = 48000); } diff --git a/python/clap_cpp/__init__.py b/python/clap_cpp/__init__.py index c856d55..f67fba6 100644 --- a/python/clap_cpp/__init__.py +++ b/python/clap_cpp/__init__.py @@ -1,5 +1,5 @@ """CPU audio/text embeddings backed by the native clap.cpp runtime.""" -from ._clap_cpp import Model, decode_audio, resample_audio +from ._clap_cpp import CLAP -__all__ = ["Model", "decode_audio", "resample_audio"] +__all__ = ["CLAP"] diff --git a/python/tests/test_bindings.py b/python/tests/test_bindings.py index fac456d..af7cd58 100644 --- a/python/tests/test_bindings.py +++ b/python/tests/test_bindings.py @@ -1,7 +1,6 @@ """CPU integration tests: no downloads, mocks, or optional model dependencies.""" from concurrent.futures import ThreadPoolExecutor import gc -import math import wave import numpy as np @@ -22,13 +21,13 @@ def model_paths(tmp_path_factory): @pytest.fixture def model(model_paths): - with clap_cpp.Model(model_paths[0], threads=2) as result: + with clap_cpp.CLAP(model_paths[0], threads=2) as result: yield result @pytest.fixture def mert(model_paths): - with clap_cpp.Model(str(model_paths[1]), threads=2) as result: + with clap_cpp.CLAP(str(model_paths[1]), threads=2) as result: yield result @@ -47,9 +46,12 @@ def assert_array(value, shape): def test_module_exports(): - assert callable(clap_cpp.Model) - assert callable(clap_cpp.decode_audio) - assert callable(clap_cpp.resample_audio) + assert clap_cpp.__all__ == ["CLAP"] + assert callable(clap_cpp.CLAP) + for name in ("Model", "decode_audio", "resample_audio"): + assert not hasattr(clap_cpp, name) + for name in ("text", "audio"): + assert not hasattr(clap_cpp.CLAP, name) @pytest.mark.parametrize("as_string", [False, True]) @@ -58,10 +60,10 @@ def test_model_loads_from_unicode_path(tmp_path, as_string): directory.mkdir() path = directory / "音声 modèle 🎶.gguf" clap_fixture(path) - with clap_cpp.Model(str(path) if as_string else path, threads=2) as model: + with clap_cpp.CLAP(str(path) if as_string else path, threads=2) as model: assert model.projection_dimension == 3 tokens = model.tokenize("hello") - np.testing.assert_allclose(model.text("hello"), text_reference(tokens), + np.testing.assert_allclose(model.encode_text("hello"), text_reference(tokens), rtol=2e-4, atol=2e-5) @@ -81,16 +83,16 @@ def test_text_matches_independent_numpy_oracle(model, text): assert isinstance(tokens, list) assert tokens and all(isinstance(token, int) for token in tokens) assert tokens[0] == 0 and tokens[-1] == 2 - actual = model.text(text) + actual = model.encode_text(text) assert_array(actual, (3,)) np.testing.assert_allclose(actual, text_reference(tokens), rtol=2e-4, atol=2e-5) - np.testing.assert_allclose(model.text(tokens), actual, rtol=1e-5, atol=1e-6) - np.testing.assert_array_equal(model.text(text), actual) + np.testing.assert_allclose(model.encode_text(tokens), actual, rtol=1e-5, atol=1e-6) + np.testing.assert_array_equal(model.encode_text(text), actual) assert np.linalg.norm(actual) == pytest.approx(1, abs=1e-5) def test_text_depends_on_input(model): - assert np.max(np.abs(model.text("hello") - model.text("world"))) > 1e-7 + assert np.max(np.abs(model.encode_text("hello") - model.encode_text("world"))) > 1e-7 def test_tokenizer_limit(model): @@ -99,26 +101,24 @@ def test_tokenizer_limit(model): def test_waveform_inference_and_explicit_resampling(model, pcm): # Exercise the actual 64-mel waveform frontend, not feature-only tensors. - actual = model.audio(pcm, sample_rate=24000) + actual = model.encode_audio(pcm, sample_rate=24000) assert_array(actual, (3,)) assert np.linalg.norm(actual) == pytest.approx(1, abs=1e-5) - np.testing.assert_array_equal(model.audio(pcm, sample_rate=24000), actual) - np.testing.assert_allclose(model.audio(pcm.tolist(), sample_rate=24000), actual, + np.testing.assert_array_equal(model.encode_audio(pcm, sample_rate=24000), actual) + np.testing.assert_allclose(model.encode_audio(pcm.tolist(), sample_rate=24000), actual, rtol=1e-5, atol=1e-6) - upsampled = clap_cpp.resample_audio(pcm, 24000, 48000) - np.testing.assert_allclose(model.audio(upsampled), actual, rtol=1e-5, atol=1e-6) # Non-contiguous float64 input must be converted safely, not read as F32. padded = np.empty((pcm.size, 2), dtype=np.float64) padded[:, 0] = pcm padded[:, 1] = 999 - np.testing.assert_allclose(model.audio(padded[:, 0], sample_rate=24000), actual, + np.testing.assert_allclose(model.encode_audio(padded[:, 0], sample_rate=24000), actual, rtol=1e-5, atol=1e-6) def test_mert_frames_and_mean_pooling(mert, pcm): frames = mert.audio_frames(pcm, sample_rate=24000) assert_array(frames, (75, 4)) - np.testing.assert_allclose(mert.audio(pcm, sample_rate=24000), frames.mean(axis=0), + np.testing.assert_allclose(mert.encode_audio(pcm, sample_rate=24000), frames.mean(axis=0), rtol=1e-5, atol=1e-6) np.testing.assert_array_equal(mert.audio_frames(pcm, sample_rate=24000), frames) @@ -132,101 +132,83 @@ def test_mert_clamp3_features(mert, pcm): assert_array(mert.clamp3_features(extended), (2, 4)) -@pytest.mark.parametrize("name", ["stereo input.wav", "音声 café 🎶/立体声 modèle 🎵.wav"]) -def test_decoded_stereo_wav_preserves_rate_and_averages_channels(tmp_path, name): - path = tmp_path / name - path.parent.mkdir(parents=True, exist_ok=True) - samples = np.array([[32767, -32768], [16384, 0], [-16384, 8192], [0, 0]], - dtype=" vectors; if (!options.audio_path.empty()) { if (options.crop_requested && model.audio_sample_rate() != 48000) { @@ -82,16 +82,16 @@ int run_embedding(const EmbeddingOptions &options) { std::cout << json{{"frames", sequence.frames}, {"dimension", sequence.dimension}, {"values", sequence.values}}.dump() << '\n'; } else { - vectors.push_back(model.audio(decoded.samples, decoded.sample_rate, options.crop_offset)); + vectors.push_back(model.encode_audio(decoded.samples, decoded.sample_rate, options.crop_offset)); } } else if (options.single_text) { - vectors.push_back(model.text(options.text)); + vectors.push_back(model.encode_text(options.text)); } else { // Preserve empty and trailing separator-delimited prompt/file segments. std::size_t start = 0; while (true) { const auto end = options.text.find(options.separator, start); - vectors.push_back(model.text(options.text.substr(start, end == std::string::npos + vectors.push_back(model.encode_text(options.text.substr(start, end == std::string::npos ? end : end - start))); if (end == std::string::npos) break; start = end + options.separator.size(); diff --git a/src/model.cpp b/src/model.cpp index 9af2bde..ce1c81b 100644 --- a/src/model.cpp +++ b/src/model.cpp @@ -7,7 +7,7 @@ namespace clap { -struct Model::Impl { +struct CLAP::Impl { // Declaration order ensures the architecture is destroyed before its mapping. detail::ModelLoader loader; std::unique_ptr architecture; @@ -35,26 +35,26 @@ struct Model::Impl { } }; -Model::Model(const std::string &path, int threads) : impl_(std::make_unique(path, threads)) {} -Model::~Model() = default; -Model::Model(Model &&) noexcept = default; -Model &Model::operator=(Model &&) noexcept = default; +CLAP::CLAP(const std::string &path, int threads) : impl_(std::make_unique(path, threads)) {} +CLAP::~CLAP() = default; +CLAP::CLAP(CLAP &&) noexcept = default; +CLAP &CLAP::operator=(CLAP &&) noexcept = default; -int Model::projection_dimension() const { return impl_->architecture->projection_dimension(); } -std::vector Model::tokenize(const std::string &utf8, size_t max_length) const { +int CLAP::projection_dimension() const { return impl_->architecture->projection_dimension(); } +std::vector CLAP::tokenize(const std::string &utf8, size_t max_length) const { if (impl_->vocabulary == nullptr) throw std::runtime_error("CLAP: this model has no text tokenizer"); return impl_->vocabulary->tokenize(utf8, max_length); } -std::vector Model::text(const std::string &utf8) { +std::vector CLAP::encode_text(const std::string &utf8) { if (impl_->vocabulary == nullptr) throw std::runtime_error("CLAP: this model has no text encoder"); - return text(tokenize(utf8, impl_->architecture->text_parameters().context_length)); + return encode_text(tokenize(utf8, impl_->architecture->text_parameters().context_length)); } -std::vector Model::text(const std::vector &ids) { +std::vector CLAP::encode_text(const std::vector &ids) { return impl_->architecture->text(ids); } -std::vector Model::audio(const std::vector &pcm, int sample_rate, size_t crop_offset) { +std::vector CLAP::encode_audio(const std::vector &pcm, int sample_rate, size_t crop_offset) { if (impl_->architecture->has_waveform_embeddings()) { if (sample_rate <= 0) throw std::runtime_error("CLAP: audio sample rate must be positive"); const int target = impl_->architecture->sample_rate(); @@ -74,7 +74,7 @@ std::vector Model::audio(const std::vector &pcm, int sample_rate, throw std::runtime_error("CLAP: this model has no waveform audio encoder"); } -AudioFrames Model::audio_frames(const std::vector &pcm, int sample_rate) { +AudioFrames CLAP::audio_frames(const std::vector &pcm, int sample_rate) { if (!(impl_->architecture->is_waveform_encoder())) throw std::runtime_error("CLAP: this model does not expose frame representations"); if (!(sample_rate > 0)) @@ -84,8 +84,8 @@ AudioFrames Model::audio_frames(const std::vector &pcm, int sample_rate) return impl_->architecture->audio_frames(resample_audio({pcm, sample_rate}, target).samples); } -int Model::audio_sample_rate() const { return impl_->architecture->sample_rate(); } -AudioFrames Model::clamp3_features(const std::vector &pcm, int sample_rate) { +int CLAP::audio_sample_rate() const { return impl_->architecture->sample_rate(); } +AudioFrames CLAP::clamp3_features(const std::vector &pcm, int sample_rate) { if (sample_rate <= 0) throw std::runtime_error("CLAP: audio sample rate must be positive"); if (sample_rate == 24000) return impl_->architecture->clamp3_features(pcm); return impl_->architecture->clamp3_features(resample_audio({pcm, sample_rate}, 24000).samples); diff --git a/src/models/models.h b/src/models/models.h index dbeeddd..76d00ae 100644 --- a/src/models/models.h +++ b/src/models/models.h @@ -8,7 +8,7 @@ namespace clap::detail { // Per-architecture declarations belong here, following llama.cpp/src/models. -// GGUF I/O, tokenizer ownership, generic tensor operations and the public Model +// GGUF I/O, tokenizer ownership, generic tensor operations and the public CLAP // facade are implemented in the core src/ files, not in this specialization. class ClapModel final : public ModelArchitecture { public: diff --git a/src/server.cpp b/src/server.cpp index bfd5a7d..76b4415 100644 --- a/src/server.cpp +++ b/src/server.cpp @@ -28,7 +28,7 @@ void invalid_request(httplib::Response &response, const std::string &message) { } void embeddings(const httplib::Request &request, httplib::Response &response, - clap::Model &model, const std::string &identifier, int dimensions, + clap::CLAP &model, const std::string &identifier, int dimensions, std::mutex &inference_mutex) { const auto body = json::parse(request.body, nullptr, false); if (!body.is_object() || !body.contains("input")) { @@ -72,13 +72,13 @@ void embeddings(const httplib::Request &request, httplib::Response &response, const std::lock_guard lock(inference_mutex); for (std::size_t index = 0; index < inputs.size(); ++index) { const auto &text = inputs[index].get_ref(); - const auto embedding = model.text(text); + const auto embedding = model.encode_text(text); if (embedding.size() != static_cast(dimensions)) throw std::runtime_error("Model returned an unexpected embedding dimension"); for (float value : embedding) if (!std::isfinite(value)) throw std::runtime_error("Model returned a non-finite embedding"); data.push_back({{"object", "embedding"}, {"index", index}, {"embedding", embedding}}); - // Model::text(string) chooses its architecture's full context. + // CLAP::encode_text(string) chooses its architecture's full context. // The public tokenize() default is 512 and can truncate or be // unsupported for smaller contexts. Omit usage rather than // report an inaccurate count in either situation. @@ -108,7 +108,7 @@ int run_server(const std::string &host, int port, const std::string &model_path, try { // Load once, before opening the listening socket. A bad model must fail // startup rather than leave a live server that can never serve requests. - clap::Model model(model_path, threads); + clap::CLAP model(model_path, threads); const auto identifier = std::filesystem::path(model_path).filename().string(); const int dimensions = model.projection_dimension(); if (dimensions <= 0) throw std::runtime_error("Model projection dimension must be positive"); diff --git a/tests/models/clamp3-bundled-test.cpp b/tests/models/clamp3-bundled-test.cpp index a64b9ad..b76b067 100644 --- a/tests/models/clamp3-bundled-test.cpp +++ b/tests/models/clamp3-bundled-test.cpp @@ -19,7 +19,7 @@ template static void rejects(F fn) { int main(int argc, char **argv) { try { if (argc!=4) throw std::runtime_error("bundled, feature-only, MERT models required"); - clap::Model bundled(argv[1],2), target(argv[2],2), mert(argv[3],2); + clap::CLAP bundled(argv[1],2), target(argv[2],2), mert(argv[3],2); if (bundled.audio_sample_rate()!=24000) throw std::runtime_error("bundled sample rate"); if (target.audio_sample_rate()!=24000) throw std::runtime_error("mandatory frontend sample rate"); std::vector pcm(48000); @@ -27,18 +27,18 @@ int main(int argc, char **argv) { const auto resampled=clap::resample_audio({pcm,48000},24000).samples; const auto features=mert.clamp3_features(resampled,24000); const auto expected=clap::detail::FeatureTestAccess::encode(argv[2],features.values,features.frames); - equal(bundled.audio(pcm,48000),expected); - equal(bundled.audio(resampled,24000),expected); + equal(bundled.encode_audio(pcm,48000),expected); + equal(bundled.encode_audio(resampled,24000),expected); equal(clap::detail::FeatureTestAccess::encode(argv[1],features.values,features.frames),expected); - equal(bundled.text("ab c"),target.text("ab c")); - rejects([&]{ bundled.audio({},24000); }); - rejects([&]{ bundled.audio(std::vector(23999),24000); }); - rejects([&]{ bundled.audio(pcm,0); }); - rejects([&]{ bundled.audio(pcm,24000,1); }); + equal(bundled.encode_text("ab c"),target.encode_text("ab c")); + rejects([&]{ bundled.encode_audio({},24000); }); + rejects([&]{ bundled.encode_audio(std::vector(23999),24000); }); + rejects([&]{ bundled.encode_audio(pcm,0); }); + rejects([&]{ bundled.encode_audio(pcm,24000,1); }); rejects([&]{ bundled.audio_frames(pcm,24000); }); rejects([&]{ bundled.clamp3_features(pcm,24000); }); pcm[0]=std::numeric_limits::quiet_NaN(); - rejects([&]{ bundled.audio(pcm,24000); }); + rejects([&]{ bundled.encode_audio(pcm,24000); }); std::cout<<"Bundled CLaMP3 native waveform/resampling/features/text/invalid-input passed\n"; return 0; } catch (const std::exception &e) { std::cerr<> fixture; - clap::Model model(argv[1], 2); + clap::CLAP model(argv[1], 2); require(model.projection_dimension() == fixture.at("projection_dim"), "projection dimension"); const float tolerance = fixture.value("embedding_tolerance", 8e-5f); for (const auto &test : fixture.at("tokenizer_cases")) { @@ -39,10 +39,10 @@ int main(int argc, char **argv) { require(ids == test.at("ids").get>(), "tokenizer mismatch: " + test.at("text").get()); } for (const auto &test : fixture.at("text_cases")) { - compare(model.text(test.at("ids").get>()), + compare(model.encode_text(test.at("ids").get>()), test.at("expected").get>(), tolerance, test.at("name")); if (test.contains("utf8")) - compare(model.text(test.at("utf8").get()), + compare(model.encode_text(test.at("utf8").get()), test.at("expected").get>(), tolerance, test.at("name")); } for (const auto &test : fixture.at("audio_cases")) diff --git a/tests/models/mert-model-test.cpp b/tests/models/mert-model-test.cpp index 0c4ccd5..344bb4d 100644 --- a/tests/models/mert-model-test.cpp +++ b/tests/models/mert-model-test.cpp @@ -44,16 +44,16 @@ void wav(const std::string &path, const clap::Audio &audio) { int main(int argc, char **argv) { try { require(argc == 3, "usage: mert-model-test model.gguf output-directory"); - clap::Model model(argv[1], 2); + clap::CLAP model(argv[1], 2); require(model.audio_sample_rate() == 24000, "wrong MERT sample rate"); - rejects([&] { model.text("unsupported"); }); - rejects([&] { model.text(std::vector{0}); }); + rejects([&] { model.encode_text("unsupported"); }); + rejects([&] { model.encode_text(std::vector{0}); }); rejects([&] { model.audio_frames({}); }); rejects([&] { model.audio_frames(std::vector(399), 24000); }); rejects([&] { model.clamp3_features(std::vector(23999), 24000); }); rejects([&] { model.clamp3_features(std::vector(24000), 0); }); rejects([&] { model.audio_frames(std::vector(24000), -1); }); - rejects([&] { model.audio(std::vector(24000), 24000, 1); }); + rejects([&] { model.encode_audio(std::vector(24000), 24000, 1); }); auto nonfinite = std::vector(24000); nonfinite[0] = std::numeric_limits::quiet_NaN(); rejects([&] { model.audio_frames(nonfinite, 24000); }); @@ -76,7 +76,7 @@ int main(int argc, char **argv) { for (int row = 0; row < frames.frames; ++row) for (int col = 0; col < frames.dimension; ++col) mean[col] += frames.values[static_cast(row)*frames.dimension+col] / frames.frames; - near(mean, model.audio(control.samples, 24000), 0); + near(mean, model.encode_audio(control.samples, 24000), 0); wav(std::string(argv[2]) + "/resampled-input.wav", input); wav(std::string(argv[2]) + "/resampled-control.wav", control); std::cout << "MERT public API: pooling, invalid inputs, native/resampled inference passed\n"; diff --git a/tests/models/mert2-model-test.cpp b/tests/models/mert2-model-test.cpp index 191075f..33ccc11 100644 --- a/tests/models/mert2-model-test.cpp +++ b/tests/models/mert2-model-test.cpp @@ -34,7 +34,7 @@ int main(int argc, char **argv) { nlohmann::json fixture; input >> fixture; - clap::Model model(argv[1], 2); + clap::CLAP model(argv[1], 2); require(model.audio_sample_rate() == fixture.at("sample_rate"), "MERT2 sample rate"); require(model.projection_dimension() == fixture.at("hidden_size"), "MERT2 hidden size"); const float tolerance = fixture.at("tolerance"); @@ -45,7 +45,7 @@ int main(int argc, char **argv) { require(output.dimension == fixture.at("hidden_size"), "MERT2 frame width"); compare(output.values, test.at("expected_frames").get>(), tolerance, test.at("name")); - compare(model.audio(waveform, fixture.at("sample_rate")), + compare(model.encode_audio(waveform, fixture.at("sample_rate")), test.at("expected_pooled").get>(), tolerance, test.at("name").get() + " pooled"); if (test.value("full_song", false)) { @@ -68,7 +68,7 @@ int main(int argc, char **argv) { catch (const std::exception &) { rejected = true; } require(rejected, "non-finite MERT2 waveform must be rejected"); rejected = false; - try { model.text("MERT2 has no text tower"); } + try { model.encode_text("MERT2 has no text tower"); } catch (const std::exception &) { rejected = true; } require(rejected, "MERT2 text input must be rejected"); std::cout << "Native MERT-v2-FullSong parity passed\n"; diff --git a/tests/models/model-test.cpp b/tests/models/model-test.cpp index b5674dc..1c65938 100644 --- a/tests/models/model-test.cpp +++ b/tests/models/model-test.cpp @@ -10,7 +10,7 @@ #include template struct has_feature_input : std::false_type {}; template struct has_feature_input> : std::true_type {}; -static_assert(!has_feature_input::value, "public feature interface must not return"); +static_assert(!has_feature_input::value, "public feature interface must not return"); static void require(bool value, const std::string &message) { if (!value) throw std::runtime_error(message); @@ -54,19 +54,19 @@ int main(int argc, char **argv) { try { clap::log_mel(long_clip, 48000, 4); } catch (const std::exception &) { rejected = true; } require(rejected, "out-of-range crop must be rejected"); rejected = false; - try { clap::Model missing("/this/path/does/not/exist.gguf"); } + try { clap::CLAP missing("/this/path/does/not/exist.gguf"); } catch (const std::exception &) { rejected = true; } require(rejected, "missing GGUF must be rejected"); if (argc == 1) return 0; require(argc == 3, "usage: clap-model-test [MODEL.gguf FIXTURE.json]"); std::ifstream input(argv[2]); nlohmann::json fixture; input >> fixture; - clap::Model model(argv[1], 2); + clap::CLAP model(argv[1], 2); const float embedding_tolerance = fixture.value("embedding_tolerance", 5e-5f); for (const auto &test : fixture.at("text_cases")) { const auto ids = model.tokenize(test.at("text").get()); require(ids == test.at("ids").get>(), "tokenizer mismatch: " + test.at("text").get()); - compare(model.text(test.at("padded_ids").get>()), + compare(model.encode_text(test.at("padded_ids").get>()), test.at("expected").get>(), embedding_tolerance, "text embedding"); } compare(clap::detail::FeatureTestAccess::encode(argv[1], fixture.at("features").get>(), fixture.at("frames")), @@ -75,7 +75,7 @@ int main(int argc, char **argv) { compare(clap::log_mel(fixture.at("waveform").get>()), fixture.at("mel").get>(), 3e-4f, "waveform log mel"); if (fixture.value("end_to_end", false)) - compare(model.audio(fixture.at("waveform").get>()), + compare(model.encode_audio(fixture.at("waveform").get>()), fixture.at("expected_audio").get>(), 5e-5f, "waveform embedding"); } std::cout << "Native CLAP parity passed\n"; diff --git a/tests/models/muq-model-test.cpp b/tests/models/muq-model-test.cpp index 2cf5ebb..b06f7c1 100644 --- a/tests/models/muq-model-test.cpp +++ b/tests/models/muq-model-test.cpp @@ -26,7 +26,7 @@ int main(int argc, char **argv) { require(argc == 3, "usage: muq-model-test MODEL.gguf FIXTURE.json"); nlohmann::json fixture; std::ifstream input(argv[2]); input >> fixture; - clap::Model model(argv[1], 2); + clap::CLAP model(argv[1], 2); require(model.audio_sample_rate() == 24000, "MuQ rate"); require(model.projection_dimension() == fixture.at("hidden_size"), "MuQ width"); const float tolerance = fixture.at("tolerance"); @@ -40,7 +40,7 @@ int main(int argc, char **argv) { // noise-floor sensitivity and catches unintended activation packing. const float bound = test.at("name") == "silence" ? std::min(tolerance, 1e-5f) : tolerance; compare(sequence.values, test.at("expected_frames").get>(), bound, test.at("name")); - compare(model.audio(wave, 24000), test.at("expected_pooled").get>(), bound, "MuQ pooled"); + compare(model.encode_audio(wave, 24000), test.at("expected_pooled").get>(), bound, "MuQ pooled"); } const auto wave = fixture.at("waveform_cases").at(1).at("waveform").get>(); const auto resampled = clap::resample_audio({wave, 48000}, 24000); @@ -51,10 +51,10 @@ int main(int argc, char **argv) { try { action(); } catch (const std::exception &) { rejected = true; } require(rejected, message); }; - rejects([&] { model.text("no text tower"); }, "MuQ text must fail"); + rejects([&] { model.encode_text("no text tower"); }, "MuQ text must fail"); rejects([&] { model.audio_frames(std::vector(1024), 24000); }, "MuQ short waveform must fail"); rejects([&] { model.audio_frames(wave, 0); }, "MuQ invalid rate must fail"); - rejects([&] { model.audio(wave, 24000, 1); }, "MuQ crop must fail"); + rejects([&] { model.encode_audio(wave, 24000, 1); }, "MuQ crop must fail"); auto invalid = wave; invalid[0] = std::numeric_limits::infinity(); rejects([&] { model.audio_frames(invalid, 24000); }, "MuQ nonfinite must fail"); std::cout << "MuQ native parity passed\n"; diff --git a/tests/models/muq-mulan-model-test.cpp b/tests/models/muq-mulan-model-test.cpp index b981d03..8af6158 100644 --- a/tests/models/muq-mulan-model-test.cpp +++ b/tests/models/muq-mulan-model-test.cpp @@ -27,33 +27,33 @@ int main(int argc, char **argv) { } require(argc == 3, "usage: muq-mulan-model-test MODEL FIXTURE"); nlohmann::json f; std::ifstream input(argv[2]); input >> f; - clap::Model m(argv[1], 2); + clap::CLAP m(argv[1], 2); require(m.projection_dimension() == f.at("dimension"), "joint width"); require(m.audio_sample_rate() == 24000, "rate"); const float tol = f.at("tolerance"); for (const auto &c : f.at("texts")) { const auto ids = c.at("ids").get>(); require(m.tokenize(c.at("text"), 10000) == ids, "token parity"); - compare(m.text(ids), c.at("expected"), tol, "IDs: " + c.at("text").get()); - compare(m.text(c.at("text").get()), c.at("expected"), tol, "UTF8"); + compare(m.encode_text(ids), c.at("expected"), tol, "IDs: " + c.at("text").get()); + compare(m.encode_text(c.at("text").get()), c.at("expected"), tol, "UTF8"); } for (const auto &c : f.at("audio")) - compare(m.audio(c.at("wave"),24000), c.at("expected"), tol, c.at("name")); + compare(m.encode_audio(c.at("wave"),24000), c.at("expected"), tol, c.at("name")); auto reject = [&](auto fn, const std::string &name) { bool caught=false; try { fn(); } catch(const std::exception &) {caught=true;} require(caught,"must reject " + name); }; const auto wave = f.at("audio").back().at("wave").get>(); - compare(m.audio(wave,48000), m.audio(clap::resample_audio({wave,48000},24000).samples,24000),0,"automatic native resampling"); + compare(m.encode_audio(wave,48000), m.encode_audio(clap::resample_audio({wave,48000},24000).samples,24000),0,"automatic native resampling"); reject([&]{m.audio_frames(wave,24000);},"frames"); reject([&]{m.clamp3_features(wave,24000);},"clamp3"); - reject([&]{m.audio(wave,24000,1);},"crop"); - reject([&]{m.audio(wave,0);},"rate"); - reject([&]{m.audio({},24000);},"empty"); - reject([&]{m.audio(std::vector(512),24000);},"short"); + reject([&]{m.encode_audio(wave,24000,1);},"crop"); + reject([&]{m.encode_audio(wave,0);},"rate"); + reject([&]{m.encode_audio({},24000);},"empty"); + reject([&]{m.encode_audio(std::vector(512),24000);},"short"); auto bad=wave; bad[0]=std::numeric_limits::quiet_NaN(); - reject([&]{m.audio(bad,24000);},"nonfinite"); - reject([&]{m.text(std::vector{});},"empty IDs"); - reject([&]{m.text(std::vector{-1});},"invalid ID"); - reject([&]{m.text(std::vector(600,0));},"over context IDs"); - reject([&]{m.text(std::string(3000,'a'));},"over context UTF8"); + reject([&]{m.encode_audio(bad,24000);},"nonfinite"); + reject([&]{m.encode_text(std::vector{});},"empty IDs"); + reject([&]{m.encode_text(std::vector{-1});},"invalid ID"); + reject([&]{m.encode_text(std::vector(600,0));},"over context IDs"); + reject([&]{m.encode_text(std::string(3000,'a'));},"over context UTF8"); std::cout << "MuLan native contracts passed" << std::endl; } catch(const std::exception &e) {std::cerr<