Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 17 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ which provides the `embedding`, `serve` and `quantize` subcommands.

| Path | Responsibility |
| --- | --- |
| `include/clap.h` | Public, dependency-light C++ API (`clap::Model`, `decode_audio`, `resample_audio`, `log_mel`) |
| `include/clap.h` | Public, dependency-light C++ API (`clap::CLAP`, `decode_audio`, `resample_audio`, `log_mel`) |
| `src/model.h`, `src/model.cpp` | Internal architecture interface, dispatch and public API implementation |
| `src/model-loader.*` | Shared GGUF metadata, mapping lifetime and tensor validation |
| `src/model-ops.*` | Shared ggml matrix operations (CPU/optional Vulkan) and CPU normalization |
Expand Down Expand Up @@ -385,7 +385,7 @@ mono decoding/resampling to 24 kHz and MERT's whole-recording normalization,
five-second windows and all-layer pooling described below. No external MERT
model or precomputed feature input is accepted. The public feature-input API
and its CLI flag have been removed; old feature-only GGUFs must be reconverted.
`Model::audio(pcm, rate)` is the C++ waveform entry point.
`CLAP::encode_audio(pcm, rate)` is the C++ waveform entry point.

The bundle's string metadata contract is:

Expand Down Expand Up @@ -447,13 +447,13 @@ is required for inference.

Two different outputs are intentionally exposed:

- `--frames` / `Model::audio_frames(pcm, rate)` normalize the complete supplied
- `--frames` / `CLAP::audio_frames(pcm, rate)` normalize the complete supplied
waveform and return the **last transformer layer**, in time-major order.
The published model has 768 channels and frame count
`floor((samples - 400) / 320) + 1` for at least 400 samples at 24 kHz.
Without either output flag, `Model::audio()` / the CLI mean-pool these final
Without either output flag, `CLAP::encode_audio()` / the CLI mean-pool these final
frames into a non-normalized 768-vector. This is **not** a CLaMP3 input vector.
- `--clamp3-features` / `Model::clamp3_features(pcm, rate)` implement the official
- `--clamp3-features` / `CLAP::clamp3_features(pcm, rate)` implement the official
CLaMP3 `preprocessing/audio/extract_mert.py --mean_features` contract at
[`sanderwood/clamp3@9016d2b0c8d12d1aa79c2e0ab201e6822bdc83a8`](https://github.com/sanderwood/clamp3/tree/9016d2b0c8d12d1aa79c2e0ab201e6822bdc83a8/preprocessing/audio):
mix to mono and resample to 24 kHz, normalize the **whole recording** once as
Expand All @@ -469,9 +469,9 @@ MERT inspection APIs remain available; CLaMP3 feature encoding itself is private
For C++ end-to-end inference:

```cpp
clap::Model clamp3("clamp3-f16.gguf");
clap::CLAP clamp3("clamp3-f16.gguf");
auto pcm = clap::decode_audio("music.flac");
auto embedding = clamp3.audio(pcm.samples, pcm.sample_rate);
auto embedding = clamp3.encode_audio(pcm.samples, pcm.sample_rate);
```

CLaMP3 waveform inference includes the leading/trailing zero vectors, chunking
Expand Down Expand Up @@ -551,10 +551,10 @@ followed by 12 width-1024 Wav2Vec2-Conformer layers. In this architecture RoPE i
applied **before Q/K projections**, not after as in MERT2; feed-forward and
convolution branches use SiLU. Encoder final LayerNorm is included.

`Model::audio_frames()` and `--frames` return the final unnormalized sequence;
`CLAP::audio_frames()` and `--frames` return the final unnormalized sequence;
for `N` native-rate samples its frame count is `ceil(floor(N / 240) / 4)` (about
25 Hz), not MERT2's floor rule. At least 1,025 samples are required.
`Model::audio()` and the default CLI output mean-pool all returned frames into
`CLAP::encode_audio()` and the default CLI output mean-pool all returned frames into
one 1,024-vector, without L2 normalization. Pooling is a convenience over MuQ
features, **not** a MuQ-MuLan text/audio projection. Text, precomputed features,
batching, training/RVQ codes, and intermediate-layer selection are unsupported.
Expand Down Expand Up @@ -615,7 +615,7 @@ quantized-storage rejection without requiring ML packages.
### MuQ-MuLan-large (native joint audio/text embeddings)

`OpenMuQ/MuQ-MuLan-large` has native, single-item text and raw-waveform
inference through `clap::Model` and `clap embedding`. Published embeddings have 512
inference through `clap::CLAP` and `clap embedding`. Published embeddings have 512
components. No Python, model source execution, or downloads occur at inference.
The model revision is `2e01c796b71dca71b45251384c04cd7b237c9020`; the reviewed
[official source](https://github.com/tencent-ailab/MuQ/tree/28847ea50cd31ac4b8b6a7dacc051ad7d1c7606a)
Expand Down Expand Up @@ -661,10 +661,10 @@ artifacts/build/clap embedding --model artifacts/muq-mulan-f16.gguf \
```

```cpp
clap::Model model("artifacts/muq-mulan-f16.gguf", 2);
auto text = model.text("hopeful piano music");
clap::CLAP model("artifacts/muq-mulan-f16.gguf", 2);
auto text = model.encode_text("hopeful piano music");
auto audio = clap::decode_audio("music.flac");
auto music = model.audio(audio.samples, audio.sample_rate);
auto music = model.encode_audio(audio.samples, audio.sample_rate);
// dot(music, text) is the official similarity operation.
```

Expand All @@ -690,8 +690,8 @@ additional padding mask for the single-item API, and no final adapter norm.
Mean pooling includes special tokens, then the learned 768→512 affine
projection and L2 normalization apply. Empty text is valid. Over-context text
is rejected, rather than truncated or chunked; the published XLM-R table allows
512 tokenizer IDs including specials. `Model::tokenize(text, max_length)` keeps
its existing explicit truncation API; `Model::text(string)` tokenizes the full
512 tokenizer IDs including specials. `CLAP::tokenize(text, max_length)` keeps
its existing explicit truncation API; `CLAP::encode_text(string)` tokenizes the full
input and checks the context. Batch inference and other backbone/use-layer
variants are unsupported.

Expand Down Expand Up @@ -784,8 +784,8 @@ rotary Conformer blocks. No Python or runtime network access is required.
Without `--frames`, the CLI mean-pools all final frames and prints one
1,024-dimensional recording embedding, matching the official model-card example.
`--frames` instead prints an object with `frames`, `dimension`, and a flat
time-major `values` array. In C++, `Model::audio_frames()` exposes the same
sequence as `clap::AudioFrames`; `Model::audio()` returns its mean. At published
time-major `values` array. In C++, `CLAP::audio_frames()` exposes the same
sequence as `clap::AudioFrames`; `CLAP::encode_audio()` returns its mean. At published
settings the frame count is `floor(input_samples / 960)` (25 Hz).

FullSong means the complete supplied waveform is processed. There is no implicit
Expand Down
26 changes: 13 additions & 13 deletions include/clap.h
Original file line number Diff line number Diff line change
Expand Up @@ -48,30 +48,30 @@ std::vector<float> log_mel(const std::vector<float> &pcm, int sample_rate = 4800
// CLAP returns unit-normalized projections. CLaMP3 returns its native unnormalized
// pooled projections. Its mandatory bundled MERT-v1-95M frontend handles
// waveform preprocessing internally; no separate feature-input API is exposed.
// MERT-v1 audio_frames() exposes final-layer frames; audio() mean-pools them.
// MERT-v1 audio_frames() exposes final-layer frames; encode_audio() mean-pools them.
// Neither is the all-layer CLaMP3 representation: use clamp3_features() for that.
// MERT-v2-FullSong and MuQ audio() mean-pool the complete input's final frames;
// MERT-v2-FullSong and MuQ encode_audio() mean-pool the complete input's final frames;
// audio_frames() returns the unpooled sequence (approximately 25 Hz for published weights).
// MuQ-MuLan text() returns a single L2-normalized joint embedding. audio() uses
// MuQ-MuLan encode_text() returns a single L2-normalized joint embedding. encode_audio() uses
// successive 10 s clips, one prefix slice for a tail, per-clip normalization and
// an unweighted average without final normalization. No amplitude normalization.
// MuLan supports neither frame/features APIs nor crop offsets; published width
// is 512. Its text string API rejects over-context input without truncation.
class Model {
class CLAP {
public:
explicit Model(const std::string &gguf_path, int threads = 0);
~Model();
Model(Model &&) noexcept;
Model &operator=(Model &&) noexcept;
Model(const Model &) = delete;
Model &operator=(const Model &) = delete;
explicit CLAP(const std::string &gguf_path, int threads = 0);
~CLAP();
CLAP(CLAP &&) noexcept;
CLAP &operator=(CLAP &&) noexcept;
CLAP(const CLAP &) = delete;
CLAP &operator=(const CLAP &) = delete;

std::vector<int> tokenize(const std::string &utf8, std::size_t max_length = 512) const;
std::vector<float> text(const std::vector<int> &token_ids);
std::vector<float> text(const std::string &utf8);
std::vector<float> encode_text(const std::vector<int> &token_ids);
std::vector<float> encode_text(const std::string &utf8);
// Mono waveform input; automatically resampled for the model.
// CLaMP3 GGUFs must contain the bundled MERT-v1-95M frontend.
std::vector<float> audio(const std::vector<float> &pcm, int sample_rate = 48000,
std::vector<float> encode_audio(const std::vector<float> &pcm, int sample_rate = 48000,
std::size_t crop_offset = 0);
AudioFrames audio_frames(const std::vector<float> &pcm, int sample_rate = 48000);
// MERT-v1 CLaMP3 --mean_features preprocessing: normalize the entire mono
Expand Down
27 changes: 13 additions & 14 deletions python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,41 +21,40 @@ Convert a supported checkpoint to GGUF with the repository's conversion tools,
then load it once and reuse it:

```python
from clap_cpp import Model, decode_audio
from clap_cpp import CLAP

with Model("model.gguf", threads=4) as model:
text_embedding = model.text("A dog barking")
pcm, sample_rate = decode_audio("dog.wav")
audio_embedding = model.audio(pcm, sample_rate=sample_rate)
with CLAP("model.gguf", threads=4) as model:
text_embedding = model.encode_text("A dog barking")
audio_embedding = model.encode_audio("dog.wav")
print(model.projection_dimension, model.audio_sample_rate)
```

`Model` uses the same GGUF architecture dispatch as the native library (CLAP,
`CLAP` uses the same GGUF architecture dispatch as the native library (CLAP,
CLaMP3, MERT-v1, MERT-v2, MuQ and MuQ-MuLan). An operation unsupported by a
particular architecture raises a native error; for example audio-only encoders
cannot embed text. CLaMP3 requires a GGUF with its bundled MERT frontend.

## API

- `Model(path, threads=0)`: `str`/`pathlib.Path`; zero selects the native thread
- `CLAP(path, threads=0)`: `str`/`pathlib.Path`; zero selects the native thread
default. `close()` is idempotent and the context manager closes on exit.
- `model.text(text_or_tokens)`: one UTF-8 string or token-ID sequence to a
- `model.encode_text(text_or_tokens)`: one UTF-8 string or token-ID sequence to a
one-dimensional float32 NumPy embedding.
- `model.tokenize(text, max_length=512)`: native tokenizer returning a list of
integer IDs. Its limit must be compatible with the loaded model; strings may
be truncated by the tokenizer. Prefer `text(str)` for normal use.
- `model.audio(pcm, sample_rate=48000, crop_offset=0)`: mono waveform to a float32
be truncated by the tokenizer. Prefer `encode_text(str)` for normal use.
- `model.encode_audio(pcm, sample_rate=48000, crop_offset=0)`: mono waveform to a float32
embedding. Automatic model-rate resampling; crop offset uses the native
model-rate sample units and is not supported by every architecture.
- `model.audio_frames(pcm, sample_rate=48000)`: native frame representations as a
`[frames, dimension]` float32 array, for architectures exposing frames.
- `model.clamp3_features(pcm, sample_rate=24000)`: MERT-v1 CLaMP3 preprocessing
features as a `[windows, dimension]` float32 array (at least one second).
- `model.projection_dimension`, `model.audio_sample_rate`: read-only integers.
- `decode_audio(path)`: `(pcm, sample_rate)`; decodes WAV, MP3 or FLAC, averages
channels to mono, preserves sample rate.
- `resample_audio(pcm, source_rate, target_rate=48000)`: mono float32 array using
the native resampler.
- `model.encode_audio(path, crop_offset=0)`: `str`/`pathlib.Path` to a WAV, MP3
or FLAC file. Decodes internally, averages channels to mono and automatically
resamples from the file's sample rate to the model's rate. Crop offset has the
same model-rate sample units as the PCM overload.

PCM accepts one-dimensional numeric arrays or sequences. Noncontiguous arrays
and other numeric dtypes are converted to contiguous float32; the binding copies
Expand Down
79 changes: 34 additions & 45 deletions python/bindings.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -51,19 +51,19 @@ py::array_t<float> matrix(const clap::AudioFrames &frames) {

// Always release the GIL BEFORE taking the model mutex; release the mutex
// BEFORE reacquiring the GIL. close(), properties and inference share this order.
class Model {
class CLAP {
public:
Model(const std::filesystem::path &path, int threads) {
CLAP(const std::filesystem::path &path, int threads) {
if (threads < 0) throw std::invalid_argument("threads must be nonnegative");
py::gil_scoped_release release;
model_ = std::make_unique<clap::Model>(path.u8string(), threads);
model_ = std::make_unique<clap::CLAP>(path.u8string(), threads);
}

template <typename F>
auto call(F &&function) {
py::gil_scoped_release release;
std::lock_guard<std::mutex> lock(mutex_);
if (!model_) throw std::runtime_error("Model is closed");
if (!model_) throw std::runtime_error("CLAP is closed");
return function(*model_);
}

Expand All @@ -74,72 +74,61 @@ class Model {
}

private:
std::unique_ptr<clap::Model> model_;
std::unique_ptr<clap::CLAP> model_;
std::mutex mutex_;
};

} // namespace

PYBIND11_MODULE(_clap_cpp, module) {
module.doc() = "Native CPU inference for clap.cpp";
py::class_<Model>(module, "Model")
py::class_<CLAP>(module, "CLAP")
.def(py::init<const std::filesystem::path &, int>(), py::arg("path"), py::arg("threads") = 0)
.def_property_readonly("projection_dimension", [](Model &self) {
return self.call([](clap::Model &model) { return model.projection_dimension(); });
.def_property_readonly("projection_dimension", [](CLAP &self) {
return self.call([](clap::CLAP &model) { return model.projection_dimension(); });
})
.def_property_readonly("audio_sample_rate", [](Model &self) {
return self.call([](clap::Model &model) { return model.audio_sample_rate(); });
.def_property_readonly("audio_sample_rate", [](CLAP &self) {
return self.call([](clap::CLAP &model) { return model.audio_sample_rate(); });
})
.def("tokenize", [](Model &self, const std::string &text, py::ssize_t max_length) {
.def("tokenize", [](CLAP &self, const std::string &text, py::ssize_t max_length) {
if (max_length <= 0) throw std::invalid_argument("max_length must be positive");
return self.call([&](clap::Model &model) { return model.tokenize(text, max_length); });
return self.call([&](clap::CLAP &model) { return model.tokenize(text, max_length); });
}, py::arg("text"), py::arg("max_length") = 512)
.def("text", [](Model &self, const std::string &text) {
return array(self.call([&](clap::Model &model) { return model.text(text); }));
.def("encode_text", [](CLAP &self, const std::string &text) {
return array(self.call([&](clap::CLAP &model) { return model.encode_text(text); }));
}, py::arg("text"))
.def("text", [](Model &self, const std::vector<int> &tokens) {
return array(self.call([&](clap::Model &model) { return model.text(tokens); }));
.def("encode_text", [](CLAP &self, const std::vector<int> &tokens) {
return array(self.call([&](clap::CLAP &model) { return model.encode_text(tokens); }));
}, py::arg("tokens"))
.def("audio", [](Model &self, const PCM &pcm, int sample_rate, py::ssize_t crop_offset) {
.def("encode_audio", [](CLAP &self, const PCM &pcm, int sample_rate, py::ssize_t crop_offset) {
validate_rate(sample_rate);
if (crop_offset < 0) throw std::invalid_argument("crop_offset must be nonnegative");
auto input = samples(pcm);
return array(self.call([&](clap::Model &model) { return model.audio(input, sample_rate, crop_offset); }));
return array(self.call([&](clap::CLAP &model) { return model.encode_audio(input, sample_rate, crop_offset); }));
}, py::arg("pcm"), py::arg("sample_rate") = 48000, py::arg("crop_offset") = 0)
.def("audio_frames", [](Model &self, const PCM &pcm, int sample_rate) {
.def("encode_audio", [](CLAP &self, const std::filesystem::path &path, py::ssize_t crop_offset) {
if (crop_offset < 0) throw std::invalid_argument("crop_offset must be nonnegative");
const auto filename = path.u8string();
return array(self.call([&](clap::CLAP &model) {
const auto decoded = clap::decode_audio(filename);
return model.encode_audio(decoded.samples, decoded.sample_rate, crop_offset);
}));
}, py::arg("path"), py::arg("crop_offset") = 0)
.def("audio_frames", [](CLAP &self, const PCM &pcm, int sample_rate) {
validate_rate(sample_rate);
auto input = samples(pcm);
return matrix(self.call([&](clap::Model &model) { return model.audio_frames(input, sample_rate); }));
return matrix(self.call([&](clap::CLAP &model) { return model.audio_frames(input, sample_rate); }));
}, py::arg("pcm"), py::arg("sample_rate") = 48000)
.def("clamp3_features", [](Model &self, const PCM &pcm, int sample_rate) {
.def("clamp3_features", [](CLAP &self, const PCM &pcm, int sample_rate) {
validate_rate(sample_rate);
auto input = samples(pcm);
return matrix(self.call([&](clap::Model &model) { return model.clamp3_features(input, sample_rate); }));
return matrix(self.call([&](clap::CLAP &model) { return model.clamp3_features(input, sample_rate); }));
}, py::arg("pcm"), py::arg("sample_rate") = 24000)
.def("close", &Model::close)
.def("__enter__", [](Model &self) -> Model & {
self.call([](clap::Model &) {});
.def("close", &CLAP::close)
.def("__enter__", [](CLAP &self) -> CLAP & {
self.call([](clap::CLAP &) {});
return self;
}, py::return_value_policy::reference_internal)
.def("__exit__", [](Model &self, py::object, py::object, py::object) { self.close(); });
.def("__exit__", [](CLAP &self, py::object, py::object, py::object) { self.close(); });

module.def("decode_audio", [](const std::filesystem::path &path) {
clap::Audio decoded;
{
py::gil_scoped_release release;
decoded = clap::decode_audio(path.u8string());
}
return py::make_tuple(array(decoded.samples), decoded.sample_rate);
}, py::arg("path"));
module.def("resample_audio", [](const PCM &pcm, int source_rate, int target_rate) {
validate_rate(source_rate);
validate_rate(target_rate);
clap::Audio input{samples(pcm), source_rate};
clap::Audio output;
{
py::gil_scoped_release release;
output = clap::resample_audio(input, target_rate);
}
return array(output.samples);
}, py::arg("pcm"), py::arg("source_rate"), py::arg("target_rate") = 48000);
}
4 changes: 2 additions & 2 deletions python/clap_cpp/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""CPU audio/text embeddings backed by the native clap.cpp runtime."""

from ._clap_cpp import Model, decode_audio, resample_audio
from ._clap_cpp import CLAP

__all__ = ["Model", "decode_audio", "resample_audio"]
__all__ = ["CLAP"]
Loading
Loading