From the README I see that it is currently possible to quantize the models:
the loader keeps the packed weights and lets ggml_mul_mat dequantize at compute, so the file just loads and runs like the bf16 one
However the inference still runs in BF16, with additional overhead from on-the-fly conversion?
Since llama.cpp supports quantized inference for language models and also for vision models, do you have adding support for quantized inference on the roadmap? It would be a great addition for CPU based inference, which is otherwise very slow, especially so with BF16.
From the README I see that it is currently possible to quantize the models:
However the inference still runs in BF16, with additional overhead from on-the-fly conversion?
Since llama.cpp supports quantized inference for language models and also for vision models, do you have adding support for quantized inference on the roadmap? It would be a great addition for CPU based inference, which is otherwise very slow, especially so with BF16.