GGUF Quantization with Imatrix and K-Quantization to Run LLMs on Your CPU

Fast and accurate GGUF models for your CPU

GGUF is a binary file format designed for efficient storage and fast large language model (LLM) loading with GGML, a C-based tensor library for machine learning.

GGUF encapsulates all necessary components for inference, including the tokenizer and code, within a single file. It supports the conversion of various language models, such as Llama 3, Phi, and Qwen2. Additionally, it facilitates model quantization to lower precisions to improve speed and memory efficiency on CPUs.

We often write “GGUF quantization” but GGUF itself is only a file format, not a quantization method. There are several quantization algorithms implemented in llama.cpp to reduce the model size and serialize the resulting model in the GGUF format.

In this article, we will see how to accurately quantize an LLM and convert it to GGUF, using an importance matrix (imatrix) and the K-Quantization method. I provide the GGUF conversion code for Gemma 2 Instruct, using an imatrix. It works the same with other models supported by llama.cpp: Qwen2, Llama 3, Phi-3, etc. We will also see how to evaluate the accuracy of the quantization and inference throughput of the resulting models.