Complete Guide to Ollama Model Quantization Formats (Q4_K_M vs Q8_0 vs EXL2)

Deploying open-weights language models on consumer hardware requires reducing the memory precision of neural network weights. Raw unquantized models are typically stored in 16-bit floating point precision (FP16), requiring 2 Gigabytes of GPU VRAM for every 1 billion parameters.

Quantization compresses 16-bit float weights down to 8-bit, 4-bit, or even 2-bit integer representations. However, selecting the wrong quantization format can either cause severe model perplexity degradation or cause Out-Of-Memory (OOM) crashes on local GPUs.

In this technical comparison guide, we evaluate the internal block architecture, perplexity loss, VRAM savings, and token decoding throughput across all major local quantization formats: Q4_K_M, Q5_K_M, Q8_0, AWQ, and EXL2.

The Mathematics of Model Quantization & Perplexity

Quantization maps a high-precision continuous floating-point range $[x_{\text{min}}, x_{\text{max}}]$ into a discrete integer grid $q \in [0, 2^b - 1]$, where $b$ is the target bit precision.

Quantization Mapping Equation

$$q = \text{round} \left( \frac{x - x_{\text{min}}}{\Delta} \right), \quad \text{where } \Delta = \frac{x_{\text{max}} - x_{\text{min}}}{2^b - 1}$$

De-quantization reconstructs the floating-point approximation $\hat{x}$:

$$\hat{x} = (q \times \Delta) + x_{\text{min}}$$

The difference $e = x - \hat{x}$ represents quantization error. Across billions of weights, quantization error compounds, measured via Perplexity ($\text{PPL}$):

$$\text{PPL} = \exp \left( -\frac{1}{N} \sum_{i=1}^{N} \ln P(w_i \mid w_1, \dots, w_{i-1}) \right)$$

Lower perplexity indicates higher language generation fidelity and lower prediction error.

flowchart LR
    FP16[FP16 Unquantized Base: 2.0 GB / 1B Params] --> Q8[Q8_0: 1.05 GB / 1B | PPL +0.02]
    FP16 --> Q5KM[Q5_K_M: 0.72 GB / 1B | PPL +0.07]
    FP16 --> Q4KM[Q4_K_M: 0.58 GB / 1B | PPL +0.14]
    FP16 --> Q2K[Q2_K: 0.35 GB / 1B | Severe PPL Loss +1.89]

Master Comparison Matrix: GGUF, AWQ, and EXL2 Formats

Detailed Format Breakdown

1. GGUF Q4_K_M (Best Overall for Ollama & Local Desktop)

GGUF is the universal binary format developed for llama.cpp and Ollama. The Q4_K_M variant utilizes K-quantization, applying dynamic bit allocations:

• Attention input/output matrices ($\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v, \mathbf{W}_o$) are preserved at higher 5-bit or 6-bit precision.
• Feed-forward network (FFN) gate and up-projection matrices ($\mathbf{W}{\text{gate}}, \mathbf{W}{\text{up}}$) are quantized to 4-bit precision.

Result: Reduces VRAM requirement by 71.2% while keeping perplexity loss under +0.14.

2. GGUF Q8_0 (Near-Lossless High Precision)

Q8_0 uses uniform 8-bit integer quantization with 32-weight block scales.

• Use Case: Recommended for mathematical reasoning, complex coding, and zero-shot entity extraction where outputs must match FP16 accuracy.
• VRAM Trade-off: Requires roughly 75% more VRAM than Q4_K_M.

3. EXL2 (ExLlamaV2 - Fastest CUDA Inference)

EXL2 is a specialized GPU quantization format supporting variable fractional bitrates (e.g., 3.5b, 4.25b, 6.0b).

• Key Advantage: Achieves up to 30% higher tokens per second on NVIDIA GPUs compared to GGUF by optimizing matrix multiplication for Tensor Cores.
• Limitation: Does not support CPU memory offloading or Apple Silicon Metal.

Practical Benchmark: Llama 3.1 8B Across Quantizations

Evaluated on an NVIDIA GeForce RTX 4060 8GB GPU:

gantt
    title Token Generation Speed (tok/sec) on RTX 4060
    dateFormat X
    axisFormat %s
    section Q4_K_M (4.8 GB)
    Speed (48.6 tok/s) : 0, 486
    section Q5_K_M (5.8 GB)
    Speed (41.2 tok/s) : 0, 412
    section Q8_0 (8.5 GB - Spillover)
    Speed (5.1 tok/s) : 0, 51

Summary & Selection Checklist

1. Default Choice for Ollama Desktop: Choose Q4_K_M.
2. If You Have Extra VRAM Headroom: Choose Q5_K_M for minor perplexity improvements.
3. For High-Speed CUDA Web Services: Choose EXL2 (4.25b) or AWQ.
4. Avoid Q2_K and Q3_K_S: Perplexity degradation leads to high hallucination rates.