Q4_K_M vs Q5_K_M vs Q8_0: Which GGUF Quant Should You Use?

Pick Q4_K_M by default, move up to Q5_K_M when the roughly 17% larger file still fits entirely in your GPU memory, and reserve Q8_0 for cases where you want near-lossless output and have memory to spare. On Llama-3-8B, llama.cpp's own quantize tool lists Q4_K_M at +0.1754 perplexity over the full-precision model, Q5_K_M at +0.0569, and Q8_0 at +0.0026 (Source: llama.cpp quantize tool).

Those three numbers explain most of the decision. Q5_K_M removes about two thirds of the quality loss Q4_K_M introduces, and Q8_0 removes almost all of it. The price is memory and speed: on Llama-3.1-8B, Q4_K_M is 4.8944 bits per weight and generates 71.93 tokens per second, Q5_K_M is 5.7036 bits at 67.23 tokens per second, and Q8_0 is 8.5008 bits at 50.93 tokens per second (Source: llama.cpp quantize tool).

Key takeaways

• Q4_K_M is the practical default: 4.89 bits per weight, about 4.9 GB for an 8B model, with a small measured quality loss.
• Q5_K_M costs 16.5% more memory and about 6.5% of generation speed, and cuts perplexity loss by roughly 68%.
• Q8_0 is close to full precision but is 74% larger than Q4_K_M and about 29% slower to generate.
• IQ4_XS is the pick when Q4_K_M is just too large for your GPU; it is slower on CPU.

What do Q4_K_M, Q5_K_M and Q8_0 mean?

GGUF is the file format llama.cpp, Ollama and LM Studio load, and the suffix names the quantization scheme used to shrink the weights. GGUF stores the tensors together with a standardized block of metadata, which is why one file carries everything a runtime needs (Source: Hugging Face GGUF docs).

The number is the base bit width. Q4 stores most weights in 4 bits, Q5 in 5 bits, Q8 in 8 bits. The K marks a K-quant, the super-block scheme introduced in llama.cpp pull request #1684. In that design, the Q4_K type packs 8 blocks of 32 weights with 6-bit scales and minimums, which works out to 4.5 bits per weight; Q5_K uses the same layout at 5.5 bits per weight (Source: llama.cpp PR #1684).

The final letter is the mix. _S (small) uses the base type for every tensor. _M (medium) upgrades sensitive tensors: the PR defines Q4_K_M and Q5_K_M as using the 6-bit Q6_K type "for half of the attention.wv and feed_forward.w2 tensors", with the base type everywhere else (Source: llama.cpp PR #1684).

That mixing is why a Q4_K_M file measures 4.89 bits per weight rather than 4.5. Q8_0 is an older, simpler format: 8-bit round-to-nearest in blocks of 32 weights, with a single scale per block and no mixing (Source: Hugging Face GGUF docs).

The quant comparison table

This table puts the common quants side by side. Bits per weight are llama.cpp's measured file-level figures for Llama-3.1-8B. File sizes are calculated for other model sizes, and quality columns come from two separate published measurements.

The file size formula is parameters x bits per weight / 8, giving gigabytes when parameters are in billions. For an 8B model at Q4_K_M, that is 8 x 4.8944 / 8 = 4.89 GB. llama.cpp reports the same file as 4.58 GiB, which is the same size in binary units (Source: llama.cpp quantize tool).

The perplexity deltas are the figures printed by llama-quantize itself. The KLD column is the median Kullback-Leibler divergence from an independent Mistral-7B study, which added IQ4_XS in February 2024 (Sources: llama.cpp quantize tool, Artefact2 KLD study).

How big is the quality gap between Q4_K_M and Q5_K_M?

The gap is small and consistent. Three data sets on three different base models rank the quants the same way, and all show Q5_K_M closing most of Q4_K_M's loss.

The benchmark row matters most for everyday use. Across GSM8K, HellaSwag, IFEval, MMLU and TruthfulQA, Q4_K_M lands 0.32 points below FP16, Q5_K_M 0.11 points below, and Q8_0 0.06 points below. The same paper found 3-bit Q3_K_M dropping to 68.07 and concluded that 5-bit options make the better default when quality matters and memory is limited (Source: arXiv 2601.14277).

Do not average these rows; each used a different model, dataset and llama.cpp build. What transfers is the ordering and the shape: loss falls steeply from 3 to 5 bits, then flattens. The original PR noted that its 6-bit perplexity was "within 0.1% or better" of fp16 (Source: llama.cpp PR #1684).

Inference: for chat, summarization and most coding help on an 8B model, the Q4_K_M to Q5_K_M difference is hard to notice. It shows up first in long reasoning chains and exact-recall tasks.

Q4_K_M vs Q5_K_M: when do the extra bits pay off?

Q5_K_M pays off when it fits entirely on the GPU with room left for your context window. Its file is 16.5% larger than Q4_K_M at every model size, because the bits-per-weight ratio is fixed: 5.7036 divided by 4.8944 (Source: llama.cpp quantize tool).

In concrete terms, that is 0.81 GB more for an 8B model, 1.41 GB more for a 14B model, and 3.23 GB more for a 32B model. On Llama-3.1-8B, llama.cpp measured generation at 67.23 tokens per second for Q5_K_M against 71.93 for Q4_K_M, a 6.5% slowdown (Source: llama.cpp quantize tool).

The practical rule follows from those two numbers. If moving from Q4_K_M to Q5_K_M forces any layers off the GPU, stay on Q4_K_M. Inference: layers that spill to system RAM cost far more speed than the 6.5% gap between the two quants. If both fit comfortably, Q5_K_M is a cheap upgrade.

A second rule concerns model size. An independent KLD study that tested a range of quants gives this advice: "use the largest that fully fits in your GPU. If you can comfortably fit Q4_K_S, try using a model with more parameters." (Source: Artefact2 KLD study)

Inference: by that logic, a 14B model at Q4_K_M (8.57 GB) is usually a better use of the same memory than an 8B model at Q8_0 (8.50 GB).

If you run models on a small machine, our 8GB RAM local LLM test shows how tight 4-bit files get once the operating system takes its share.

Q4_K_M vs Q8_0: is near-lossless worth it?

Q8_0 is worth it only when memory is not the constraint. It is 73.7% larger than Q4_K_M (8.5008 against 4.8944 bits per weight), and on Llama-3.1-8B it generated 50.93 tokens per second against 71.93, about 29% slower (Source: llama.cpp quantize tool).

The quality you buy is small. Q8_0's perplexity delta on Llama-3-8B is +0.0026, essentially noise, while Q4_K_M's is +0.1754. On the five-benchmark average, Q8_0 scored 69.41 and Q4_K_M 69.15, a 0.26-point gap (Sources: llama.cpp quantize tool, arXiv 2601.14277).

Q8_0 makes sense when memory far exceeds the model's needs, such as an 8B model on a 24 GB card, or when you want a reference file that behaves like the original weights.

For most people choosing a file to run daily, Q6_K is the better high-quality stop. It keeps perplexity within +0.0217 on Llama-3-8B while being 23% smaller than Q8_0 (Source: llama.cpp quantize tool).

The runtime matters less than the quant: Ollama and llama.cpp read the same GGUF files, as our Ollama vs llama.cpp comparison explains.

Where do IQ4_XS and Q4_K_S fit?

IQ4_XS is the answer when Q4_K_M is slightly too large. It is an I-quant, a family that uses a non-linear grid and usually an importance matrix, and llama.cpp lists the base type at 4.25 bits per weight. On Llama-3.1-8B the full file measures 4.4597 bits per weight and 4.17 GiB, against 4.58 GiB for Q4_K_M (Sources: llama.cpp quantize tool, Hugging Face GGUF docs).

The independent KLD study put IQ4_XS at a median divergence of 0.0088, next to Q4_K_S at 0.0083 and Q4_K_M at 0.0075. In practice, IQ4_XS buys about 9% less memory than Q4_K_M for a quality level close to Q4_K_S (Source: Artefact2 KLD study).

The catch is hardware. Quantizer bartowski's model cards note that I-quants are strong on GPU backends such as cuBLAS and rocBLAS but slower on CPU than a K-quant of similar size. If part of your model runs on CPU, or you are on Apple Silicon, test Q4_K_S before assuming IQ4_XS is faster (Source: bartowski Qwen3.8-27B GGUF).

Q4_K_S sits between them. It uses the 4-bit K type for every tensor, which makes it 4.6672 bits per weight and gives it a +0.2689 perplexity delta on Llama-3-8B, about 50% more loss than Q4_K_M for 4.6% less size (Source: llama.cpp quantize tool).

Will it fit in your VRAM?

A quant fits when its file size plus the KV cache plus runtime overhead stays under your GPU memory. llama.cpp's documentation states that memory and disk requirements are the same for the weights, so the file size is your starting number (Source: llama.cpp quantize tool).

Our formula slightly underestimates real files, because embeddings and output layers are kept at higher precision. Checked against bartowski's Qwen3.8-27B files, the calculation runs 1.5% low for Q8_0 (29.12 GB actual), 5.6% low for Q4_K_M (17.44 GB) and 8.7% low for Q5_K_M (20.92 GB). The same cards advise choosing a file 1 to 2 GB smaller than your total VRAM (Source: bartowski Qwen3.8-27B GGUF).

Put together, the fit rule is: calculated file size, plus 10%, plus the KV cache for your context length, plus 1 to 2 GB, must be at or under your VRAM.

Two worked examples show how it plays out:

• On a 12 GB card, a 14B model at Q4_K_M (8.57 GB, about 9.4 GB with margin) fits with room for a moderate context. Q5_K_M (9.98 GB, about 11 GB with margin) leaves almost nothing for the KV cache and overhead.
• On a 24 GB card, a 32B model at Q4_K_M (19.58 GB, about 21.5 GB with margin) fits only with a short context. IQ4_XS at 17.84 GB is the safer choice for long prompts.

The KV cache is the part people forget, and it grows with context length. Our guide to KV cache size shows how to calculate it, and the LLM VRAM calculator adds weights, cache and overhead for a specific model and quant.

Why perplexity alone can mislead you

Perplexity measures how surprised a model is by reference text, and it is an average, so it can hide damage. llama.cpp's documentation names both perplexity and Kullback-Leibler divergence (KLD) as the usual loss measures. KLD compares the quantized model's token probabilities directly with the original model's (Source: llama.cpp quantize tool).

Recent reports show the limits in practice. A September 2026 llama.cpp issue, #28879, found that on a hybrid linear-attention model, a full F16 file scored worse perplexity than its own Q4_K_M, which cannot reflect real quality. An open Ollama issue, #18252, reports that the library's 2-bit and 3-bit builds of one 3B coding model failed every task in a small code suite while the 4-bit build passed. A long-running llama.cpp thread, discussion #4110, argues that perplexity is an inaccurate loss benchmark and that KLD is the better data point, which is why the table above carries both (Source: Artefact2 KLD study).

Treat published deltas as a ranking, not a guarantee for your model, and be cautious below 4 bits on small models. Test your own prompts on two candidate quants: the arXiv study found multi-step math on GSM8K moved more under quantization than general knowledge on MMLU (Source: arXiv 2601.14277).

FAQ

What does Q4_K_M mean?

Q4_K_M is a GGUF quantization that stores most weights in 4 bits using llama.cpp's K-quant super-block scheme. In the original design, the "M" (medium) mix upgrades half of the attention value and feed-forward output tensors to 6 bits. The result averages about 4.89 bits per weight on Llama-3.1-8B, about 4.9 GB for an 8B model.

Is Q5_K_M better than Q4_K_M?

Yes, Q5_K_M is measurably better, but the difference is small. On Llama-3-8B, its perplexity delta is +0.0569 against +0.1754 for Q4_K_M, and a five-benchmark average moved from 69.15 to 69.36. It costs 16.5% more memory and about 6.5% of generation speed, so choose it only when it fits on the GPU.

Q4_K_M vs Q8_0: which should I use?

Use Q4_K_M unless you have memory to spare. Q8_0 is near-lossless, with a +0.0026 perplexity delta on Llama-3-8B, but its file is 74% larger and it generated about 29% slower on Llama-3.1-8B. With the same memory, a larger model at Q4_K_M usually beats a smaller model at Q8_0.

What is GGUF?

GGUF is a binary model file format from llama.cpp creator Georgi Gerganov, built for fast loading and inference. Unlike tensor-only formats such as safetensors, a GGUF file stores the weights plus a standardized set of metadata, so one file carries what a runtime needs. Ollama, LM Studio and llama.cpp all run GGUF files (Source: Hugging Face GGUF docs).

Related coverage

• LLM VRAM calculator: RAM, quantization and model fit
• KV cache size: how much VRAM it uses and how to cut it
• LM Studio vs Ollama for local AI workflows
• Ollama vs llama.cpp: which is faster for local LLMs

References

• Artefact2 KLD study - https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9
• arXiv 2601.14277 - https://arxiv.org/abs/2601.14277
• bartowski Qwen3.8-27B GGUF - https://huggingface.co/bartowski/Qwen3.8-27B-GGUF
• Hugging Face GGUF docs - https://huggingface.co/docs/hub/gguf
• llama.cpp PR #1684 - https://github.com/ggml-org/llama.cpp/pull/1684
• llama.cpp quantize tool - https://github.com/ggml-org/llama.cpp/tree/master/tools/quantize