Q4_K_M vs Q5_K_M vs Q8_0: Which GGUF Quant Should You Use?

Q4_K_M vs Q5_K_M is a trade between a little quality and a noticeable amount of speed. In my llama.cpp run of the official Qwen2.5-1.5B-Instruct GGUF files at Q4_K_M, Q5_K_M and Q8_0 on an 8 GB M1, Q5_K_M stayed closer to Q8_0, picking the same next token 94.078% of the time against 90.039% for Q4_K_M, but it generated 38.31 tokens per second against Q4_K_M's 50.12, roughly a quarter slower. My pick: Q4_K_M by default, Q5_K_M when the roughly 17% larger file fits entirely in GPU memory and you can give up some generation speed, and Q8_0 for near-lossless output or for small models whose files are tiny anyway.

The published numbers point the same way on quality. On Llama-3-8B, the llama.cpp quantize tool lists Q4_K_M at +0.1754 perplexity over the full-precision model, Q5_K_M at +0.0569, and Q8_0 at +0.0026, so Q5_K_M removes about two thirds of the loss Q4_K_M introduces. Speed is where my result differs. On Llama-3.1-8B, llama.cpp lists Q4_K_M at 4.8944 bits per weight and 71.93 tokens per second, Q5_K_M at 5.7036 bits and 67.23 tokens per second, and Q8_0 at 8.5008 bits and 50.93 tokens per second: a 6.5% gap between the two K-quants, far smaller than the one I saw on the M1. That is why I no longer call Q5_K_M a free upgrade on a Mac. A model that spills off the GPU loses more than either gap, so fit still decides first.

What I measured on an 8 GB M1

The model is Qwen2.5-1.5B-Instruct, using the three files from Qwen's official Qwen2.5-1.5B-Instruct-GGUF repository, and the table holds every headline number from the run. Perplexity covers 100 chunks of 512 tokens of wikitext-2; the divergence and same-token columns compare each quant with Q8_0 over the first 20 of those chunks.

On quality, Q5_K_M's mean divergence is less than a third of Q4_K_M's, and the gap is wider in the tail: the worst 1% of tokens reached a divergence of 0.067402 for Q5_K_M and 0.254363 for Q4_K_M. Perplexity tells the same story more weakly. Q5_K_M's distance from Q8_0 is smaller than the +/- 0.17275 uncertainty llama-perplexity printed for its run, while Q4_K_M's is larger than its +/- 0.17961.

Three things limit how far these numbers travel:

• Model size: a 1.5B model is close to the worst case for quantization loss, because small models usually lose more from it than 7B and larger ones. I would read the Q4_K_M loss here as an upper bound for what you will see on an 8B model, not a typical figure.
• Reference: the divergence is measured against Q8_0, not F16, because the F16 download did not finish. Q8_0 is close to full precision (+0.0026 perplexity on Llama-3-8B in the quantize tool's list), but not identical, so both quants are probably a little further from the original weights than the table shows.
• Hardware: generation on an M1 with unified memory is mostly limited by how fast the weights can be read, which is why the largest file, Q8_0, generated slowest at 31.71 tokens per second. But Q5_K_M is slower than its size alone would predict. Its file is much closer in size to Q4_K_M than to Q8_0, yet its generation speed sits closer to Q8_0's. My guess is that the 5-bit K-quant blocks cost more to unpack on Metal than the 4-bit and 8-bit ones, but I did not profile that.

Prompt processing ran the other way: Q8_0 was fastest at 586.87 tokens per second and Q5_K_M slowest at 500.14. Prompt processing is batched compute rather than a memory read per token, and I suspect Q8_0's simple blocks are the cheapest to unpack there. For chat on a Mac, generation speed is the number you feel, which is why the M1 result moves my verdict toward Q4_K_M.

What do Q4_K_M, Q5_K_M and Q8_0 mean?

GGUF is the binary model file format from llama.cpp creator Georgi Gerganov that llama.cpp, Ollama and LM Studio load, and the suffix names the quantization scheme used to shrink the weights. Unlike tensor-only formats such as safetensors, the Hugging Face GGUF docs explain, GGUF stores the tensors together with a standardized block of metadata, so one file carries everything a runtime needs.

The number is the base bit width. Q4 stores most weights in 4 bits, Q5 in 5 bits, Q8 in 8 bits. The K marks a K-quant, the super-block scheme introduced in llama.cpp pull request #1684. In that design, the Q4_K type packs 8 blocks of 32 weights with 6-bit scales and minimums, which works out to 4.5 bits per weight; Q5_K uses the same layout at 5.5 bits per weight.

The final letter is the mix. _S (small) uses the base type for every tensor. _M (medium) upgrades sensitive tensors: the PR defines Q4_K_M and Q5_K_M as using the 6-bit Q6_K type "for half of the attention.wv and feed_forward.w2 tensors", with the base type everywhere else.

That mixing is why a Q4_K_M file measures 4.89 bits per weight rather than 4.5. Q8_0 is an older, simpler format: 8-bit round-to-nearest in blocks of 32 weights, with a single scale per block and no mixing.

The quant comparison table

This table puts the common quants side by side. Bits per weight are llama.cpp's measured file-level figures for Llama-3.1-8B. File sizes are calculated for other model sizes, and the two quality columns come from two separate published measurements, neither of them ours.

The file size formula is parameters x bits per weight / 8, giving gigabytes when parameters are in billions. For an 8B model at Q4_K_M, that is 8 x 4.8944 / 8 = 4.89 GB. llama.cpp reports the same file as 4.58 GiB, which is the same size in binary units.

The perplexity deltas are the figures printed by llama-quantize itself. The KLD column is the median Kullback-Leibler divergence reported by an independent study, Artefact2's KLD measurements on Mistral-7B, which added IQ4_XS in February 2024. Those divergence figures are the study's results, quoted as published.

How big is the quality gap between Q4_K_M and Q5_K_M?

The gap is small and consistent. Three data sets on three different base models rank the quants the same way, and all show Q5_K_M closing most of Q4_K_M's loss.

The benchmark row matters most for everyday use. Across GSM8K, HellaSwag, IFEval, MMLU and TruthfulQA, the arXiv 2601.14277 paper puts Q4_K_M 0.32 points below FP16, Q5_K_M 0.11 points below, and Q8_0 0.06 points below. The same paper found 3-bit Q3_K_M dropping to 68.07 and concluded that 5-bit options make the better default when quality matters and memory is limited.

Do not average these rows; each used a different model, dataset and llama.cpp build. What transfers is the ordering and the shape: loss falls steeply from 3 to 5 bits, then flattens. The original PR noted that its 6-bit perplexity was "within 0.1% or better" of fp16.

For chat, summarization and most coding help on an 8B model, I think the Q4_K_M to Q5_K_M difference is hard to notice. I would expect it to show up first in long reasoning chains and exact-recall tasks. On a small model it is easier to see: in my 1.5B run, Q4_K_M chose a different top token from Q8_0 at roughly one position in ten, against about one in seventeen for Q5_K_M.

Q4_K_M vs Q5_K_M: when do the extra bits pay off?

Q5_K_M pays off when it fits entirely on the GPU with room left for your context window. Its file is 16.5% larger than Q4_K_M at every model size, because the bits-per-weight ratio is fixed: 5.7036 divided by 4.8944.

In concrete terms, that is 0.81 GB more for an 8B model, 1.41 GB more for a 14B model, and 3.23 GB more for a 32B model. On Llama-3.1-8B, llama.cpp measured generation at 67.23 tokens per second for Q5_K_M against 71.93 for Q4_K_M, a 6.5% slowdown.

The practical rule follows from those numbers. If moving from Q4_K_M to Q5_K_M forces any layers off the GPU, stay on Q4_K_M, because layers that spill to system RAM should cost far more speed than the gap between the two quants. If both fit comfortably, Q5_K_M is a cheap upgrade by llama.cpp's 6.5% figure, but not on my M1, where it generated 38.31 tokens per second against 50.12. On Apple Silicon I would run both files through llama-bench before settling on Q5_K_M, and keep Q4_K_M when generation speed matters more than the last bit of quality.

A second rule concerns model size. The Artefact2 study, which tested a range of quants, gives this advice: "use the largest that fully fits in your GPU. If you can comfortably fit Q4_K_S, try using a model with more parameters." By that logic, I would usually take a 14B model at Q4_K_M (8.57 GB) over an 8B model at Q8_0 (8.50 GB) in the same memory.

On a small machine, the 8GB RAM local LLM test shows how tight 4-bit files get once the operating system takes its share.

Q4_K_M vs Q8_0: is near-lossless worth it?

Q8_0 is worth it only when memory is not the constraint. It is 73.7% larger than Q4_K_M (8.5008 against 4.8944 bits per weight), and on Llama-3.1-8B it generated 50.93 tokens per second against 71.93, about 29% slower. On my M1 with the 1.5B model the gap was wider still: 31.71 against 50.12 tokens per second.

The quality you buy is small. Q8_0's perplexity delta on Llama-3-8B is +0.0026, essentially noise, while Q4_K_M's is +0.1754. On the arXiv paper's five-benchmark average, Q8_0 scored 69.41 and Q4_K_M 69.15, a 0.26-point gap.

Q8_0 makes sense when memory far exceeds the model's needs, such as an 8B model on a 24 GB card, or when you want a reference file that behaves like the original weights.

For most people choosing a file to run daily, Q6_K is the better high-quality stop. It keeps perplexity within +0.0217 on Llama-3-8B while being 23% smaller than Q8_0.

The runtime matters less than the quant: Ollama and llama.cpp read the same GGUF files, as the Ollama vs llama.cpp comparison explains.

Q4_K_M vs Q4_K_S and IQ4_XS: where do the smaller 4-bit quants fit?

IQ4_XS is the answer when Q4_K_M is slightly too large. It is an I-quant, a family that uses a non-linear grid and usually an importance matrix, and llama.cpp lists the base type at 4.25 bits per weight. On Llama-3.1-8B the full file measures 4.4597 bits per weight and 4.17 GiB, against 4.58 GiB for Q4_K_M.

The Artefact2 study put IQ4_XS at a median divergence of 0.0088, next to Q4_K_S at 0.0083 and Q4_K_M at 0.0075. Read with those numbers, IQ4_XS buys about 9% less memory than Q4_K_M for a quality level close to Q4_K_S.

The catch is hardware. The bartowski Qwen3.8-27B GGUF model card notes that I-quants are strong on GPU backends such as cuBLAS and rocBLAS but slower on CPU than a K-quant of similar size. If part of your model runs on CPU, or you are on Apple Silicon, test Q4_K_S before assuming IQ4_XS is faster.

Q4_K_S sits between them. It uses the 4-bit K type for every tensor, which makes it 4.6672 bits per weight and gives it a +0.2689 perplexity delta on Llama-3-8B, about 50% more loss than Q4_K_M for 4.6% less size.

Q4_K_M vs Q4_0 and Q4_1: skip the legacy quants

Q4_0 and Q4_1 are the original llama.cpp 4-bit formats, and on quality per gigabyte the K-quants beat them. On Llama-3-8B, llama.cpp lists Q4_0 at 4.34 GiB with a +0.4685 perplexity delta and Q4_1 at 4.78 GiB with +0.4511. Q4_K_S is about the same size as Q4_0 (4.37 GiB) with +0.2689, and Q4_K_M is smaller than Q4_1 (4.58 GiB) with +0.1754.

Put another way, Q4_0 carries about 74% more loss than Q4_K_S at the same size, and Q4_1 carries about 2.6 times the loss of Q4_K_M while being 4% larger. The only reason to pick a legacy quant is a runtime or accelerator that lacks K-quant kernels; otherwise choose the K version.

Q4_K_M vs Q4_K_XL: what the Unsloth Dynamic quants change

Q4_K_XL is not a llama.cpp type. It is Unsloth's name for its "Dynamic" GGUF builds, usually published as UD-Q4_K_XL, which pick a quantization type per layer instead of following the fixed Q4_K_M recipe, calibrated on an importance-matrix dataset Unsloth says is weighted toward agentic coding, chat and multilingual text (Unsloth Dynamic GGUF docs).

Unsloth's own MMLU run on Gemma 3 27B shows the effect is small but real. Its Q4_K_M scored 71.23 at 15.41 GB, UD-Q4_K_XL 71.47 at 15.64 GB, and Q5_K_M 71.77 at 17.95 GB, in the same table. These are vendor-reported numbers on one model, measured by the team that makes the quants.

When an Unsloth UD-Q4_K_XL build exists for your model and the extra 1 to 2% of file size fits, I'd treat it as a reasonable replacement for Q4_K_M. It does not change the fit rule below, and Q5_K_M still scored higher in the same table.

Q4_K_M vs MXFP4: the gpt-oss exception

MXFP4 is a 4-bit microscaling floating-point format, and OpenAI post-trained gpt-oss's mixture-of-experts weights in it, so the published evals were run on the MXFP4 weights themselves. Q4_K_M is an integer K-quant applied after training to a higher-precision model. For gpt-oss, use the MXFP4 build; for most other GGUF models, Q4_K_M is the standard 4-bit choice (OpenAI gpt-oss-20b card).

How to download Q4_K_M, Q5_K_M or Q8_0

Most runtimes pick Q4_K_M for you, so getting another quant means naming it. In the Ollama library, quant builds are separate tags on the model's Tags page, for example ollama pull qwen2.5-coder:7b-instruct-q5_K_M or qwen2.5-coder:7b-instruct-q8_0 (Ollama qwen2.5-coder tags).

For any GGUF repository on Hugging Face, Ollama can pull a specific file directly with ollama run hf.co/{username}/{repository}:{quantization}, such as :Q8_0 or :IQ3_M. Hugging Face notes that the quantization name is case-insensitive and that the full filename also works as a tag (Hugging Face Ollama guide). A GGUF file carries its quantization in its metadata, so any runtime can report it. The Ollama commands reference covers ollama show, which prints the quantization of whatever you pulled.

Will it fit in your VRAM?

A quant fits when its file size plus the KV cache plus runtime overhead stays under your GPU memory. llama.cpp's documentation states that memory and disk requirements are the same for the weights, so the file size is your starting number.

The formula above slightly underestimates real files, because embeddings and output layers are kept at higher precision. Compared with bartowski's Qwen3.8-27B files, the calculation runs 1.5% low for Q8_0 (29.12 GB actual), 5.6% low for Q4_K_M (17.44 GB) and 8.7% low for Q5_K_M (20.92 GB). The same cards advise choosing a file 1 to 2 GB smaller than your total VRAM.

Put together, the fit rule is: calculated file size, plus 10%, plus the KV cache for your context length, plus 1 to 2 GB, must be at or under your VRAM.

Two worked examples show how it plays out:

• On a 12 GB card, a 14B model at Q4_K_M (8.57 GB, about 9.4 GB with margin) fits with room for a moderate context. Q5_K_M (9.98 GB, about 11 GB with margin) leaves almost nothing for the KV cache and overhead.
• On a 24 GB card, a 32B model at Q4_K_M (19.58 GB, about 21.5 GB with margin) fits only with a short context. IQ4_XS at 17.84 GB is the safer choice for long prompts.

The KV cache is the part people forget, and it grows with context length. The KV cache size guide shows how to calculate it, and the LLM VRAM calculator adds weights, cache and overhead for a specific model and quant. For coding agents, the best Ollama model for coding guide applies this fit rule to named model tags at each memory tier.

Why perplexity alone can mislead you

Perplexity measures how surprised a model is by reference text, and it is an average, so it can hide damage. llama.cpp's documentation names both perplexity and Kullback-Leibler divergence (KLD) as the usual loss measures. KLD compares the quantized model's token probabilities directly with the original model's.

Recent reports show the limits in practice. A September 2026 llama.cpp issue, #28879, found that on a hybrid linear-attention model, a full F16 file scored worse perplexity than its own Q4_K_M, which cannot reflect real quality. An open Ollama issue, #18252, reports that the library's 2-bit and 3-bit builds of one 3B coding model failed every task in a small code suite while the 4-bit build passed. A long-running llama.cpp thread, discussion #4110, argues that perplexity is an inaccurate loss benchmark and that KLD is the better data point, which is why the table above carries both.

Treat published deltas as a ranking, not a guarantee for your model, and be cautious below 4 bits on small models. Test your own prompts on two candidate quants: the arXiv study found multi-step math on GSM8K moved more under quantization than general knowledge on MMLU.

How I tested this

I ran this on October 4 and 5, 2026, on a MacBook Air with an M1 and 8 GB of unified memory, with desktop apps open (the exact macOS build is in the raw record). The tool was llama.cpp build b11351, the official llama-b11351-bin-macos-arm64 release, on the Metal backend.

The model files were the official Q4_K_M, Q5_K_M and Q8_0 GGUFs of the Qwen2.5 Instruct model named above, downloaded from Qwen's own GGUF repository on Hugging Face. The text was the wikitext-2-raw-v1 test split (wiki.test.raw). For each file I ran perplexity and llama-bench, with the context, chunk, prompt and generation lengths given above the results table (C, N, P and G below):

llama-perplexity -m <gguf> -f wiki.test.raw -c C --chunks N
llama-bench -m <gguf> -p P -n G -r 3 -o json

For the divergence columns, the Q8_0 file first wrote reference logits for the smaller chunk count K, and then Q5_K_M and Q4_K_M were compared with it:

llama-perplexity -m <q8_0 gguf> -f wiki.test.raw -c C --chunks K --kl-divergence-base base.kld
llama-perplexity -m <gguf> --kl-divergence-base base.kld --kl-divergence

The speed figures are the mean of three llama-bench repetitions each. The capture script has the exact flags, and a second script re-derives every metric in the raw record from the saved logs.

What I did not test: an F16 baseline (the download did not finish, so Q8_0 stands in as the reference), any larger model, discrete NVIDIA or AMD GPUs, the other quants in the comparison table, Ollama or LM Studio, longer contexts, and task benchmarks or real prompts. The quality columns measure how far each quant drifts from Q8_0 on reference text, not whether its answers are correct.

The raw record, with every metric above, is at https://www.agenticwire.news/benchmarks/q4-k-m-vs-q5-k-m/results.json.

Related coverage

• LLM VRAM calculator: RAM, quantization and model fit
• KV cache size: how much VRAM it uses and how to cut it
• LM Studio vs Ollama for local AI workflows
• Ollama vs llama.cpp: which is faster for local LLMs

References

• Artefact2 KLD study - https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9
• arXiv 2601.14277 - https://arxiv.org/abs/2601.14277
• bartowski Qwen3.8-27B GGUF - https://huggingface.co/bartowski/Qwen3.8-27B-GGUF
• Hugging Face GGUF docs - https://huggingface.co/docs/hub/gguf
• llama.cpp PR #1684 - https://github.com/ggml-org/llama.cpp/pull/1684
• llama.cpp quantize tool - https://github.com/ggml-org/llama.cpp/tree/master/tools/quantize