KV Cache Size: Formula, Calculator, and VRAM by Model

KV cache size is 2 x layers x kv_heads x head_dim x context x batch x bytes_per_element. I checked that formula against what llama.cpp and Ollama actually allocated for Qwen2.5 Coder 7B on my M1: at f16 it matched to the MiB at every context length I tried (224, 896, and 1,792 MiB), while q8_0 and q4_0 caches came out 1.0625 and 1.125 times the prediction because of a small per-block scale. For Llama 3.1 8B running an 8,192-token (8k) context in f16, the formula gives 1 GiB of VRAM on top of the model weights, growing to 16 GiB at the model's full 128k context. If you need it smaller, quantize the cache to q8_0 first: it cut the cache from 896 to 476 MiB at 16,384 tokens in my run, and it is a runtime flag, not a different model. Shorter context and a model with fewer KV heads are the next levers.

The KV cache stores the key and value vectors a transformer computed for every token already processed, so attention does not recompute them on each new token, as the not-lain KV caching explainer on Hugging Face describes. Without it, generating token 1,000 would require redoing the attention math for all 999 prior tokens. That is also why cache size scales with context length: the longer the conversation, the more vectors sit in VRAM.

The KV cache size formula

The full formula is bytes = 2 x layers x kv_heads x head_dim x context x batch x bytes_per_element. Each term is a fixed model or serving choice:

• 2 accounts for storing both the key and value tensors.
• layers is the transformer's decoder block count (num_hidden_layers in the model config).
• kv_heads is the number of key-value attention heads, not the number of query heads.
• head_dim is the dimensionality of each attention head, usually hidden_size / num_attention_heads.
• context is the number of tokens cached (prompt plus generated so far).
• batch is the number of concurrent sequences the server holds in memory.
• bytes_per_element is set by the cache data type: 2 bytes for f16/bf16, roughly 1 byte for q8_0 or fp8, roughly 0.5 bytes for q4_0. In llama.cpp the real figures for q8_0 and q4_0 are 1.0625 and 0.5625 bytes, as the test further down shows.

The term most people get wrong is kv_heads. Grouped-query attention (GQA) lets several query heads share one key-value head, so kv_heads is smaller than the model's total attention head count, not equal to it. The Llama 3.1 8B config has 32 query heads but only 8 key-value heads, a 4:1 grouping ratio; Qwen2.5-7B goes further, at 28 query heads to 4 key-value heads, a 7:1 ratio. Plugging Llama 3.1 8B's real numbers in: 2 x 32 layers x 8 kv_heads x 128 head_dim x 2 bytes gives 131,072 bytes, or 128 KiB, per cached token at f16. Multiply that by context length and batch size and you get the table below.

KV cache size by model and context length

Every cell is calculated from the formula above and each model's official config. The table uses batch 1; multiply any cell by your batch size. I checked the Qwen2.5-7B shape against real allocations in the next section.

Calculated from 2 x layers x kv_heads x head_dim x context x bytes_per_element; the q8_0/q4_0 columns use 1 and 0.5 bytes per element and ignore per-block scale overhead. Multiply them by 1.0625 and 1.125 for what llama.cpp actually allocates.

At batch 4, Llama 3.1 8B's 128k row alone becomes 64 GiB of cache in f16, before the model's own weights are loaded. Qwen2.5-7B's smaller kv_heads count keeps its cache at roughly 44% of Llama 3.1 8B's size at every context length (57,344 versus 131,072 bytes per token), purely from architecture.

I checked the formula against llama.cpp and Ollama

A formula is only useful if the server agrees with it, so I loaded qwen2.5-coder:7b, which has the same attention shape as the Qwen2.5-7B rows above (28 layers, 4 KV heads, head dimension 128), and read the llama_kv_cache: size = ... line each runtime prints at load time. The predicted column comes from the same kvCacheBytes() function the calculator on this page runs.

The f16 rows match exactly, and the 32k row is the same 1.75 GiB the table above gives for Qwen2.5-7B. The quantized rows do not, and the gap is not noise. llama.cpp stores a q8_0 cache in blocks of 32 values, and each block carries a 2-byte scale next to its 32 one-byte values, so the real cost is 1.0625 bytes per element, not 1. A q4_0 block packs the same 32 values into 16 bytes plus the same 2-byte scale, which is 0.5625 bytes per element, not 0.5. Dividing my q8_0 and q4_0 cache sizes by the f16 size and multiplying by 2 gives exactly those two numbers.

Ollama 0.35.1 allocated the same bytes as standalone llama.cpp for the same settings, which makes sense because it starts a bundled llama-server and the log line is identical. The calculator's q8_0 and q4_0 options still use the round 1 and 0.5 figures, so treat their output as the floor and multiply by 1.0625 or 1.125 for a llama.cpp or Ollama cache.

One thing the test did not show is a smaller number in ollama ps. It reported 5.9 GB for the model with an f16 cache and 5.9 GB again with a q8_0 cache at the same 16,384-token context, so I would read the server log, not the ollama ps size column, when you want to know what the cache itself costs.

KV cache size calculator

The table above covers two models at four context lengths. For anything else, the KV cache size calculator at the top of this page runs the exact same formula on your own numbers instead of making you find a spreadsheet.

Pick a model preset or enter its architecture by hand: transformer layers, KV heads, and head dimension, all read from a model's published config.json the same way this article's table was built. Set context length, batch size, and cache precision, and the tool returns the cache size in GiB or MiB, the bytes held per cached token, and what the same settings would cost at f16 for comparison. It flags when your requested context exceeds the model's own trained window, since extending past that point is a different claim than simply paying for it.

Why your KV cache is bigger than the formula says

The formula gives the theoretical minimum, and there are two separate reasons a server reports more.

The first is the cache type. On llama.cpp and Ollama an f16 cache matched the formula exactly in my test, so if you run f16 and still see a bigger cache line, check the context and parallel-slot settings before anything else. A q8_0 or q4_0 cache is 1.0625 or 1.125 times the round-number prediction because of the per-block scale described above. That is 28 MiB at 16,384 tokens for this model, and it grows linearly with context. Also keep the cache line apart from the rest of the log: llama.cpp prints a separate compute buffer, and that memory is not KV cache.

The second is allocation strategy. Some servers reserve memory in fixed-size blocks rather than growing the cache one token at a time. The PagedAttention paper found that before it, systems like Orca and FasterTransformer used only 20.4% to 38.2% of allocated KV cache memory for actual token states, wasting 61.8% to 79.6% to internal fragmentation, external fragmentation, and over-reservation for sequences that never reached their max length.

PagedAttention fixed this by managing the cache in fixed-size, non-contiguous blocks, the same idea as OS virtual memory paging, and vLLM implements it by default. llama.cpp and Ollama instead pre-allocate a contiguous cache sized to --ctx-size / OLLAMA_CONTEXT_LENGTH at load time, so their allocation matches the formula (to the MiB at f16 in my run) but does not shrink between requests (see the llama.cpp server README and the Ollama FAQ). A server reporting more memory than the table predicts usually means block scales, block granularity, or pre-allocation, not a bug.

That allocation difference is part of why vLLM and Ollama suit different workloads; the vLLM vs Ollama comparison covers when many concurrent users make vLLM the better choice.

Should you offload KV cache to GPU memory?

The cache belongs in GPU memory whenever your workload is latency-sensitive, because reading it from system RAM or NVMe on every token adds a round trip attention cannot hide. Offloading pays off in a narrower case: long-lived sessions with reusable prefixes, where keeping cold cache blocks in CPU RAM and reusing them on a cache hit costs less than recomputing them.

The vLLM docs describe automatic prefix caching, which keeps a hash-indexed cache of shared prompt prefixes so repeated system prompts and few-shot examples do not get recomputed. Priority-based eviction goes further, letting a developer pin a system prompt's cache blocks at maximum retention priority so they survive memory pressure longer than a plain least-recently-used policy would allow. Offloading destroys tokens-per-second the moment a needed block was evicted to slower storage and must be recomputed before generation resumes.

The rule I would use: offload for high cache-hit workloads like chat with repeated system prompts, and keep everything resident in VRAM for single-pass workloads like one-off summarization.

Four ways to shrink it, ranked by cost to quality

Cache quantization is the safest first move because it is a runtime flag with no architecture change. Budget for slightly less than the full halving: the per-block scale means q8_0 costs 1.0625 bytes per value rather than 1, and q4_0 costs 0.5625 rather than 0.5. On Ollama, q8_0 also needs flash attention on; I set OLLAMA_FLASH_ATTENTION=1 alongside OLLAMA_KV_CACHE_TYPE=q8_0 for the run above. Choosing a GQA model with fewer key-value heads saves more in absolute terms but means picking a different checkpoint, not a flag.

The flags that actually change it

Each fix below is documented behavior in that project's own docs.

On Ollama these settings are environment variables such as OLLAMA_KV_CACHE_TYPE and OLLAMA_CONTEXT_LENGTH; the Ollama commands reference lists them alongside the CLI.

KV cache vs context window, prompt caching, and prefix caching

These terms describe overlapping but distinct things, and mixing them up is the fastest way to misdiagnose a memory problem.

• KV cache: the underlying mechanism, the stored key-value tensors for tokens already processed, present in every transformer inference run regardless of server.
• Prompt caching: a hosted-API feature (as in the Anthropic and OpenAI APIs) that reuses a previously computed prefix across separate API calls to cut cost and latency; it is a product feature built on top of KV cache reuse.
• Prefix caching: the self-hosted equivalent. vLLM's automatic prefix caching hashes shared prompt prefixes across requests on the same server so a repeated system prompt is not recomputed.
• Context window: a limit measured in tokens, while the KV cache is the memory those tokens occupy.

The context window is the one most often confused with cache size. A 128k context window is a ceiling the model was trained to handle; the KV cache is the bill you pay in GiB for actually filling it, which is why two models with the same advertised context window can have wildly different memory footprints. Raising --ctx-size or --max-model-len does not allocate more context so much as it allocates more cache.

If your bill is the problem, look at prompt caching. If your GPU is out of memory, the KV cache formula above is what to size. For the hosted-API cost side, see the guide to reducing costs with prompt caching.

KV cache vs paged attention

Paged attention is not a competing cache; it is how a server manages the cache the formula above sizes. Standard allocation reserves one contiguous block per sequence up to its maximum length, which is why pre-PagedAttention systems wasted 61.8% to 79.6% of allocated KV cache memory on fragmentation and over-reservation. Paged attention splits that same cache into fixed-size, non-contiguous blocks, so a short sequence stops paying for a long sequence's worst case. It changes how efficiently the bytes from the formula get packed into VRAM, not how many bytes the formula produces.

KV cache vs flash attention

Flash attention optimizes the attention computation itself, not the cache's stored size. It restructures the matrix-multiply-and-softmax steps to avoid materializing the full attention matrix in memory, cutting the working memory and time an attention pass needs mid-computation; llama.cpp exposes it as the --flash-attn flag. The KV cache persists between tokens; flash attention's savings exist only during the attention operation itself. Turning flash attention on does not shrink a single row of the GiB table above, though most servers enable it by default because it makes the same cache faster to read.

How I tested this

I ran this on October 4, 2026, on a MacBook Air with an M1 and 8 GB of unified memory, with desktop apps open (the exact macOS build is in the raw record). Runtimes were llama.cpp build b11351 (the official llama-b11351-bin-macos-arm64 release) and Ollama 0.35.1.

The model was qwen2.5-coder:7b from the Ollama library, a Q4_K_M GGUF. llama.cpp read 28 layers, 4 KV heads, and a head dimension of 128 from its metadata, the same attention shape as Qwen2.5-7B. The weight quantization does not enter the cache formula; only the cache type does.

For llama.cpp I loaded the Ollama blob directly, CPU-only so the full cache is allocated regardless of GPU fit, and generated 4 tokens per configuration:

llama-completion -m <gguf> -c N -ngl 0 -fa on -ctk T -ctv T -n 4 -p Hello -no-cnv --temp 0 --log-verbosity 4

N and T covered the five llama.cpp rows in the table above: three context lengths at f16, then the middle context with q8_0 and q4_0. For Ollama I restarted the server with OLLAMA_CONTEXT_LENGTH set to that same middle context, then again with OLLAMA_KV_CACHE_TYPE=q8_0 and OLLAMA_FLASH_ATTENTION=1, sent one request each, and saved the llama_kv_cache line from the server log plus the ollama ps output. Predictions came from the calculator's own kvCacheBytes() function with the architecture numbers llama.cpp printed.

What I did not test: vLLM (its paged block allocation and --kv-cache-dtype fp8 are described from its docs, not measured), NVIDIA or other discrete GPUs, the Llama rows or any other architecture, batch sizes or parallel slots above 1, contexts longer than the largest one in the table, and the speed or output-quality effect of a quantized cache. Allocation size is all this run measures.

The raw record, with every metric above, is at https://www.agenticwire.news/benchmarks/kv-cache-size/results.json.

FAQ

Does ChatGPT use KV cache?

OpenAI has not published ChatGPT's serving internals, but every transformer-based autoregressive model needs some form of key-value caching to generate text at usable speed, and OpenAI's API documents a related prompt-caching feature that reuses cached prefixes across calls. That is why I'd say ChatGPT almost certainly uses KV caching internally, though it is an architectural necessity rather than a confirmed implementation detail.

Is Redis a KV cache?

No. Redis is a key-value database, a general-purpose store for arbitrary application data addressed by key. The KV cache in this article is a transformer-specific structure holding attention key and value tensors in GPU memory during inference. The shared term "key-value" is a naming coincidence, not a shared system.

What happens when KV cache is full?

The result depends on the serving engine. vLLM can fail memory profiling or refuse a request when its allocated blocks cannot hold the configured context and concurrent sequences; another runner may reduce concurrency or offload cache. Lower maximum context or parallel requests, or choose a supported lower-precision cache, then measure again. The formula estimates tensor bytes, not every allocator reservation.

Related coverage

• LLM VRAM calculator: will a model fit your GPU
• Running a local LLM on 8GB of RAM
• Ollama vs llama.cpp
• GPU cost per million tokens

References

• llama.cpp server README - https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• Meta Llama 3.1 8B config - https://huggingface.co/NousResearch/Meta-Llama-3.1-8B
• not-lain KV caching explainer - https://huggingface.co/blog/not-lain/kv-caching
• Ollama docs (FAQ) - https://github.com/ollama/ollama/blob/main/docs/faq.mdx
• PagedAttention / vLLM paper (Kwon et al.) - https://arxiv.org/abs/2309.06180
• vLLM docs (engine args + conserving memory) - https://docs.vllm.ai/en/latest/configuration/engine_args.html