KV Cache Size: How Much VRAM It Uses and How to Cut It

KV cache size is 2 x layers x kv_heads x head_dim x context x batch x bytes_per_element, and for Llama 3.1 8B running an 8,000-token context in f16 that works out to 1 GiB of VRAM on top of the model weights, growing to 16 GiB at the model's full 128k context. Every serving stack from llama.cpp to vLLM to Ollama exposes a flag to cut that number, and which flag to reach for depends on whether you are trading precision, context length, or model architecture.

The KV cache stores the key and value vectors a transformer computed for every token already generated, so attention does not recompute them on each new token (Source: not-lain KV caching explainer). That is why cache size scales with context length: the longer the conversation, the more vectors sit in VRAM. Most explainers stop at that mechanism. This page starts from the formula, applies it to real model configs, and ends at the exact server flag that resolves an out-of-memory error.

The KV cache size formula

The full formula is bytes = 2 x layers x kv_heads x head_dim x context x batch x bytes_per_element. Each term is a fixed model or serving choice:

• 2 accounts for storing both the key and value tensors.
• layers is the transformer's decoder block count (num_hidden_layers in the model config).
• kv_heads is the number of key-value attention heads, not the number of query heads.
• head_dim is the dimensionality of each attention head, usually hidden_size / num_attention_heads.
• context is the number of tokens cached (prompt plus generated so far).
• batch is the number of concurrent sequences the server holds in memory.
• bytes_per_element is set by the cache data type: 2 bytes for f16/bf16, roughly 1 byte for q8_0 or fp8, roughly 0.5 bytes for q4_0.

The term most people get wrong is kv_heads. Grouped-query attention (GQA) lets several query heads share one key-value head, so kv_heads is smaller than the model's total attention head count, not equal to it. Llama 3.1 8B has 32 query heads but only 8 key-value heads, a 4:1 grouping ratio; Qwen2.5-7B goes further, at 28 query heads to 4 key-value heads, a 7:1 ratio (Source: Meta Llama 3.1 8B config). Plugging Llama 3.1 8B's real numbers in: 2 x 32 layers x 8 kv_heads x 128 head_dim x 2 bytes gives 131,072 bytes, or 128 KiB, per cached token at f16. Multiply that by context length and batch size and you get the table below.

KV cache size by model and context length

Every cell is calculated from the formula above and each model's official config, not benchmarked on hardware. Multiply any cell by your batch size; the table below uses batch 1.

calculated from 2 x layers x kv_heads x head_dim x context x bytes_per_element; q8_0/q4_0 are architectural approximations that ignore small per-block scale overhead.

At batch 4, Llama 3.1 8B's 128k row alone becomes 64 GiB of cache in f16, before the model's own weights are loaded. Qwen2.5-7B's smaller kv_heads count keeps its cache roughly a third the size of Llama 3.1 8B's at every context length, purely from architecture (Source: Meta Llama 3.1 8B config).

Why your KV cache is bigger than the formula says

The formula gives the theoretical minimum. Real servers allocate more, because they reserve memory in fixed-size blocks rather than growing the cache one token at a time. Before PagedAttention, systems like Orca and FasterTransformer used only 20.4% to 38.2% of allocated KV cache memory for actual token states, wasting 61.8% to 79.6% to internal fragmentation, external fragmentation, and over-reservation for sequences that never reached their max length (Source: PagedAttention paper).

PagedAttention fixed this by managing the cache in fixed-size, non-contiguous blocks, the same idea as OS virtual memory paging, and vLLM implements it by default (Source: PagedAttention paper). llama.cpp and Ollama instead pre-allocate a contiguous cache sized to --ctx-size / OLLAMA_CONTEXT_LENGTH at load time, so their allocation matches the formula but does not shrink between requests (Source: llama.cpp server README; Ollama docs). A server reporting more memory than the table predicts usually means block granularity or pre-allocation, not a bug.

Should you offload KV cache to GPU memory?

The cache belongs in GPU memory whenever your workload is latency-sensitive, because reading it from system RAM or NVMe on every token adds a round trip attention cannot hide. Offloading pays off in a narrower case: long-lived sessions with reusable prefixes, where keeping cold cache blocks in CPU RAM and reusing them on a cache hit costs less than recomputing them.

vLLM's automatic prefix caching keeps a hash-indexed cache of shared prompt prefixes so repeated system prompts and few-shot examples do not get recomputed (Source: vLLM docs). Priority-based eviction goes further, letting a developer pin a system prompt's cache blocks at maximum retention priority so they survive memory pressure longer than a plain least-recently-used policy would allow. Offloading destroys tokens-per-second the moment a needed block was evicted to slower storage and must be recomputed before generation resumes. Decision rule: offload for high cache-hit workloads like chat with repeated system prompts, keep everything resident in VRAM for single-pass workloads like one-off summarization.

Four ways to shrink it, ranked by cost to quality

Cache quantization is the safest first move because it is a runtime flag with no architecture change (Source: llama.cpp server README; vLLM docs). Choosing a GQA model with fewer key-value heads saves more in absolute terms but means picking a different checkpoint, not a flag (Source: Meta Llama 3.1 8B config).

The flags that actually change it

Each of these is documented behavior in that project's own docs, not a workaround discovered by trial and error (Source: llama.cpp server README; vLLM docs; Ollama docs).

KV cache vs prompt caching vs prefix caching

These three terms describe overlapping but distinct things, and mixing them up is the fastest way to misdiagnose a memory problem. KV cache is the underlying mechanism: the stored key-value tensors for tokens already processed, present in every transformer inference run regardless of server (Source: not-lain KV caching explainer). Prompt caching is a hosted-API feature (as in the Anthropic and OpenAI APIs) that reuses a previously computed prefix across separate API calls to cut cost and latency; it is a product feature built on top of KV cache reuse. Prefix caching is the self-hosted equivalent: vLLM's automatic prefix caching hashes shared prompt prefixes across requests on the same server so a repeated system prompt is not recomputed (Source: vLLM docs). If your bill is the problem, look at prompt caching. If your GPU is out of memory, the KV cache formula above is what to size. For the hosted-API cost side of this, see AgenticWire's guide to reducing costs with prompt caching.

FAQ

What is KV cache for?

KV cache stores the key and value vectors a transformer computes for each token it has already processed, so attention does not recompute them for every new token. Without it, generating token 1,000 would require redoing the attention math for all 999 prior tokens, making long outputs far slower (Source: not-lain KV caching explainer).

Does ChatGPT use KV cache?

OpenAI has not published ChatGPT's serving internals, but every transformer-based autoregressive model needs some form of key-value caching to generate text at usable speed, and OpenAI's API documents a related prompt-caching feature that reuses cached prefixes across calls. Inference: ChatGPT almost certainly uses KV caching internally; this is architectural necessity, not a confirmed implementation detail.

Is Redis a KV cache?

No. Redis is a key-value database, a general-purpose store for arbitrary application data addressed by key. The KV cache in this article is a transformer-specific structure holding attention key and value tensors in GPU memory during inference. The shared term "key-value" is a naming coincidence, not a shared system.

Should I offload KV cache to GPU memory?

Keep the cache in GPU memory for latency-sensitive or single-pass workloads; offload cold cache blocks to CPU RAM only for long-lived sessions with reusable prefixes, where a cache hit avoiding recomputation outweighs the round trip. Offloading a block that then gets evicted before reuse costs more than never offloading it.

What is the difference between KV cache and prefix cache?

KV cache is the general mechanism storing key-value tensors for processed tokens. Prefix caching is a specific optimization that indexes and reuses KV cache blocks across requests sharing an identical prompt prefix, so a repeated system prompt is computed once instead of on every request (Source: vLLM docs).

Related coverage

• LLM VRAM calculator: will a model fit your GPU
• Running a local LLM on 8GB of RAM
• Ollama vs llama.cpp
• GPU cost per million tokens

References

• llama.cpp server README - https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• Meta Llama 3.1 8B config - https://huggingface.co/NousResearch/Meta-Llama-3.1-8B
• not-lain KV caching explainer - https://huggingface.co/blog/not-lain/kv-caching
• Ollama docs (FAQ) - https://github.com/ollama/ollama/blob/main/docs/faq.mdx
• PagedAttention / vLLM paper (Kwon et al.) - https://arxiv.org/abs/2309.06180
• vLLM docs (engine args + conserving memory) - https://docs.vllm.ai/en/latest/configuration/engine_args.html