vLLM vs SGLang: Throughput, Latency, When to Use Each

vLLM and SGLang are both open source engines for serving large language models in production, and the honest answer is that neither wins outright. vLLM optimizes for fast startup, broad hardware support, and steady throughput across mixed traffic. SGLang optimizes for workloads that reuse the same prompt prefix again and again: retrieval-augmented generation, agent scaffolding, and multi-turn chat. The benchmark that gets quoted most right now reports SGLang around 16,200 tokens per second against vLLM's roughly 12,500 on an 8B model, a 29% gap, but that specific figure traces back to unlinked third-party testing relayed through a secondary source, so treat it as reported rather than independently verified (Source: Particula Tech). At larger model sizes with unique, non-repeating prompts, independent benchmarks show that gap narrow to single digits or vanish entirely.

Key takeaways:

• SGLang's RadixAttention lead is real on shared-prefix traffic (RAG, agents, chat history) and shrinks toward zero when every prompt is unique.
• vLLM wins on breadth: more hardware targets, a larger contributor base, and faster onboarding for a brand-new model.
• The widely repeated 29% throughput gap comes from unlinked secondary benchmarks; treat it as directional, not verified.
• Under real saturation load, vLLM has measured a faster time-to-first-token than SGLang on unique-prompt traffic.

What vLLM and SGLang Actually Are

vLLM is an open source inference and serving library that started at UC Berkeley's Sky Computing Lab and now has contributions from more than 2,000 developers. It supports over 200 model architectures across NVIDIA and AMD GPUs, CPUs, TPUs, and accelerators from Intel, IBM, and Huawei (Source: vLLM docs). Its core technique, PagedAttention, manages the GPU memory used for attention key-value caches the way an operating system manages virtual memory: in fixed-size pages instead of one contiguous block per request, so memory does not sit wasted when requests finish at different times.

SGLang is a serving framework hosted by LMSYS, the non-profit behind the original Chatbot Arena. The project states it powers production deployments generating trillions of tokens a day across more than 400,000 GPUs (Source: SGLang docs). Its core technique, RadixAttention, stores the KV cache in a tree structure keyed on prompt content, so when two requests share a prefix, such as a system prompt, a RAG document, or the first half of a chat thread, SGLang reuses the cached computation instead of redoing it. The project's own GitHub repository claims RadixAttention delivers "up to 5x faster inference" on workloads where that reuse applies (Source: SGLang GitHub).

Throughput and Latency: The Comparison Table

Benchmark numbers for vLLM and SGLang vary widely because model size, prompt structure, and hardware all shift the result, and no two public benchmarks share a setup. The table below lines up three real test conditions so you can see where the gap is large, where it is marginal, and where it flips.

The 16,200-vs-12,500 row is the one most often repeated online, but it originates from unlinked third-party testing (attributed to Prem AI and LocalAIMaster) relayed through a secondary blog with no published methodology, so read it as an estimate (Source: Particula Tech). The 70B and saturation-load rows come from benchmarks that publish their hardware, driver, and framework versions, which is why they carry more weight here (Source: Spheron benchmark) (Source: ComputingForGeeks).

Why the Gap Shrinks or Flips by Workload

RadixAttention's advantage is conditional: it only pays off when requests share a prefix. Spheron's own benchmark, run on a single H100 with Llama 3.3 70B in FP8 precision and unique prompts throughout, found SGLang ahead by only about 2.5% at 100 concurrent requests. The benchmark's own methodology note explains why: "our benchmark used unique prompts throughout, so you are seeing the baseline behavior" (Source: Spheron benchmark). Remove the shared prefix and RadixAttention does roughly the same work PagedAttention does.

Under heavier saturation load, the result can flip. ComputingForGeeks ran SGLang 0.5.15 against vLLM 0.25.1 at 64 concurrent unique-prompt requests and measured vLLM's median time-to-first-token at 849 milliseconds against SGLang's 3,045 milliseconds, more than three times slower to first token, even though SGLang's per-token decode speed stayed slightly ahead at 29.1 milliseconds versus vLLM's 34.0 (Source: ComputingForGeeks). Writing up that same saturation run, ComputingForGeeks author Josphat Mutai put the tradeoff plainly: "vLLM reaches the first token far sooner," while "SGLang flips the result on per-token decode speed" (Source: ComputingForGeeks). That is the practical version of the tradeoff: vLLM gets the first token out faster under load, and SGLang decodes each following token a little faster once it starts.

Memory, Cold Starts, and Deployment Complexity

KV cache footprint and startup time matter as much as raw throughput once either engine is running in production. In the ComputingForGeeks test, SGLang allocated a smaller KV cache (113,739 tokens) than vLLM (121,968 tokens) on identical hardware, and cold-started faster too: 58 seconds against vLLM's 79 (Source: ComputingForGeeks). If workers restart often, autoscaling, spot instances, frequent redeploys, that gap compounds across a fleet.

Operator note (first-hand): the flag that actually decides this for me is not a benchmark chart, it is the launch command. vllm serve <model> --enable-prefix-caching --max-num-seqs 256 gets a mixed-traffic endpoint running with sane defaults in under two minutes. SGLang's equivalent, python -m sglang.launch_server --model-path <model> --mem-fraction-static 0.85, asks you to tune the KV-cache memory fraction up front, a config choice that rewards already knowing your prefix-sharing rate. If I cannot yet describe how much of my traffic reuses a prefix, I start on vLLM and switch once I can measure it.

When to Use Each: The vLLM vs SGLang Decision Rule

The vLLM vs SGLang decision comes down to one question: does most of your traffic share a prefix? RAG pipelines that re-inject the same retrieved context, agent frameworks that resend the same system prompt and tool schema every turn, and multi-turn chat carrying growing history are all shared-prefix workloads, and that is where RadixAttention earns its keep (Source: SGLang GitHub).

If your traffic looks more like one-off completions, batch scoring jobs, or you need same-day support for a newly released model architecture, vLLM's broader hardware support and larger contributor base make it the safer default (Source: vLLM docs). Teams running both engines behind a gateway increasingly treat this as a routing decision instead of a one-time platform pick: shared-prefix traffic goes to SGLang, everything else goes to vLLM, with a router in front handling the split. For a three-way view that also covers TensorRT-LLM's throughput ceiling at high concurrency, see the vendor benchmark comparing all three engines (Source: Particula Tech).

FAQ

Why is SGLang better than vLLM?

SGLang is not universally better. Its RadixAttention design caches repeated prompt prefixes, so workloads that reuse a system prompt, RAG context, or chat history see real throughput and latency gains. On workloads where every prompt is unique, that advantage shrinks toward zero or reverses, because there is nothing left to cache (Source: Spheron benchmark).

What is better than vLLM?

For shared-prefix-heavy traffic such as RAG, agents, and multi-turn chat, SGLang typically posts higher throughput and can decode faster once generation starts. For broad hardware support, the largest contributor base, and same-day support for new model architectures, vLLM stays the safer default. Neither replaces the other outright (Source: vLLM docs).

Is SGLang faster than vLLM?

Sometimes. Independent tests report SGLang ahead on 8B models with shared-prefix traffic and roughly even at 70B with unique prompts. Under saturation load with unique prompts, vLLM measured a faster time-to-first-token while SGLang held a small per-token decode edge, so "faster" depends on the metric and workload (Source: ComputingForGeeks).

When should I use SGLang vs vLLM?

Use SGLang when most requests share a prefix: RAG pipelines re-injecting the same context, agents resending the same system prompt, or long multi-turn chats. Use vLLM for one-off completions, batch scoring, mixed unique-prompt traffic, or when you need a newly released model supported immediately (Source: SGLang GitHub).

Does SGLang support the same models as vLLM?

Mostly, but not identically. Both support major open-weight families like Llama, Qwen, and DeepSeek, and both expose OpenAI-compatible APIs. vLLM's architecture list runs past 200 with broader accelerator coverage (TPU, Trainium, Gaudi); SGLang's list is narrower but treats DeepSeek-family models as a fast-tracked, first-class target (Source: SGLang docs).

Related coverage

• Ollama vs vLLM: Speed, Cost, and Which to Use for Local Models
• LiteLLM vs OpenRouter: Self-Hosted Proxy or Managed Gateway
• LiteLLM vs Portkey: Which AI Gateway to Self-Host
• Local LLM agentic coding spikes on r/LocalLLaMA

References

• ComputingForGeeks - https://computingforgeeks.com/sglang-vs-vllm/
• Particula Tech - https://particula.tech/blog/sglang-vs-vllm-inference-engine-comparison
• SGLang docs - https://docs.sglang.io/
• SGLang GitHub - https://github.com/sgl-project/sglang
• Spheron benchmark - https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
• vLLM docs - https://docs.vllm.ai/en/latest/