vLLM vs Ollama: Which to Run for Local Dev or Serving

Run Ollama if you are one developer or a small team working on a laptop or workstation, including Windows PCs and Apple Silicon Macs. Run vLLM if you are serving many concurrent users or agents from a Linux GPU server. That is the vLLM vs Ollama decision in one line. When I tried to install vLLM on my 8 GB M1 MacBook Air, both the plain uv pip install vllm route and the vLLM-Metal installer failed before vLLM served a single request, which is part of why I still point Mac users to Ollama. Ollama optimizes for setup time and, per the Ollama docs, runs natively on Windows, macOS, and Linux. vLLM optimizes for throughput under load, and the gap grows with concurrency: in Red Hat's benchmark on one A100, vLLM peaked at 793 output tokens per second against Ollama's 41.

What is vLLM, and how is it different from Ollama?

vLLM is a free, open-source Python library for LLM inference and serving, built around PagedAttention memory management and continuous batching of incoming requests. The vLLM README says it was originally developed in the Sky Computing Lab at UC Berkeley, and it is Apache-2.0 licensed. Ollama is a local model runner with a CLI, a desktop app, and a REST API that downloads and serves models with one command. Both expose an OpenAI-compatible endpoint, so application code written against one usually moves to the other by changing the base URL and model name.

The vLLM column follows the vLLM GPU installation guide, the Ollama column the Ollama docs. Checked with vLLM v0.30.0 (released September 22, 2026) and Ollama v0.35.0 (released September 28, 2026), the latest stable releases on September 30, 2026.

Is vLLM faster than Ollama?

Yes, under concurrent load, and by a wide margin. For one user at a time the difference matters much less. The cleanest public test is Red Hat's. It ran vLLM 0.9.1 with meta-llama/Llama-3.1-8B-instruct and Ollama 0.9.2 with llama3.1:8b-instruct-fp16, both on a single NVIDIA A100-PCIE-40GB, with the GuideLLM load tool driving 1 to 256 concurrent users for 300 seconds per run. The table below reports their numbers next to a second public benchmark. None of the figures are my own measurements.

These rows come from different machines, models, and tools, so I have not averaged them or compared them with each other. In McDermott's benchmark, Ollama climbed to about 22 requests per second at 32 concurrent requests and then stayed flat; adding requests beyond that only raised latency.

There is one trade-off worth knowing. At default settings, Red Hat saw vLLM's inter-token latency rise above 16 concurrent users while Ollama's stayed low. That happened because Ollama was making most requests wait in a queue, which shows up as a much higher time to first token.

How concurrency works in each engine

The speed gap is mostly a scheduling gap. Ollama gives each loaded model a fixed number of parallel slots, set by OLLAMA_NUM_PARALLEL. The current documented default is 1. Requests beyond that queue up, and once the queue passes OLLAMA_MAX_QUEUE (default 512) the server returns HTTP 503.

Raising the slot count is not free. Ollama's docs say memory scales with OLLAMA_NUM_PARALLEL multiplied by OLLAMA_CONTEXT_LENGTH, so a 2K context with 4 parallel slots allocates an 8K context. Red Hat found 32 to be the highest stable value on a 40GB A100, and beyond the throughput ceiling, tuned Ollama's inter-token latency "became extremely erratic" at higher concurrency.

vLLM has no fixed slot count. It batches incoming requests continuously, and PagedAttention manages the attention key and value memory in fixed-size blocks rather than one large reservation per request. By default vLLM claims 92% of GPU memory at startup (gpu_memory_utilization=0.92) and uses what the weights leave over for KV cache. The KV cache size calculator shows how many concurrent sequences that space holds for a given model and context.

Older articles often say Ollama allows four parallel requests by default. That matched the version Red Hat tested, but today's docs list 1. Check yours before you benchmark.

Can you run vLLM on Windows or a Mac?

On Windows, not natively. The vLLM installation guide lists Linux as the supported operating system and says: "vLLM does not support Windows natively." The official routes are Windows Subsystem for Linux (WSL) with a compatible Linux distribution, or a community-maintained fork such as SystemPanic/vllm-windows. Ollama, by contrast, installs as a native Windows app on Windows 10 22H2 or newer, with NVIDIA and AMD Radeon GPU support.

On a Mac, the answer has changed since late 2025, and many ranking pages still say vLLM cannot use Apple GPUs. vLLM's docs now route Apple Silicon GPU inference to vLLM-Metal, a community-maintained plugin that uses Apple's MLX framework as the compute backend. It serves the same OpenAI-compatible API and works with MLX-format models from the mlx-community Hugging Face organization. The plain macOS CPU build is still marked experimental and supports FP32 and FP16 only. PyPI has no macOS wheel for it, although the vLLM-Metal installer fetched a prebuilt macOS arm64 CPU wheel from vLLM's GitHub release when I ran it (details below).

The two sources also disagree on the minimum OS: vLLM's install page says macOS Sonoma or later, while the vLLM-Metal README asks for macOS 15 Sequoia.

Ollama runs on Apple M-series chips with CPU and GPU support through Metal, and on Intel Macs in CPU-only mode. On a Mac laptop I still think Ollama is the lower-friction choice. vLLM-Metal is worth testing mainly if your production target is vLLM and you want the same server locally.

What happened when I tried vLLM on a Mac

I gave vLLM a fixed timebox on a MacBook Air with an M1 and 8 GB of memory, trying the generic install first and then the documented Apple Silicon route. Neither got as far as vllm serve.

The first failure is a local toolchain problem. The build log has 8 "fatal error" lines for missing C++ standard headers (cstddef, and once map), and vLLM's macOS installation docs list that exact error with a fix: remove and reinstall the Xcode Command Line Tools. That needs admin rights, and I did not try it, so I can't say whether the source build would have finished afterwards.

The second and third failures were my network, not vLLM. The installer picked a compatible wheel and resolved its dependencies without complaint; it simply could not download llvmlite (pulled in by numba) over a slow connection in the time allowed. On a faster line I would expect that route to get further, but I have not seen it serve a request.

So I have no vLLM speed number for this Mac, and this page makes no vLLM vs Ollama throughput comparison on Apple Silicon. Ollama, on the same laptop, installs and serves small models with one command; my Ollama run record from that comparison has the numbers. What the attempt does show is the setup cost: the generic install command is a source build on macOS, and the supported route pulls a large Python dependency tree before anything runs. If you only want a local model with an OpenAI-compatible API on a Mac, I think that cost is hard to justify, because Ollama already provides one. If your production target is vLLM, budget for a reliable connection and working Command Line Tools before you start.

Model formats: GGUF vs AWQ, GPTQ, and FP8

The file you already have often decides the engine for you. Ollama is built around GGUF, the quantized model format used by the llama.cpp project. It can also import Safetensors weights through a Modelfile. It does not quantize GGUF files during import, so you prepare the quantization first; the Q4_K_M vs Q5_K_M comparison helps with choosing a GGUF level.

vLLM is built around Hugging Face checkpoints. Its documented quantization paths include AutoAWQ, GPTQModel, BitsAndBytes, and LLM Compressor schemes such as FP8 W8A8, INT4 W4A16, and INT8. It also supports NVIDIA ModelOpt, AMD Quark, TorchAO, and quantized KV cache.

GGUF on vLLM exists but is not the main path. The docs call it "highly experimental and under-optimized," say it "might be incompatible with other features," and now require installing the separate vllm-gguf-plugin. They also recommend loading the tokenizer from the base model, because converting a GGUF tokenizer is slow and unstable.

Hardware support varies by format too. vLLM's compatibility table marks FP8 W8A8 through LLM Compressor as supported on Ada and Hopper NVIDIA GPUs and AMD, but not on Ampere cards such as the A100.

My rule of thumb: if your model library is a folder of GGUF files, stay on Ollama. If you are deploying vendor-published AWQ, GPTQ, or FP8 checkpoints, vLLM is their native home.

vLLM install and Docker setup compared

Setup cost is where Ollama earns its reputation. On Linux, vLLM installs as a Python package (uv pip install vllm --torch-backend=auto) into a Python 3.10 to 3.13 environment, and vllm serve <model> starts an OpenAI-compatible server at http://localhost:8000. Each server hosts one model at a time.

For containers, vLLM publishes the vllm/vllm-openai image. The documented run commands pass --gpus all and --ipc=host and publish port 8000. Ollama's image is ollama/ollama, published on port 11434, with a --gpus=all variant for NVIDIA and a rocm tag for AMD. The container toolkit steps are in the Ollama Docker GPU setup guide.

The API difference is smaller than it looks. Ollama exposes its own REST API plus an OpenAI-compatible endpoint at http://localhost:11434/v1/. vLLM exposes OpenAI's Completions, Chat Completions, Responses, and Embeddings APIs. Code using the OpenAI SDK moves from Ollama to vLLM by changing its base URL from port 11434 to port 8000 and its model name. The real work in that switch is the model: vLLM wants a Hugging Face checkpoint, so you usually swap a GGUF file for a safetensors or AWQ version. One caution from vLLM's docs: --api-key only protects /v1, /v2, and /inference paths, so do not rely on it alone for a server exposed to a network.

Day-to-day model management on Ollama runs through ollama pull, ollama ps, and environment variables, all listed in the Ollama commands cheat sheet. For a single machine there is also a third option, llama.cpp's own server; the llama-server guide shows its OpenAI-compatible endpoints and parallel-slot settings.

Which should you run? A decision matrix

Use the row that matches your machine and user count.

The "5+ users" line is my rule of thumb, drawn from where both public benchmarks show Ollama flattening; it is not a hard threshold. If you are sizing a GPU for that load, the local LLM hardware calculator and the GPU cost per million tokens breakdown help price it.

If you are weighing Ollama against the engine it grew out of rather than against vLLM, that is a different trade-off, covered in Ollama vs llama.cpp.

How I tested this

The install attempts ran on October 5, 2026, on a MacBook Air with an Apple M1 and 8 GB of unified memory (the exact macOS build is in the raw record). Each attempt used a fresh Python virtual environment managed by uv, and each ran under a shell watchdog so it could not hang indefinitely.

• Attempt 1: a fresh venv, then uv pip install vllm, which resolved and tried to build vllm 0.11.0 from source.
• Attempt 2: the vLLM-Metal install.sh curl installer, with its virtual environment redirected into a lab folder. It selected the vLLM 0.30.0+cpu macOS arm64 wheel.
• Attempt 3: the same installer with a longer uv HTTP timeout, stopped by the watchdog before it finished.

What I did not test: any NVIDIA or other discrete GPU, the Command Line Tools reinstall that vLLM's docs suggest, a retry on a faster connection, Docker, and any throughput or latency comparison. Because vLLM never started, no request was sent and no vLLM tokens per second figure exists. Every speed number on this page comes from the Red Hat and McDermott benchmarks cited above. The raw record, with exit codes, timings, and the error lines, is at https://www.agenticwire.news/benchmarks/vllm-vs-ollama/results.json.

FAQ

Which is better, Ollama, vLLM, or SGLang?

I would put SGLang in the vLLM column, not the Ollama one: it is a GPU serving engine for production traffic, so the real choice is Ollama for local single-user work, then vLLM or SGLang for multi-user serving. Between those two, benchmark your own model and hardware. Hugging Face TGI is no longer a third option: the TGI README says it is in maintenance mode and recommends vLLM or SGLang.

Does LM Studio use vLLM?

No. The LM Studio docs say it runs models with llama.cpp on Mac, Windows, and Linux, and with Apple's MLX on Apple Silicon Macs. That makes it a desktop alternative to Ollama rather than to vLLM; the LM Studio vs Ollama comparison covers that choice in detail.

Related coverage

• Ollama vs llama.cpp: which local runtime to use
• LM Studio vs Ollama
• KV cache size calculator
• Ollama Docker GPU setup

References

• LM Studio docs - https://lmstudio.ai/docs/app
• McDermott benchmark - https://robert-mcdermott.medium.com/performance-vs-practicality-a-comparison-of-vllm-and-ollama-104acad250fd
• Ollama docs - https://docs.ollama.com/faq
• Red Hat benchmark - https://developers.redhat.com/articles/2025/08/08/ollama-vs-vllm-deep-dive-performance-benchmarking
• TGI README - https://github.com/huggingface/text-generation-inference
• vLLM docs - https://docs.vllm.ai/en/latest/getting_started/installation/gpu/
• vLLM README - https://github.com/vllm-project/vllm
• vLLM-Metal repo - https://github.com/vllm-project/vllm-metal