llama-server: Flags, API, Web UI and Docker (Tested Guide)

llama-server is the HTTP server that ships with llama.cpp. It loads a GGUF model and serves an OpenAI-compatible API plus a browser chat UI, and you start it with one command: llama-server -hf ggml-org/gemma-4-E4B-it-GGUF:Q4_0, or -m model.gguf for a local file. According to the llama.cpp server README, it listens on http://127.0.0.1:8080 by default, so the chat UI and the /v1/chat/completions endpoint are both at that address once the model loads. Despite the name, it runs most open model families, including Qwen, Gemma and Mistral, not only Meta's Llama.

I tested it on an Apple M1 with 8 GB of RAM, using the official macOS arm64 download of the stable llama.cpp v0.5.0 release (build b11146, commit 7fe450e19) and a 0.5B Qwen model. With the flags recommended below, the server answered /health in 793 ms, held 610 MB resident and decoded at a median 94.2 tokens per second. Two results are worth knowing before you deploy it: --api-key locks the API routes but leaves /health and the web UI page public, and with -c and -np unset the server chose four slots sharing the model's full 32768-token context.

The current release also exposes the server as llama serve, with identical flags, per the llama.app serve docs. In my run, llama serve --help matched llama-server --help once log lines were stripped.

Install llama-server and check the version

llama-server comes inside every llama.cpp release archive, so there is nothing separate to install. llama.cpp is the whole project: the C/C++ inference library plus a set of tools. llama-server is the tool that wraps the library with an OpenAI-compatible API, parallel slots and a web UI, and llama-cli is the terminal chat tool from the same release, sharing most model flags.

The GitHub release page ships a macOS arm64 tarball, Linux builds for CPU, CUDA, ROCm and Vulkan, and Windows zips per GPU backend, each with llama-server inside. Stable releases now use semantic versions, and each points at a nightly build tag: v0.5.0 maps to b11146.

The llama-b11146-bin-macos-arm64.tar.gz asset I downloaded was 10.7 MB, unpacked to 27 MB, and contained 22 llama* executables. ./llama-server --version printed version: 0.5.0-dev (build 11146, commit 7fe450e19), and ./llama --help listed serve cli download version licenses help.

On macOS, run xattr -dr com.apple.quarantine . in the unpacked folder before the first start. On Windows the binary is llama-server.exe in the zip for your GPU, with the same flags. If a model does not fit, pick a smaller quant with the Q4_K_M vs Q5_K_M comparison.

Start llama-server: the one command and a real one

The minimal start needs only a model. -m takes a local GGUF path, as in the first line below. -hf downloads from Hugging Face and caches the file, as in the second, with an optional :quant suffix that defaults to Q4_K_M.

llama-server -m ./qwen2.5-0.5b-instruct-q4_k_m.gguf
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF:Q4_0

For a server you keep running, set context, GPU offload, slots, an alias and a key explicitly. This is the command I tested, key redacted:

llama-server -m qwen2.5-0.5b-instruct-q4_k_m.gguf -a qwen2.5-0.5b \
  -c 4096 -ngl all -np 2 \
  --host 127.0.0.1 --port 8089 \
  --api-key "$LLAMA_API_KEY" --jinja \
  -ctk q8_0 -ctv q8_0 --metrics

That command reached a 200 on /health in 793 ms, the process held 610 MB resident, and the log printed listening on http://127.0.0.1:8089. The Qwen model decoded at a median 94.2 tokens per second over five 64-token runs, with every layer on the M1 GPU.

The serve docs add that llama serve launched with no model at all starts router mode, which loads and unloads models from the Hugging Face cache, a --models-dir folder or a --models-preset file on demand.

llama-server flags reference

Every option can also be set through an environment variable, which is how Docker setups configure the server. A command-line flag wins when both are set.

llama-server --help in build 11146 printed 731 lines with 259 option rows, and 148 of those options list an env: variable. The table covers the flags almost every setup touches, with defaults copied from that output.

Two defaults changed from older guides. --jinja is now on unless you pass --no-jinja, and -ngl defaults to auto with a --fit pass that trims unset options to fit device memory. The KV cache type trades memory for accuracy; our KV cache size guide shows how many gigabytes q8_0 saves at long context.

Parallel slots and context size

A slot is one in-flight conversation, and -np sets how many the server keeps. The llama.app docs describe the context as shared across slots, but the split you get depends on whether the KV cache is unified.

With an explicit -c 4096 -np 2, the server divided the context evenly. The log showed n_slots = 2, n_ctx_slot = 2048, kv_unified = 'false', and /slots reported 2048 tokens for each of the two slots. Two simultaneous /completion requests came back with id_slot values [0, 1], so they ran side by side.

Leaving both -c and -np unset behaved differently. llama serve -m model.gguf chose n_slots = 4, n_ctx_slot = 32768, kv_unified = 'true': the model's full trained context of 32768 tokens, in one unified pool that the four slots draw from.

If a client reports less context than the model supports, I would check -np first. Going by the even split above, four explicit slots on -c 32768 would leave each conversation 8,192 tokens.

Endpoints: the OpenAI-compatible API and native routes

The llama.app API docs describe three API families on one port: OpenAI-compatible routes under /v1, an Anthropic-compatible /v1/messages, and native llama.cpp routes with extra options.

The status codes come from my run; the purpose column comes from the docs. The README also lists /v1/completions, /v1/responses, /v1/embeddings, /v1/rerank and /infill, which I did not call.

A temperature-0 chat request returned Paris with system_fingerprint b11146-7fe450e19, 38 prompt tokens, and a llama.cpp timings object alongside the standard OpenAI usage block. /metrics exposed 15 series such as llamacpp:prompt_tokens_total, and without -a the model id was the full file path.

The built-in web UI

Open http://127.0.0.1:8080 in a browser and llama-server serves its own chat interface, enabled by default and switched off with --no-ui. It handles plain chat, and with a multimodal model you can drag images into the conversation. The screenshot below was taken in a separate session on a newer build, b11344, as its caption says; every measurement on this page comes from the recorded b11146 run.

The page itself does not ask for the API key. GET / returned 200 with text/html and no key even though --api-key was set, while /v1/models, /v1/chat/completions and /v1/messages returned 401 without it. Because the page is public, anyone who reaches the port can tell that a llama.cpp server runs there, even without chat access.

A plain curl of / without Accept-Encoding: gzip returned Error: gzip is not supported by this browser, so use a real browser or curl --compressed when you check it.

Tool calling with --jinja

Tool calling works through the normal OpenAI tools and tool_choice fields. llama.cpp parses the model's native format for families trained on tool use, with a generic fallback for the rest. It needs the model's Jinja chat template, which is why older guides tell you to add --jinja; in b11146 it is already the default.

With a get_weather tool defined, even the small Qwen model returned finish_reason tool_calls and the call {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}.

Keep --tools separate from this. That flag turns on the server's built-in agent tools, including exec_shell_command and write_file, and the README warns not to enable it in untrusted environments.

Run llama-server in Docker

The llama.cpp Docker docs name the official image ghcr.io/ggml-org/llama.cpp:server, which contains only the llama-server binary. GPU builds are tagged server-cuda (CUDA 12), server-cuda13, server-rocm, server-vulkan and server-intel.

docker run -p 127.0.0.1:8080:8080 -v ./models:/models \
  ghcr.io/ggml-org/llama.cpp:server \
  -m /models/model.gguf -c 4096 \
  --host 0.0.0.0 --port 8080 --api-key "$LLAMA_API_KEY"

The command binds --host 0.0.0.0 because, inside the container, the server has to listen beyond its own loopback for the published port to reach it. Publishing as 127.0.0.1:8080:8080 then keeps it on the host's loopback rather than the LAN. On NVIDIA hosts, add --gpus all and use the server-cuda tag. I did not run the container on the M1, so treat this block as untested; the image names and flags come from the docs.

For Compose, every flag has an LLAMA_ARG_* form, and the README example sets LLAMA_ARG_MODEL, LLAMA_ARG_CTX_SIZE and LLAMA_ARG_N_PARALLEL.

Connect Open WebUI and the OpenAI SDK

Any OpenAI client works once you change the base URL to http://127.0.0.1:8080/v1 and pass the key as the API key.

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="YOUR_KEY")
r = client.chat.completions.create(
    model="qwen2.5-0.5b",
    messages=[{"role": "user", "content": "Hello"}],
)
print(r.choices[0].message.content)

The openai Python package 3.22.1 completed a chat round trip against my server with only base_url and api_key changed.

In Open WebUI, go to Settings, Admin, Connections, OpenAI and add a connection. The Open WebUI llama.cpp guide says to use http://127.0.0.1:8080/v1, or http://host.docker.internal:8080/v1 when Open WebUI runs in Docker, and to keep the /v1 on the end. Enter your key under Bearer auth, and set Provider to llama.cpp to get the loaded-model indicator and the Eject button. For other front-ends, our Ollama GUI roundup covers clients that also accept an OpenAI-compatible URL.

Secure llama-server: API key and 0.0.0.0

The default bind is 127.0.0.1, so a fresh server is reachable only from the same machine. Both of my servers listened on 127.0.0.1 only according to lsof, and the second one, started without --api-key, answered /v1/models with 200 to anyone who could reach it.

Passing --host 0.0.0.0 changes that. Copy-paste examples, including the llama.app Compose file, bind 0.0.0.0 without a key, which leaves an open inference endpoint on your network. The README's own CORS guidance for a public server is to set an API key and put the server behind a reverse proxy.

A safe default, in order:

• Keep --host 127.0.0.1 unless another machine needs access.
• Set --api-key, --api-key-file or the LLAMA_API_KEY variable before binding any other address. Clients send Authorization: Bearer <key>, and a wrong key returns 401.
• Terminate TLS at a reverse proxy, or build with SSL and use --ssl-key-file and --ssl-cert-file.
• Avoid --tools and --agent on any shared host.
• Remember /health and the web UI page stay public by design.

How I tested this

• Machine: Apple M1 with 8 GB of RAM, macOS (Darwin 26.3, arm64).
• Versions: llama.cpp v0.5.0, build b11146 (commit 7fe450e19), from the official llama-b11146-bin-macos-arm64.tar.gz asset; Qwen2.5-0.5B-Instruct Q4_K_M GGUF (468.6 MB); openai Python package 3.22.1.
• Date: September 30, 2026.
• What I ran: a shell harness that started two servers. The first used the full command above on port 8089 with a key; the harness called each endpoint in the table with no key, a wrong key and the right key, sent chat, tool-call and native completion requests, timed five decode runs and two concurrent requests, and read /slots, /props and /metrics. The second ran llama serve on port 8090 with --no-slots, no key and no -c or -np, to record the defaults.
• What I did not test: the Docker image, Open WebUI, router mode, Windows and Linux builds, larger models, and binding to 0.0.0.0. The web UI screenshot comes from a separate session on build b11344 and is not part of the record.
• Raw record: results.json, published with the harness script.

Related coverage

• Ollama vs llama.cpp: if you are still choosing between llama-server and Ollama's server, start with this comparison.
• Q4_K_M vs Q5_K_M vs Q8_0: which GGUF quant to pass to -hf.
• KV cache size: what -c, -np and -ctk q8_0 cost in memory.
• Local LLM hardware calculator: check whether a model fits before you set -ngl.

References

• llama.app API docs - https://llama.app/docs/api
• llama.app serve docs - https://llama.app/docs/serve
• llama.cpp Docker docs - https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md
• llama.cpp server README - https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• llama.cpp v0.5.0 release - https://github.com/ggml-org/llama.cpp/releases/tag/v0.5.0
• Open WebUI llama.cpp guide - https://docs.openwebui.com/getting-started/quick-start/connect-a-provider/starting-with-llama-cpp/