Ollama Docker GPU: Fix CPU Fallback and Open WebUI

If Ollama in Docker runs on the CPU, loses its models, or will not connect to Open WebUI, the cause usually sits at one of three boundaries: the container never received the GPU, the model volume is not persistent, or the client is using the wrong network address. Verify the runtime and API first, then run ollama ps to see where the model loaded. For the network, use ollama:11434 inside Compose and host.docker.internal:11434 when Open WebUI reaches an Ollama process on the host. The commands below follow the Ollama Docker docs and the Ollama FAQ.

A Mac is the one case where CPU is the expected result rather than a fault. I tested it on October 5, 2026: on an 8 GB M1, the official ollama/ollama:0.35.1 image found no GPU, ollama ps put the whole model on the CPU, and warm generation ran at 9.1 tok/s against 46.56 tok/s for native Ollama on the same model file. On a Mac, run Ollama natively and keep only Open WebUI in Docker. On Linux with an NVIDIA or AMD GPU, the steps below get the container onto the GPU.

Start with these checks before changing models or rebuilding everything:

• curl http://127.0.0.1:11434/api/version proves that the published Ollama API answers from the host.
• docker logs --tail=200 ollama shows whether the container started with the expected runtime.
• docker exec -it ollama ollama ps shows whether the loaded model is on the GPU, CPU, or split across both.
• docker inspect ollama confirms that the model volume and device request are attached to the container.

Choose an Ollama Docker deployment before changing flags

Identify where Ollama and its client run first. The table below is a decision map, not a promise that every GPU is supported by every backend.

The column that matters most is the endpoint. 127.0.0.1 means the current network namespace; it means the host only when the client uses host networking. A Compose service should use its service name, while a bridge container reaching an Ollama process on the host usually needs host.docker.internal.

Ollama in Docker on a Mac: CPU only, about 5x slower

Docker Desktop on macOS runs Linux containers inside a virtual machine and does not pass the Mac GPU through the way native Ollama uses it. To put numbers on that, I ran the same model file through native Ollama and through the official image on an 8 GB M1 MacBook Air, one runtime at a time, with the same prompt and options. The container started with no GPU flags and read the host's model folder through a read-only mount, so there was no second download:

docker run -d --name aw-ollama -p 11435:11434 -e OLLAMA_NOPRUNE=1 \
  -v ~/aw-lab/ollama-models:/root/.ollama/models:ro ollama/ollama:0.35.1

Each runtime got one cold request after unloading the model, then three warm requests with a 52-token prompt and 128 output tokens. Speeds are Ollama's own response fields.

Native generation was 5.12 times faster than the container on warm requests. The Docker prompt-processing runs swung from 72.7 to 844.4 tok/s, so treat that median as noisy. The cold Docker numbers are the worst of the set: the Mac was already deep in swap, and I would read 96.7 s and 0.65 tok/s as what a memory-starved 8 GB machine does, not a typical first load.

The container log, trimmed to the GPU lines, shows why. Ollama looked for GPUs at startup, found only the CPU, and sized its default context for zero VRAM, even though OLLAMA_VULKAN was true in its server config:

msg="discovering available GPUs..."
msg="inference compute" id=cpu library=cpu compute="" name=cpu description=cpu libdirs=ollama driver="" pci_id="" type="" total="3.8 GiB" available="3.8 GiB"
msg="vram-based default context" total_vram="0 B" default_num_ctx=4096
msg="disabling mmap for llama-server load by default" reason=cpu

docker inspect recorded HostConfig.DeviceRequests as null, and the CPU-only load put 934.14 MiB into a repacked CPU weight buffer plus 112 MiB of KV cache inside a VM that had 3.8 GiB to give. docker stats showed the container at 1.322GiB of its 3.828GiB limit with the model loaded. The arm64 image alone took 7.05GB on disk.

The ollama ps output from each side makes the difference easy to spot:

# native
NAME            ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen2.5:1.5b    65ec06548149    1.2 GB    100% GPU     4096       4 minutes from now

# docker exec aw-ollama ollama ps
NAME            ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen2.5:1.5b    65ec06548149    1.4 GB    100% CPU     4096       4 minutes from now

So on a Mac, run Ollama natively for the GPU and put only Open WebUI in a container, pointed at http://host.docker.internal:11434; the command is in the Open WebUI section below. I would still use Ollama in Docker on a Mac to check an API integration or a Compose file before moving it to a Linux GPU host, as long as CPU speed is acceptable for that job.

Start Ollama with the right image and runtime

For a CPU-only container, use the plain official ollama/ollama image on a host where Docker is already installed. The named volume keeps downloaded models outside the writable container layer. This command is written for Bash, so it runs on Linux, macOS, WSL, or Git Bash on Windows:

docker run -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

On Ubuntu Linux with NVIDIA, install the NVIDIA Container Toolkit, configure Docker, restart the daemon, and add the GPU request. The nvidia-ctk and systemctl lines apply to a Linux Docker daemon with a configured NVIDIA Docker runtime. On Windows, WSL2 users with Docker Desktop should configure Docker Desktop and WSL2 GPU integration instead of assuming that the Linux daemon commands apply unchanged.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

For AMD on Linux with a working ROCm device stack, use the ROCm image and pass the device nodes Ollama documents. Do not copy this /dev mapping into a native Windows setup and assume it is equivalent. The documented Docker path is a Linux device-mapping path; AMD support and the container runtime still depend on the host stack and GPU model.

docker run -d \
  --device /dev/kfd \
  --device /dev/dri \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama:rocm

In an NVIDIA Compose deployment, reserve a GPU in Compose and put Open WebUI on the same default network. Remove the deploy.resources...devices block for a CPU-only run. The file follows the pattern in Docker's Compose GPU documentation and the Ollama docs. Compose cannot repair a missing host runtime: on NVIDIA, the host still needs a working driver and the NVIDIA Container Toolkit.

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "11434:11434"
    environment:
      OLLAMA_HOST: 0.0.0.0:11434
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      OLLAMA_BASE_URL: http://ollama:11434
    volumes:
      - open-webui:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama:
  open-webui:

Keep the environment variables in three groups: listener, model path, and GPU selection. OLLAMA_HOST controls where the server listens. OLLAMA_MODELS changes the model directory and must agree with a mounted path. CUDA_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES select GPU devices, as the Ollama GPU docs describe, while a value of -1 intentionally forces CPU execution.

Make model storage survive container replacement

An Ollama container and its model store are separate objects. Mounting ollama:/root/.ollama puts the model store on a named Docker volume, so recreating the container can reuse it. Open WebUI has its own application data at /app/backend/data; keep that on open-webui:/app/backend/data if you want users, settings, and chats to persist separately.

Before deleting a container, inspect the named volume and the mounts on the running container, then confirm the model list from inside Ollama:

docker volume inspect ollama
docker inspect ollama --format '{{json .Mounts}}'

docker exec -it ollama ollama list

If a recreated container has no models, compare the old and new mount definitions. A changed Compose project, a different bind path, a missing -v flag, or a cleanup action that removed the volume can all produce the same symptom. docker inspect tells you what the current container actually received; it does not prove that an older container used the same volume. I would compare the mount before changing images or environment variables. For model fit, use the LLM VRAM calculator before assuming storage caused a load failure.

Do not treat OLLAMA_MODELS as a backup setting. It changes where Ollama looks. A backup needs a copy or snapshot of the named volume, and a restore needs the same path mounted into the replacement container. Model storage and WebUI state are different volumes, so back them up independently.

Outside Docker, the same variables apply to a native install; the Ollama commands reference lists every CLI command, the environment variables, and where each OS stores models.

Verify the GPU before diagnosing model behavior

Diagnose GPU use from the outside in, and stop at the first failed layer.

1. Prove the API is alive. On the Docker host, run the published-port checks curl -fsS http://127.0.0.1:11434/api/version, curl -fsS http://127.0.0.1:11434/api/tags, and curl -fsS http://127.0.0.1:11434/api/ps. In the Ollama API docs, /api/version proves API reachability, /api/tags proves that the model-list endpoint answers, and /api/ps reports models currently loaded.
2. Read the container boundary. Check the process, recent logs, and device request with docker ps --filter name=ollama, docker logs --tail=200 ollama, and docker inspect ollama --format '{{json .HostConfig.DeviceRequests}}'. A running container only proves that the process started, not that Docker passed a GPU through.
3. Check the host GPU. Use nvidia-smi for NVIDIA on Ubuntu or WSL2. Use rocminfo for AMD Linux. If the host cannot see the device, Ollama cannot repair the driver or runtime from inside its container.
4. Check the container GPU. For NVIDIA, try docker exec -it ollama nvidia-smi if the image contains the utility. For AMD, inspect /dev/kfd and /dev/dri with docker exec. A missing diagnostic utility is not conclusive by itself; the device request, logs, and Ollama processor report still matter.
5. Check placement after a model is loaded. Run docker exec -it ollama ollama ps. Ollama documents processor states such as 100% GPU, 100% CPU, and split CPU/GPU placement. A split result is not the same as a missing GPU; it means the model is not entirely on the GPU.

If Ollama is not using the GPU, fix the first failed layer instead of adding random flags. Common causes include a missing NVIDIA Container Toolkit, an absent --gpus=all, a missing AMD device mapping, CUDA_VISIBLE_DEVICES=-1, ROCR_VISIBLE_DEVICES=-1, a Linux GPU resume issue, or a device permission policy such as SELinux. Ollama also documents CUDA_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, and Vulkan device selection, so inspect the effective environment before changing it.

Connect Open WebUI to Ollama without guessing localhost

A Docker deployment with Open WebUI and Ollama can put both services in Compose, keep Ollama on the host, or use another machine. As the Open WebUI connection troubleshooting guide explains, the URL is chosen by the Open WebUI backend, not the browser. http://localhost:3000 names the browser's machine; http://localhost:11434 inside a container names that container.

Use the address that matches the topology:

• Same Compose project: set OLLAMA_BASE_URL=http://ollama:11434. ollama is the Compose service name.
• Ollama on the host, WebUI in a bridge container: set OLLAMA_BASE_URL=http://host.docker.internal:11434. On Linux, add --add-host=host.docker.internal:host-gateway when needed.
• Ollama on another machine: use its reachable IP or DNS name and port, not localhost. The host process must listen beyond its loopback address.
• Intentional host networking: use --network=host and OLLAMA_BASE_URL=http://127.0.0.1:11434 only when host mode is chosen. Do not combine it with a normal -p 3000:8080 assumption.

For the second layout, a WebUI container on the Docker bridge with Ollama running on the host:

docker run -d \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

When Open WebUI cannot connect to Ollama, test the endpoint before changing the model. When native Ollama listens only on 127.0.0.1:11434, a bridge container cannot use that loopback address to reach the host. Configure the host process with OLLAMA_HOST=0.0.0.0:11434, restart it, and restrict the exposed port with the host firewall. If the model selector spins or logs show connection refused, test /api/version from the same network context and inspect whether a saved WebUI connection overrides the environment variable.

That override is documented behavior: a URL saved in Open WebUI's database can take precedence over a later environment change. When an old endpoint remains, update the admin connection or use the documented persistent-config reset options carefully. Resetting configuration is different from deleting the Ollama model volume.

After the Ollama container is healthy, follow the Open WebUI Docker update and rollback checklist. It covers backups, pinning, and recovery when the UI container changes.

Open WebUI is one of several front-ends for Ollama; the best Ollama GUI comparison covers lighter options, such as Ollama's own desktop app and browser-extension clients, if you do not need a Docker web UI.

How I tested this

• Machine: a MacBook Air with an Apple M1 and 8 GB of unified memory, with desktop apps open.
• Date: October 5, 2026.
• Versions: native Ollama 0.35.1, the macOS binary using Metal; the official ollama/ollama:0.35.1 arm64 image in Docker Desktop, whose Linux VM gave the container 8 vCPUs. Both reported 0.35.1 from /api/version.
• What was run: native Ollama first; then native Ollama was stopped and the container started with the docker run command shown in the Mac section, with no GPU flags. Both runtimes loaded the same GGUF file from the host's model folder. For each, the model was unloaded, then one cold and three warm /api/generate requests used the same fixed prompt at temperature 0 with a fixed seed. After the runs I captured ollama ps, docker stats, the container's device requests, and the Ollama startup log.
• Caveat: the Mac was under memory pressure, with more than 10 GB of swap in use before each phase. That hits the cold Docker figures hardest, which is why I treat the 96.7 s load and 0.65 tok/s as a worst case.
• Not tested: NVIDIA (Container Toolkit, --gpus=all, Compose GPU reservations), AMD with ROCm (/dev/kfd, /dev/dri), Linux hosts, and Windows with WSL2, because this Mac has none of them; nothing here measures those paths. I also did not test pulling a model into a named volume inside the container, the Compose file with Open WebUI, or Open WebUI reaching native Ollama through host.docker.internal.
• Raw record: every metric is in the public results file, next to the capture scripts and the container's GPU log lines.

Related coverage

• Open WebUI Docker Update: Backup, Pinning, and Rollback
• Best Ollama GUI: Which Front-End to Use on Each Platform
• LLM VRAM Calculator: RAM, Quantization, and Model Fit
• Cloud GPU Pricing Comparison: Cost per Million Tokens

References

• Docker Compose GPU support - https://docs.docker.com/compose/how-tos/gpu-support/
• Ollama API docs - https://docs.ollama.com/api/introduction
• Ollama Docker docs - https://docs.ollama.com/docker
• Ollama FAQ - https://docs.ollama.com/faq
• Ollama GPU docs - https://docs.ollama.com/gpu
• Open WebUI connection troubleshooting - https://docs.openwebui.com/troubleshooting/connection-error/