Ollama Docker GPU Setup: Fix CPU Fallback, WebUI Connections

ollama docker works when the container gets the right device access, the Ollama data directory is mounted, and every client uses the address that matches its network namespace. A GPU flag alone is not enough. Choose the image for the hardware, verify the GPU from inside the container, keep /root/.ollama on a named volume, and point Open WebUI at ollama:11434, host.docker.internal:11434, or 127.0.0.1:11434 according to the topology. (Source: Ollama Docker docs)

Start with these checks before changing models or rebuilding everything:

• curl http://127.0.0.1:11434/api/version proves that the published Ollama API answers from the host.
• docker logs --tail=200 ollama shows whether the container started with the expected runtime.
• docker exec -it ollama ollama ps shows whether the loaded model is on the GPU, CPU, or split across both.
• docker inspect ollama confirms that the model volume and device request are attached to the container.

The checks below cover NVIDIA, AMD, WSL2, CPU, Compose, persistence, and Open WebUI networking. No local Docker test was run. (Source: Ollama FAQ)

Choose an Ollama Docker deployment before changing flags

Identify where Ollama and its client run first. The table below is a decision map, not a promise that every GPU is supported by every backend.

The important distinction is the endpoint column. 127.0.0.1 means the current network namespace. It means the host only when the client uses host networking. A Compose service should use its service name, while a bridge container reaching an Ollama process on the host usually needs host.docker.internal. If you are comparing a native install with an ollama container, keep runner differences separate from Docker wiring; see our Ollama versus llama.cpp comparison. (Source: Open WebUI connection troubleshooting)

Start Ollama with the right image and runtime

The ollama docker image for a CPU-only container is deliberately plain. Run the official command on any host where Docker is already installed, including Windows through Docker Desktop. The named volume keeps downloaded models outside the writable container layer. (Source: Ollama Docker docs)

# CPU scope: Linux, macOS, or Windows Docker hosts
docker run -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

For ollama docker gpu ubuntu, install the NVIDIA Container Toolkit, configure Docker, restart the daemon, and add the GPU request. The nvidia-ctk and systemctl lines apply to a Linux Docker daemon. For ollama docker gpu windows, WSL2 users with Docker Desktop should configure the Docker Desktop and WSL2 GPU integration instead of assuming that the Linux daemon commands apply unchanged. (Source: Ollama Docker docs)

# NVIDIA scope: Ubuntu Linux with a configured NVIDIA Docker runtime
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

For AMD on Linux, use the ROCm image and pass the device nodes Ollama documents. Do not copy this /dev mapping into a native Windows setup and assume it is equivalent. The reviewed Docker path is a Linux device-mapping path; AMD support and the container runtime still depend on the host stack and GPU model. (Source: Ollama Docker docs)

# AMD scope: Linux with a working ROCm device stack
docker run -d \
  --device /dev/kfd \
  --device /dev/dri \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama:rocm

For an NVIDIA ollama docker compose deployment, reserve a GPU in Compose and put Open WebUI on the same default network. Remove the deploy.resources...devices block for a CPU-only run. This example is a configuration pattern from the Docker and Ollama documentation, not a result from a local experiment. (Source: Docker Compose GPU support)

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "11434:11434"
    environment:
      OLLAMA_HOST: 0.0.0.0:11434
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      OLLAMA_BASE_URL: http://ollama:11434
    volumes:
      - open-webui:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama:
  open-webui:

For ollama docker environment variables, separate listener, model-path, and GPU-selection settings. OLLAMA_HOST controls where the server listens. OLLAMA_MODELS changes the model directory and must agree with a mounted path. CUDA_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES select GPU devices, while a value of -1 intentionally forces CPU execution. (Source: Ollama FAQ) (Source: Ollama GPU docs)

Make model storage survive container replacement

An Ollama container and its model store are separate objects. Mounting ollama:/root/.ollama puts the model store on a named Docker volume, so recreating the container can reuse it. Open WebUI has its own application data at /app/backend/data; keep that on open-webui:/app/backend/data if you want users, settings, and chats to persist separately. (Source: Ollama Docker docs)

Use these checks before deleting a container:

# Inspect the named volume and the mounts on the running container
docker volume inspect ollama
docker inspect ollama --format '{{json .Mounts}}'

# Confirm the model list from inside Ollama
docker exec -it ollama ollama list

If a recreated container has no models, compare the old and new mount definitions. A changed Compose project, a different bind path, a missing -v flag, or a cleanup action that removed the volume can all produce the same symptom. docker inspect tells you what the current container actually received; it does not prove that an older container used the same volume. For model fit, use our LLM VRAM calculator before assuming storage caused a load failure. Inference: compare the mount before changing images or environment variables.

Do not treat OLLAMA_MODELS as a backup setting. It changes where Ollama looks. A backup needs a copy or snapshot of the named volume, and a restore needs the same path mounted into the replacement container. Model storage and WebUI state are different volumes, so back them up independently. (Source: Ollama FAQ)

Verify the GPU before diagnosing model behavior

An ollama docker gpu diagnosis should move from the outside in. Stop at the first failed layer.

1. Prove the API is alive. On the Docker host, run the published-port checks below. /api/version proves API reachability, /api/tags proves that the model-list endpoint answers, and /api/ps reports models currently loaded. (Source: Ollama API docs)curl -fsS http://127.0.0.1:11434/api/version curl -fsS http://127.0.0.1:11434/api/tags curl -fsS http://127.0.0.1:11434/api/ps
2. Read the container boundary. Check the process and recent logs. A running container only proves that the process started, not that Docker passed a GPU through. (Source: Ollama Docker docs)docker ps --filter name=ollama docker logs --tail=200 ollama docker inspect ollama --format '{{json .HostConfig.DeviceRequests}}'
3. Check the host GPU. Use nvidia-smi for NVIDIA on Ubuntu or WSL2. Use rocminfo for AMD Linux. If the host cannot see the device, Ollama cannot repair the driver or runtime from inside its container. (Source: Ollama GPU docs)
4. Check the container GPU. For NVIDIA, try docker exec -it ollama nvidia-smi if the image contains the utility. For AMD, inspect /dev/kfd and /dev/dri with docker exec. A missing diagnostic utility is not conclusive by itself; the device request, logs, and Ollama processor report still matter. (Source: Ollama GPU docs)
5. Check placement after a model is loaded. Run docker exec -it ollama ollama ps. Ollama documents processor states such as 100% GPU, 100% CPU, and split CPU/GPU placement. A split result is not the same as a missing GPU; it means the model is not entirely on the GPU. (Source: Ollama FAQ)

If Ollama is not using the GPU, check the first failed layer instead of adding random flags. Common causes include a missing NVIDIA Container Toolkit, an absent --gpus=all, a missing AMD device mapping, CUDA_VISIBLE_DEVICES=-1, ROCR_VISIBLE_DEVICES=-1, a Linux GPU resume issue, or a device permission policy such as SELinux. Ollama also documents CUDA_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, and Vulkan device selection, so inspect the effective environment before changing it. (Source: Ollama GPU docs)

Connect Open WebUI to Ollama without guessing localhost

An open webui docker ollama setup can put both services in Compose, keep Ollama on the host, or use another machine. The URL is chosen by the Open WebUI backend, not the browser. http://localhost:3000 names the browser's machine; http://localhost:11434 inside a container names that container. If you are also maintaining the UI container, the Open WebUI Docker update checklist is a useful next read for backups, version pinning, and rollback. (Source: Open WebUI connection troubleshooting)

Use the address that matches the topology:

• Same Compose project: set OLLAMA_BASE_URL=http://ollama:11434. ollama is the Compose service name.
• Ollama on the host, WebUI in a bridge container: set OLLAMA_BASE_URL=http://host.docker.internal:11434. On Linux, add --add-host=host.docker.internal:host-gateway when needed.
• Ollama on another machine: use its reachable IP or DNS name and port, not localhost. The host process must listen beyond its loopback address.
• Intentional host networking: use --network=host and OLLAMA_BASE_URL=http://127.0.0.1:11434 only when host mode is chosen. Do not combine it with a normal -p 3000:8080 assumption.

# Docker bridge scope: WebUI container, Ollama running on the host
docker run -d \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

If the symptom is open webui not connecting to ollama, test the endpoint before changing the model. When native Ollama listens only on 127.0.0.1:11434, a bridge container cannot use that loopback address to reach the host. Configure the host process with OLLAMA_HOST=0.0.0.0:11434, restart it, and restrict the exposed port with the host firewall. If the model selector spins or logs show connection refused, test /api/version from the same network context and inspect whether a saved WebUI connection overrides the environment variable. (Source: Open WebUI connection troubleshooting)

Open WebUI documents persistent configuration behavior: a URL saved in its database can take precedence over a later environment change. When an old endpoint remains, update the admin connection or use the documented persistent-config reset options carefully. Resetting configuration is different from deleting the Ollama model volume. (Source: Open WebUI connection troubleshooting)

FAQ

Can I run Ollama in Docker?

Yes. Run the official ollama/ollama image, publish port 11434, and mount ollama:/root/.ollama. Use the CPU command when no accelerator is needed, then add NVIDIA or AMD device configuration only for the matching host runtime. Docker packages the server, but it does not select a GPU automatically. (Source: Ollama Docker docs)

Can Ollama Docker use GPU?

Yes, with a configured accelerator runtime. NVIDIA uses the NVIDIA Container Toolkit and --gpus=all; AMD Linux uses the rocm image with /dev/kfd and /dev/dri. Verify device visibility, logs, and ollama ps. A container that starts successfully can still run a model on the CPU. (Source: Ollama Docker docs)

Why is Ollama not using my GPU in Docker?

Check the host driver first, then the container device request, then Ollama's processor report. On NVIDIA, inspect the toolkit and --gpus=all. On AMD, inspect the ROCm image and device nodes. Also check CUDA_VISIBLE_DEVICES or ROCR_VISIBLE_DEVICES; an invalid or -1 selection can force CPU execution. (Source: Ollama GPU docs)

How do I connect Open WebUI to Ollama in Docker?

Use http://ollama:11434 when both services share a Compose network. Use http://host.docker.internal:11434 when Open WebUI is in Docker and Ollama runs on the host. Do not use 127.0.0.1 from a bridge container unless host networking is intentional. Set the matching OLLAMA_BASE_URL and restart WebUI. (Source: Open WebUI connection troubleshooting)

Why can't Open WebUI connect to Ollama?

The usual causes are a wrong network address, Ollama listening only on its host loopback, a blocked port, or an old connection saved in Open WebUI. Test /api/version from the backend's network context, read the WebUI logs, and confirm the URL uses a Compose service name, host gateway, host IP, or host networking as appropriate. (Source: Open WebUI connection troubleshooting)

Related coverage

• Run a local LLM on 8GB RAM
• Cloud GPU pricing comparison

References

• Docker Compose GPU support - https://docs.docker.com/compose/how-tos/gpu-support/
• Ollama API docs - https://docs.ollama.com/api/introduction
• Ollama Docker docs - https://docs.ollama.com/docker
• Ollama FAQ - https://docs.ollama.com/faq
• Ollama GPU docs - https://docs.ollama.com/gpu
• Open WebUI connection troubleshooting - https://docs.openwebui.com/troubleshooting/connection-error/