Run ollama with docker-compose and using gpu

Question

I want run ollama with docker-compose and using nvidia-gpu. What should I write in the docker-compose.yml file?

I run ollama with docker-compose, but gpu was not been used, this is what i write:

  ollama:
    container_name: ollama
    image: ollama/ollama:rocm
    ports:
      - 11434:11434
    volumes:
      - ollama:/root/.ollama
    networks:
      - fastgpt
    restart: always

I need a docker-compose.yaml file example.

Answer 1

I'm assuming that you have the GPU configured and that you can successfully execute nvidia-smi.

If do then you can adapt your docker-compose.yml as follows:

version: "3.9"

services:  
  ollama:
    container_name: ollama
    image: ollama/ollama:rocm
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            capabilities: ["gpu"]
            count: all
    volumes:
      - ollama:/root/.ollama
    restart: always

volumes:
    ollama:

When you bring it up you should see "Nvidia GPU detected via cudart" in the logs.