Ollama

Prerequisites

This guide assumes you have the following prerequisites in place:

Ollama

Ollama is a platform for running and managing large language models (LLMs) on your own infrastructure.

Installation

  1. Create the following directory structure for Ollama:
  ollama/
  ├── pvc.yml
  ├── helmrepository.yml
  └── helmrelease.yml
  1. Add the following content to pvc.yml:
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-longhorn
  namespace: tools
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi
  storageClassName: longhorn
  1. Add the following content to helmrepository.yml:
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: ollama
  namespace: tools
spec:
  interval: 6h
  chart:
    spec:
      chart: ollama
      version: "1.45.0"
      sourceRef:
        kind: HelmRepository
        name: ollama
        namespace: flux-system
      interval: 6h
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
  values:
    ollama:
      gpu:
        enabled: true
        type: nvidia
    service:
      type: ClusterIP
    runtimeClassName: nvidia
    persistentVolume:
      enabled: true
      existingClaim: ollama-longhorn
  1. Add the following content to helmrepository.yml:
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: ollama
  namespace: flux-system
spec:
  interval: 6h
  url: https://otwld.github.io/ollama-helm/
  1. Commit and push the files to your Git repository. Flux will automatically deploy the resources to your cluster.