SHSelfHostIndex

Ollama

Run large language models locally with a simple CLI and API

Local LLM Runners ★ 181.5k stars Easy setup MIT

Ollama lets you download, run, and manage open large language models such as Llama, Mistral, Gemma, and Qwen on your own machine. It provides a simple command line interface and a built-in REST API so other tools can use local models.

Official siteSource code

Ollama setup guides & articles

Hands-on coverage of Ollama from the blog.

Self-Hosting Guides

A Complete Self-Hosted AI Stack: Ollama, Open WebUI, Whisper

Ollama, Open WebUI, and faster-whisper in one compose file — GPU passthrough, model sizing per VRAM tier, and speech-to-text that never leaves your LAN.

Jul 11, 2026 · 5 min read AI Optimisation

Quantisation Explained: GGUF, AWQ, and What You Actually Lose

GGUF, AWQ, and FP8 quantisation explained — size and quality by bit depth, what degrades first, and how to A/B a quant honestly before trusting it.

Jul 11, 2026 · 4 min read AI Development

Open-Weight Models in 2026: What's Actually Usable

Which open-weight models are production-usable in 2026, task by task, plus licence gotchas and the maths for when they beat APIs on cost and privacy.

Jul 11, 2026 · 5 min read

Key features

  • One-command model downloads
  • OpenAI-compatible API
  • GPU and CPU support
  • Modelfile customization

Quick deploy

A starting point for self-hosting Ollama - check the official docs for the full set of options.

  • Image ollama/ollama:latest
  • Web port 11434
  • Persist /root/.ollama
Docker Compose
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ./.ollama:/root/.ollama
    restart: unless-stopped
docker run
docker run -d --name ollama \
  -p 11434:11434 \
  -v ./.ollama:/root/.ollama \
  --restart unless-stopped \
  ollama/ollama:latest

Watch out for

  • GPU access needs the NVIDIA Container Toolkit and --gpus=all (AMD uses the :rocm image)
  • Models are multi-gigabyte - put the volume on a disk with room to grow
  • The API has no authentication - never publish port 11434 to the internet

Pros & cons

Strengths

  • Extremely easy to set up
  • Large model library

Trade-offs

  • Limited fine-grained inference tuning

Ollama replaces

Self-hosted ChatGPT alternativesSelf-hosted OpenAI API alternatives

Last reviewed Aug 22, 2026 · 767 words

Ollama makes running a local LLM a one-liner — ollama run llama3.2 downloads the model and drops you into a chat. That earned it the biggest community in local AI, and it deserves the reputation. The two things the quickstart doesn't teach: how to match models to your actual hardware, and the fact that the API ships with no authentication whatsoever, which turns one careless port mapping into a free compute service for strangers.

VRAM decides everything — size models to your card

A model needs to fit in memory, and quantized models (the 4-bit q4 variants Ollama defaults to) are what make consumer hardware viable. Rough working table:

HardwareComfortable modelsExperience
8 GB VRAM (RTX 3060 Ti class)7–8B q4Fast, genuinely useful
12–16 GB VRAM13–14B q4The sweet spot per dollar
24 GB VRAM (3090/4090)27–32B q4Approaches small cloud models
CPU only, 16 GB RAM3–8B, patientlyFine for chat, slow for work

A model that exceeds VRAM spills into system RAM and drops from tens of tokens per second to single digits — if a model feels broken-slow, check ollama ps to see the GPU/CPU split. The VRAM math post covers the arithmetic, and hardware for self-hosted LLMs covers what to buy. In Docker, GPU access needs the NVIDIA Container Toolkit and --gpus=all (AMD cards use the :rocm image tag); models are multi-gigabyte, so put /root/.ollama on a volume with real headroom.

The API has no auth — act accordingly

Port 11434 answers to anyone who can reach it: pull models, run generations, delete models, no credentials asked. Shodan is full of open Ollama instances donating GPU time to the internet. The rules are simple. Bind to localhost (the default) or a LAN/VPN interface, never 0.0.0.0 on a public box. If a remote tool needs it, tunnel over Tailscale or put an authenticating reverse proxy in front. And in compose, prefer 127.0.0.1:11434:11434 mappings so a firewall mistake can't expose it.

The API itself is Ollama's superpower: it speaks an OpenAI-compatible dialect, so nearly every AI tool — editors, agents, chat UIs — can point at http://localhost:11434/v1 instead of OpenAI by changing a base URL. Pair it with Open WebUI and you have a private ChatGPT with multi-user logins, RAG over your documents, and model switching; that combination is the backbone of the self-hosted AI stack.

Small habits that improve daily use

  • Keep-alive: models unload after 5 minutes by default, so the next request pays a multi-second reload. OLLAMA_KEEP_ALIVE=1h (or -1 for always) fixes the "why is the first reply slow" complaint at the cost of held VRAM.
  • Context length: the default context is short; for long documents or agent work, raise num_ctx — and know that KV cache eats VRAM fast as context grows, which is usually the real reason a previously fine model starts spilling to CPU.
  • Modelfiles: a five-line Modelfile bakes a system prompt and parameters into a named model (ollama create support-bot -f Modelfile), which beats re-sending instructions from every client.
  • Disk hygiene: ollama list then ollama rm the experiments. Model hoarding at 4–40 GB each is how a 1 TB drive fills in a season.

When Ollama is the wrong tool

Ollama optimises for convenience, not throughput. For one user on one box it is the right default; for serving many concurrent users, vLLM roughly triples throughput on the same hardware via continuous batching. If you want maximum control over quantization and every last token per second from bare metal, llama.cpp — the engine Ollama builds on — exposes everything Ollama deliberately hides; that trade is mapped in Ollama vs llama.cpp. And if your use case is an occasional hard question rather than volume, a frontier cloud model will simply be smarter than anything that fits in 24 GB.

What I'd do

Install natively (not Docker) on whatever machine has the best GPU, run an 8B and a 14B q4 and keep whichever your hardware serves fast, set keep-alive, bind to localhost with Tailscale for remote use, and put Open WebUI in front for the household. Treat local models as what they are in 2026 — private, free, and good enough for 80% of tasks — and keep a cloud key around for the other 20%.

Compare Ollama

22 head-to-head comparisons.

Similar local llm runners apps