A Complete Self-Hosted AI Stack: Ollama, Open WebUI, Whisper

Three containers give you a private ChatGPT-shaped stack: Ollama serves the models, Open WebUI provides the chat interface with users and RAG, and a faster-whisper container handles speech-to-text. On a used RTX 3060 12GB (about $200 on eBay), that means 8B-class models at 35–50 tokens/s, an hour of audio transcribed in a few minutes, and precisely zero bytes of any of it leaving your network. The compose file below is the whole install.

The stack in one compose file

Prerequisite on the host: NVIDIA drivers plus the NVIDIA Container Toolkit (nvidia-smi working inside a test container). Then:

services:
  ollama:
    image: ollama/ollama:latest
    volumes:
      - ./ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=1h
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - ./webui:/app/backend/data
    restart: unless-stopped

  whisper:
    image: onerahmet/openai-whisper-asr-webservice:latest-gpu
    ports:
      - "9000:9000"
    environment:
      - ASR_MODEL=large-v3-turbo
      - ASR_ENGINE=faster_whisper
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

Note Ollama exposes no port to the LAN — Open WebUI reaches it on the compose network, which is the correct default since the Ollama API is unauthenticated. If you must expose it, put it behind your reverse proxy with auth in front.

What fits on your GPU

The model menu is a function of VRAM, and the arithmetic is worth knowing rather than memorising — weights plus KV cache plus overhead, walked through with worked examples in this site's VRAM math guide. The practical tiers:

CPU-only deserves an honest sentence: it works, at 3–10 tokens/s for small models on a desktop CPU — fine for background summarisation jobs, patience-testing for interactive chat. The hardware guide for self-hosted LLMs covers the used-market GPU options tier by tier.

Model management is Ollama's strong suit: docker exec -it ollama ollama pull whatever you're testing, and models load on first request and unload after the OLLAMA_KEEP_ALIVE window. The two settings that matter once family members share the box: OLLAMA_MAX_LOADED_MODELS (VRAM is a timeshare; two loaded models halve your headroom) and keep-alive itself — 1h avoids the 5–20 second load penalty on every cold prompt at the cost of holding VRAM between chats.

Open WebUI carries the household use case

Open WebUI is more than a chat skin, and two features do most of the work of making one GPU serve several people. Knowledge collections let you upload documents and chat against them with citations — the embeddings run locally through Ollama, so the RAG pipeline inherits the same privacy as the chat. And it's properly multi-user: per-user chat history, admin-controlled model visibility, sign-ups you can disable after the family is enrolled. Everyone gets their own account at http://server:3000, nobody reads anyone else's transcripts, and you decide which models a new user even sees. It's the difference between a demo and a service you can hand to non-technical relatives as-is.

Wiring in ears: Whisper

The whisper-asr-webservice container wraps faster-whisper behind a simple HTTP API. large-v3-turbo is the model to run in 2026: near-identical accuracy to large-v3 at several times the speed, roughly 6GB of VRAM in fp16 — or drop to int8 (ASR_QUANTIZATION=int8) at ~3GB to coexist politely with a loaded LLM on a 12GB card. Real-world throughput on the 3060 is many times realtime; a one-hour meeting recording comes back as text in the low single-digit minutes.

Open WebUI consumes it directly: Admin Settings → Audio, STT engine to OpenAI-compatible, endpoint http://whisper:9000, and the microphone button in every chat now transcribes locally. The same endpoint serves anything else on your LAN that speaks the OpenAI transcription API shape — meeting-notes scripts, voice memos pipelines, Home Assistant voice experiments.

A voice, if you want one

Text-to-speech is the optional fourth piece. Piper runs comfortably on CPU with tolerable quality and near-instant latency; the newer small neural TTS models trade a little latency for much better voices. Open WebUI's Audio settings accept any OpenAI-compatible TTS endpoint the same way as STT, and Piper voices are a one-time ~50–100MB download that then runs offline like everything else here. I'd still call TTS the least load-bearing part of the stack — add it after everything else works.

What "private" means here, concretely

With this stack, prompts, documents fed to RAG, audio recordings, and transcripts all terminate on your hardware. There's no per-token bill, no data-retention policy to parse. Verify rather than trust: run it for a day and watch egress from the containers — the expected outbound traffic is image pulls and model downloads, nothing at inference time. The local AI privacy audit covers what to check. The honest trade: an 8–14B local model is not a frontier model, and it will lose on hard reasoning tasks. For summarisation, drafting, transcription, and anything involving documents you'd rather not upload, the gap stopped mattering a while ago.

What I'd do

Used RTX 3060 12GB into whatever box has a spare PCIe slot, the compose file above, one 8B general model and one 14B for heavier lifting, whisper on int8, keep-alive at an hour. Total spend if you have any workable host already: about $200, or roughly ten months of a hosted chat subscription — except this one transcribes unlimited audio, serves the whole household, and answers to nobody's usage policy but yours.