Ollama will serve a small team from one GPU without drama, but not with its defaults, which are tuned for one person on a laptop. Four environment variables do most of the work: OLLAMA_NUM_PARALLEL, OLLAMA_KEEP_ALIVE, OLLAMA_FLASH_ATTENTION, and OLLAMA_KV_CACHE_TYPE — plus the non-obvious fact that every parallel slot allocates its own context window. Tuned like that, Ollama comfortably handles 10–15 concurrent chat users running an 8B model on a 24GB card. Past that point, stop tuning and move the workload to vLLM.

The defaults are laptop defaults

Out of the box Ollama binds to 127.0.0.1, unloads models after 5 minutes of idle, runs without flash attention, and keeps the KV cache in FP16. Every one of those is wrong for a shared server. On a systemd install, override them properly instead of editing the unit file:

# systemctl edit ollama.service
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=-1"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_MAX_LOADED_MODELS=2"

Then systemctl restart ollama. KEEP_ALIVE=-1 pins models in VRAM permanently: a cold load of an 8B Q4 model from NVMe takes 3–6 seconds and a 70B closer to 30, and users interpret both as "it's down". MAX_LOADED_MODELS=2 lets a chat model and an embedding model coexist instead of evicting each other on every alternating request. The full variable list lives in the Ollama FAQ.

Parallelism multiplies your KV cache

OLLAMA_NUM_PARALLEL=4 gives you four simultaneous request slots, and each slot gets its own context allocation. The arithmetic matters: a Llama-class 8B model costs about 128KB of KV cache per token at FP16, so four slots at 8,192 tokens each is roughly 4GB of KV on top of ~5GB of Q4 weights. OLLAMA_KV_CACHE_TYPE=q8_0 halves that with no change I can detect in chat output (it requires flash attention to be on, which is why the two settings travel together).

Two related traps. First, Ollama's default context length is 4,096 tokens regardless of what the model supports — raise it deliberately with OLLAMA_CONTEXT_LENGTH or num_ctx, and budget VRAM as (total − weights − ~1GB overhead) ÷ (KV per token × parallel slots). Second, requests beyond your slot count queue rather than fail, up to OLLAMA_MAX_QUEUE (default 512), so an undersized NUM_PARALLEL shows up as mysterious latency, not errors. And shape the slots to the traffic: for short-prompt workloads like classification or autocomplete, 8 slots at 4k context serve more users than 4 slots at 8k for exactly the same KV budget.

The API surface people miss

Ollama has shipped an OpenAI-compatible endpoint at /v1 for ages; point any OpenAI SDK at http://server:11434/v1 with a dummy key and most tooling works unmodified. The underrated part is embeddings: /api/embed (or /v1/embeddings) with nomic-embed-text (a 274MB model) or bge-m3 turns the same box into your RAG embedding service. Batch your embed calls — one request with 64 inputs is dramatically faster than 64 requests — and keep the embedding model resident via MAX_LOADED_MODELS so it never fights the chat model for a load slot. Tool calling also works through the /v1 endpoint, but template support varies by model — verify your specific model emits well-formed calls before building an agent on it, because a bad template fails as garbled text, not as an error.

Never expose port 11434 raw

Ollama has no authentication. Anyone who can reach the port can run prompts, pull 40GB models onto your disk, and delete the ones you use. Shodan reliably turns up thousands of open Ollama instances, and scraping them is a hobby now. On a LAN, firewall the port to trusted hosts. For a team, put Open WebUI in front: it adds user accounts, per-model permissions, and its own API keys while talking to Ollama over localhost. For remote access as a solo user, Tailscale is the least-effort answer; a reverse proxy with basic auth or OIDC works too. Whatever you pick, the invariant is that 11434 itself is never reachable from anything you don't trust.

When to graduate to vLLM

Ollama runs on llama.cpp underneath, which is optimised for running models on limited hardware, not for aggregate throughput. On my 4090 with an 8B model (estimates, single machine): single-stream decode is ~110 tokens/s under both stacks, but at 16 concurrent requests Ollama plateaus around 350 tokens/s aggregate while vLLM with an AWQ quant clears 1,800, because continuous batching keeps the GPU saturated instead of round-robining slots. Graduate when any of these are true: you sustain more than ~10 concurrent requests, you have latency SLOs, you need guided/structured output at scale, or you want tensor parallelism across GPUs. The costs of graduating are real too — vLLM wants the whole model in VRAM (no CPU offload), it's a Python service with dependencies rather than a single binary, and GGUF support is second-class. The full comparison is in vLLM vs llama.cpp.

What I'd do

For a homelab or a team of ten or fewer: keep Ollama, apply the six environment variables above, put Open WebUI in front for accounts, firewall the port, and pin your models with keep_alive=-1. Measure actual concurrency for a week (Open WebUI's usage view or your proxy logs are enough) before believing you need more. If you're sustaining 15+ concurrent users or serving a product, skip the intermediate tuning heroics and move inference to vLLM on the same hardware — the migration is mostly changing a base URL, since both speak the OpenAI protocol. And if neither fits exactly, the LLM runner category has the middle-ground options (llama-swap, LocalAI, TabbyAPI) that trade some of Ollama's convenience for more control.