LiteLLM

Last reviewed Aug 26, 2026 · 883 words

LiteLLM is the container that stops your apps from caring which model they talk to. Point Open WebUI, a coding agent, and your own scripts at one http://litellm:4000/v1 endpoint, and LiteLLM routes each request to Ollama on the LAN, OpenAI, Anthropic, Bedrock, or 100+ other providers, records what it cost, and fails over when a provider is down. It runs no models itself, and if you have exactly 1 app calling exactly 1 local Ollama, you do not need it. The moment you have 2 apps or 2 providers, you do.

It's a gateway, not a runner

The catalogue files it under LLM runners, which is the right shelf but the wrong mental model. LiteLLM is a Python proxy (MIT, 57,000+ GitHub stars, first released 2023) that speaks the OpenAI chat-completions dialect on the front and translates to each provider's real API on the back. The local half of your stack still needs Ollama or vLLM doing the inference, and LiteLLM's 1 GB RAM footprint sits on top of whatever those need. What that gigabyte buys is one URL, one key format, and one place to look when a bill or a latency spike surprises you.

The 12-line config that earns its keep

Everything LiteLLM does flows from config.yaml. A minimal one that fronts a local model and a cloud model under friendly names:

model_list:
  - model_name: fast
    litellm_params:
      model: ollama_chat/llama3.1
      api_base: http://ollama:11434
  - model_name: smart
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
litellm_settings:
  drop_params: true

And the container that reads it:

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    environment:
      - LITELLM_MASTER_KEY=sk-change-me
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    restart: unless-stopped

Clients now ask for fast or smart and never learn which vendor sat behind the name. Moving smart to a different provider is a 1-line edit and 0 client changes, which is the entire point of the tool. drop_params: true quietly discards OpenAI-only request parameters that a local model would reject, and it removes a class of HTTP 400 errors you would otherwise chase across two codebases. The master key must start with sk-; LiteLLM rejects anything else at startup.

Without Postgres you're running half the product

Stateless, LiteLLM is a translator, and for a single-user lab that is enough. Add DATABASE_URL=postgresql://... to the environment and it becomes a platform: virtual API keys with per-key budgets, per-team spend reports, rate limits, and an admin UI at /ui where you mint a key for each app. Give Open WebUI a key capped at $10 a month and your coding agent a separate one capped at $50, and the day one of them loops on an expensive model you lose $50, not a mortgage payment. Postgres is the only supported database for this, and a plain postgres:16 container on the same host handles it with room to spare.

Fallbacks are the feature you'll be grateful for at 2 a.m.

Two model_list entries sharing a model_name become a pool, and LiteLLM spreads requests across them. A fallbacks block under router_settings says "if smart fails, retry on fast", so a cloud outage degrades your app to a local model instead of an error page. Deployments that fail repeatedly are pulled from rotation for a cooldown and returned automatically. I run every home agent through this and have not hand-edited a base URL during a provider incident since. The LLM API resilience piece covers the retry math in detail; the short version is that LiteLLM implements most of it in about 6 lines of YAML.

Observability is 2 more lines. Set success_callback: ["langfuse"] under litellm_settings, supply the Langfuse keys as environment variables, and every request lands in a self-hosted Langfuse with cost, latency, and full prompts attached.

Config sprawl and the enterprise line are both real

The catalogue's two cons are accurate. A config that fronts 30 models with per-model rate limits, caching, and guardrails runs to several hundred lines, and there is no schema validator that catches a typo before a request fails. Keep the file in git, keep it small, and delete models nobody calls. Second, LiteLLM is open core: the proxy, routing, budgets, virtual keys, and Redis caching are MIT, while SSO for the admin UI, audit logs, and some guardrail and metrics features need an enterprise licence. For a homelab or a team of 10, none of those bite. If you want SSO on the admin panel anyway, a forward-auth proxy in front of /ui gets you 90% of the way for free.

What I'd do

The compose file above with Postgres added on day one, one virtual key per app with a monthly budget, fast on Ollama and smart on whichever cloud model you already pay for, and a fallback from smart to fast. Then point every client at port 4000 and stop pasting provider URLs into apps. It is the least glamorous container in the self-hosted AI stack and the one I would rebuild first after a disk failure.