LocalAI

Last reviewed Aug 26, 2026 · 903 words

LocalAI is the one self-hosted server that answers /v1/chat/completions, /v1/images/generations, /v1/audio/transcriptions, /v1/audio/speech and /v1/embeddings from a single port, which means an application written against the OpenAI SDK needs a changed base URL and nothing else. That breadth is the entire reason to choose it over Ollama. The cost is that each capability is a separate backend with its own model format and configuration file, and on CPU alone every one of them is slow. If you only want chat, stop reading and install Ollama; if you want the whole API on your own hardware, keep going.

Start with the all-in-one image, not the empty one

LocalAI publishes two families of images. The plain ones start with no models and expect you to install or configure each. The AIO images arrive preloaded with a sensible set: a chat model, an embedding model, Whisper for transcription, a text-to-speech voice and a Stable Diffusion model for images, all mapped to the standard OpenAI model names, so gpt-4, whisper-1 and tts-1 resolve to something local on the first request. Begin there:

services:
  localai:
    image: localai/localai:latest-aio-cpu
    ports:
      - "8080:8080"
    environment:
      - LOCALAI_THREADS=8
      - LOCALAI_API_KEY=change-me
    volumes:
      - ./models:/models
    restart: unless-stopped

The first start downloads several gigabytes of weights into ./models, so give it a volume and patience. With an NVIDIA card, swap the tag for latest-aio-gpu-nvidia-cuda-12 and add the GPU reservation to the compose file; AMD and Intel tags exist as well. Once it is up:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer change-me" -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"Say hello in five words."}]}'

Any OpenAI client library works by setting its base URL to http://localhost:8080/v1. Open WebUI connects as an OpenAI-compatible endpoint and gives you chat, image generation and voice in one screen.

The 8 GB minimum is for the AIO set at CPU speed

The catalogue's 8,192 MB figure is honest, and CPU-only is the honest qualifier. A 7B-class chat model quantised to 4 bits needs roughly 5 GB of RAM to load and, on an 8-core desktop CPU, produces somewhere between 5 and 10 tokens per second in my experience, which is usable for a chatbot and painful for anything batch. Whisper on CPU transcribes a few times faster than real time with the small models. Stable Diffusion on CPU is the one that hurts: expect minutes per image rather than seconds. A used GPU with 8 GB or more of VRAM turns all three into a different product, and the hardware guide for self-hosted LLMs walks through which cards are worth the money.

Every model is a YAML file, and that is the learning curve

Beyond the AIO defaults, a model is a YAML file in the models directory that names a backend (llama.cpp for GGUF chat models, whisper.cpp, piper for speech, the diffusers or stablediffusion backends for images, vLLM or transformers for GPU-heavy setups) plus its parameters and prompt template:

name: mistral
backend: llama-cpp
parameters:
  model: mistral-7b-instruct-v0.3.Q4_K_M.gguf
context_size: 8192

The name you write there becomes the model field in API calls. The built-in gallery, reachable in the web UI on port 8080 or with local-ai models install, writes these files for hundreds of models with the correct prompt template included, and that is the sane way to add anything. Hand-editing is for when a gallery entry is wrong, and the template is where hand edits go wrong most often: a mismatched chat template produces a model that answers but rambles, repeats itself or ignores the system prompt. When a new model misbehaves, check the template before blaming the weights.

Where Ollama is simply better

Ollama does one thing, chat and embeddings over GGUF models, with less friction: one binary, ollama pull and ollama run, automatic GPU offload, a large ready-made library. Its API is OpenAI-compatible too, so the base-URL trick works there as well. If your stack is a chat UI plus a coding assistant, Ollama is the better tool and LocalAI is extra moving parts. LocalAI earns its place when the same application needs transcription, speech or images alongside text, or when you are replacing an OpenAI bill whose usage spans several endpoints; the switching notes in OpenAI API alternatives cover that case. It also has a peer-to-peer mode that spreads inference across several machines, which is a fun answer to "I have three old laptops" and not something Ollama attempts.

What I'd do

AIO CPU image on a box with 16 GB of RAM to leave headroom, API key set, Open WebUI pointed at it, and an evening testing each endpoint with the defaults before touching a YAML file. If image generation or speech turn out to matter, that is the moment to buy a used GPU and move to the CUDA tag; if they do not, migrate the chat workload to Ollama and free the RAM. Treat LocalAI as the multi-modal API server it is, not as a slower way to run one chat model.