FastChat

Last reviewed Aug 26, 2026 · 898 words

Running one model with an OpenAI-compatible API takes a single process in vLLM or Ollama. FastChat needs four: a controller, at least one model worker, the API server, and optionally a Gradio web UI. That overhead buys you two things nobody else in the LLM runners category packages together: a controller that routes requests across many models and many GPUs behind one endpoint, and the evaluation harness (MT-Bench, pairwise judging) that the original Chatbot Arena was built on. If you want either of those, FastChat is still the reference implementation. If you want a chat endpoint for one model, it is the wrong tool and the "Hard" difficulty rating in the catalogue is a fair warning.

Four processes before the first token

The minimum working setup, run in four terminals or four systemd units:

python3 -m fastchat.serve.controller
python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5
python3 -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 8000
python3 -m fastchat.serve.gradio_web_server   # optional chat UI on 7860

The controller listens on 21001 and keeps a registry of workers; each worker announces itself with a model name and a heartbeat; the API server on port 8000 speaks /v1/chat/completions and asks the controller where a given model lives. Kill the controller and nothing else can find anything, so it is the process to supervise most carefully. The default model_worker loads weights through Hugging Face Transformers, which for a 7B model in fp16 means about 14 GB of VRAM, and the catalogue's 8 GB RAM floor assumes the model is on a GPU rather than in system memory. The VRAM math post has the arithmetic for other sizes and quantisations.

Swap the default worker for the vLLM worker on day one

The Transformers-based worker is correct and slow: no continuous batching, no paged attention, one request at a time in practice. FastChat ships a fastchat.serve.vllm_worker that wraps vLLM's engine and registers with the same controller, and with it the same 7B model handles a dozen concurrent chats at throughput the default worker cannot approach. There are also workers for llama.cpp GGUF files, SGLang, and LightLLM. The controller does not care which backend a worker uses; that indifference is the whole design. So the practical FastChat stack in 2026 is "vLLM doing the inference, FastChat doing the routing", and the vLLM vs llama.cpp piece will tell you which engine belongs under each of your models.

One endpoint, many models, is the feature

Register three workers, llama-3.1-8b on GPU 0, qwen2.5-14b on GPU 1, and a small phi model on CPU for cheap classification, and clients pick between them with nothing but the model field in a standard OpenAI request. Add a second worker for a busy model and the controller load-balances between them, either by shortest queue or lottery. This is what teams reach for when a product needs several models with different cost and quality profiles behind one internal URL, and the model-routing pattern of trying a small model first is a few lines of client code on top of it. Ollama can hold several models but serves them from one process on one machine; vLLM serves one model per process. FastChat is the layer that makes a rack of mixed GPUs look like one API.

MT-Bench is the other half of the repo

fastchat/llm_judge contains MT-Bench: 80 multi-turn questions across 8 categories, plus the scripts to generate answers from your served model and have a judge model score them, absolute or pairwise. It is how you find out whether your fine-tune got better or just different, and it is the same methodology that seeded the public Arena leaderboard before that project grew into LMArena. Running it costs a few dollars of judge-model calls against a hosted API, or nothing if you point the judge at a strong local model. Read the evals-before-vibes post on this site first; the harness is only as useful as your willingness to trust a score you did not like.

The project is quieter than its star count suggests

FastChat's 39,524 stars were mostly earned in 2023 when Vicuna was the open model to beat and the Arena ran on this code. Since then the Arena became a separate organisation, vLLM and SGLang absorbed most of the serving innovation, and FastChat's release cadence has slowed to maintenance. It still works, the worker abstraction still holds, and nothing has replaced its controller for heterogeneous fleets. But you are adopting a stable codebase, not a fast-moving one, and new model architectures may need a vLLM or Transformers upgrade rather than a FastChat change.

What I'd do

One model, one GPU: skip FastChat, run vLLM directly, put Open WebUI in front if you want a chat page. Two or more models across mixed hardware, or a fine-tuning loop that needs MT-Bench: FastChat controller and API server on the coordinating box, one vllm_worker per model on the GPU boxes, everything pinned in a virtualenv, controller under systemd with restart on failure. Point clients at port 8000 and never let them know how many machines are behind it.