SGLang

Last reviewed Aug 26, 2026 · 767 words

The deciding question for SGLang versus vLLM is how much of each request your workload repeats. If most calls share a long system prompt, a few-shot preamble, or the growing history of a multi-turn chat, SGLang's RadixAttention keeps that shared prefix's KV cache in a tree and reuses it across requests, and throughput goes up by a multiple rather than a percentage. If every request is a unique document, the two servers run close enough that the choice comes down to which one supports your model and quantisation today. Either way this is a data-centre-style serving framework: 16 GB of system RAM is the floor, a recent NVIDIA GPU is assumed, and "Hard" is the fair difficulty rating.

Launch is one command and the API is OpenAI's

docker run --gpus all -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --host 0.0.0.0 --port 30000

The server exposes /v1/chat/completions, /v1/completions, and /v1/models, so anything that talks to OpenAI talks to it by changing the base URL. Model weights come from Hugging Face on first start; mount the cache or you will download 16 GB again after every restart. pip install "sglang[all]" on a host with the CUDA toolkit is the bare-metal route, and the docs keep an installation matrix, because the compatible combinations of CUDA, PyTorch, and FlashInfer shift with releases. That matrix is most of what "complex tuning options" means in practice.

What RadixAttention buys you, in numbers you can measure

Prefix caching is not unique to SGLang any more; vLLM has it too. What SGLang does is make it the central data structure: a radix tree over token sequences, so a 2,000-token system prompt is computed once and every subsequent request that begins with it skips straight to the new tokens. On a chat product with a long instruction block that can mean the prefill stage drops from hundreds of milliseconds to near zero per request. Measure it yourself: send 50 requests with the same 2,000-token prefix through both servers and compare time-to-first-token from the metrics endpoint. The pattern shows up in the batch versus real-time inference discussion, where prefix reuse is the single biggest lever for agent workloads that call the model in loops with a fixed preamble.

Structured output and vision are first-class

SGLang started as a language for programming LLM calls (the name is "Structured Generation Language"), and the runtime still exposes that. Constrained decoding with a JSON schema or a regex is a request parameter, enforced during generation rather than by retrying, which makes structured output reliable enough to build on. Vision-language models (the Qwen-VL and LLaVA families among others) load with the same launch command and accept image inputs through the OpenAI vision message format. Multi-GPU serving is a flag (--tp 2 for tensor parallelism), and there is data-parallel routing for scaling across replicas.

Where it loses

Consumer hardware. There is no CPU offload story worth using, quantised GGUF files are not the format, and an 8 GB card is below the useful floor. For a single-user homelab with one GPU, llama.cpp or Ollama will serve a 7B model with less setup and similar single-stream latency, and the Ollama vs vLLM comparison covers the same trade at the next tier up. SGLang's win is concurrency: many users, many agent loops, or a batch job where throughput is the bill. Below about 10 concurrent requests you will not see the difference; above 100 you will not want to go back. It also moves fast (first release 2024, 32,452 GitHub stars already), which means features arrive quickly and so do breaking changes.

What I'd do

If you are serving one model to a team or to an application with a shared prompt, run SGLang in the official Docker image on a 24 GB card, put LiteLLM in front for keys and routing, and pin the image tag. If it is one person chatting with one GPU, do not; llama.cpp is the right size. Check your model against the supported list, benchmark prefix-heavy traffic against vLLM for an afternoon, and keep whichever wins on your traffic. With a fixed 3,000-token system prompt and an agent loop, the gap is visible in the first minute.