MLC LLM

Last reviewed Aug 26, 2026 · 826 words

MLC LLM is the runner I reach for when the target is not a Linux box with an NVIDIA card. On that box, Ollama or llama.cpp gets a working model in 5 minutes and MLC's compile-first design is pure overhead. On an Android phone, an iPhone, a Vulkan-only AMD or Intel GPU, or a browser tab with WebGPU, MLC is frequently the only serious option, and its throughput on those targets is the best I have measured. Treat the Hard difficulty rating as accurate: this is a compiler toolchain with a chat command bolted on, not an app.

The compile step is the whole product

Every other local runner loads a GGUF or safetensors file and interprets it at runtime. MLC instead takes the model weights plus a target (CUDA, ROCm, Metal, Vulkan, WebGPU, OpenCL for Android) and uses Apache TVM to generate a kernel library tuned for that exact hardware. The result is a directory of quantised weights in MLC's own format plus a compiled .so, .dylib or .wasm library. This is why it runs fast on odd hardware and why 4 GB of RAM is the floor for the host: the build machine needs headroom even when the target is a phone.

The saving grace is that the mlc-ai organisation on Hugging Face publishes prebuilt weights and libraries for popular models, so most people never run the compiler. A first chat, once the wheels for your CUDA or Metal version are installed per the install page at llm.mlc.ai, is one line:

mlc_llm chat HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC

The q4f16_1 suffix is the quantisation scheme: 4-bit weights, 16-bit activations. It lands in the same size class as a Q4_K_M GGUF, so the VRAM math you already know carries over: an 8B model at 4-bit wants about 5 to 6 GB of GPU memory with room for context. Four bits costs a little quality against the 8-bit original, and less than most people expect.

Where it beats Ollama, and where it does not

The last three rows are the reason to learn this tool. WebLLM, the JavaScript sibling, runs the compiled model inside Chrome or Edge with WebGPU, no server involved, which makes a private "chat with this page" demo you can host as static files. The Android and iOS apps let an 8B model at 4-bit run on a flagship phone at usable speeds, entirely offline, and that is a capability nothing in the runner category otherwise offers.

mlc_llm serve gives you an OpenAI-compatible API

mlc_llm serve HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC --host 0.0.0.0 --port 8000

That exposes /v1/chat/completions on port 8000, the same interface Open WebUI, Continue and most agent frameworks expect. Point Open WebUI at it as an OpenAI-compatible endpoint with any placeholder key. Streaming works. Tool calling and JSON mode exist, but test them against your exact model before building on them, because chat-template handling differs by model family and MLC's templates lag the newest releases by weeks.

The toolchain is where the Hard rating comes from

Compiling your own model, say a fine-tune with no prebuilt MLC weights, means installing TVM with the right backend, then running mlc_llm convert_weight, mlc_llm gen_config and mlc_llm compile in sequence. Each step has flags for quantisation, context window and target. Budget an afternoon the first time and expect CUDA version mismatches to eat most of it. If your models are all mainstream and prebuilt, skip all of this and treat MLC purely as a runtime.

The catalogue is the other limit. The prebuilt list covers Llama, Mistral, Qwen, Gemma, Phi and a few dozen others, while Ollama and llama.cpp can load nearly any GGUF the day it appears on Hugging Face. If your habit is trying new releases weekly, MLC will frustrate you within a month.

What I'd do

On a home server with an NVIDIA GPU, I would not run MLC LLM at all; Ollama is the right default there. I would install MLC for exactly three jobs: an offline assistant on a phone, a WebGPU demo I can host as static HTML, and getting real speed out of a Vulkan-only GPU that llama.cpp treats as second class. For those, use the prebuilt weights, never compile unless forced, and front the serve endpoint with Open WebUI so nobody else in the household ever sees a terminal.