LMDeploy
Toolkit for compressing and serving large language models
LMDeploy is an inference and serving toolkit from the OpenMMLab ecosystem for deploying large language models efficiently. It offers quantization, a high-throughput serving engine, and an OpenAI-compatible API.
Key features
- High-throughput inference engine
- Weight quantization support
- OpenAI-compatible serving
- Multi-GPU deployment
Strengths
- Released under the Apache-2.0 license
- First-class Docker support for quick deployment
- Mature project with 8k GitHub stars
- Written in Python
LMDeploy replaces
Compare LMDeploy
34 head-to-head comparisons.
- LMDeploy vs Ollama
- LMDeploy vs Hugging Face Transformers
- LMDeploy vs llama.cpp
- LMDeploy vs vLLM
- LMDeploy vs GPT4All
- LMDeploy vs GPT4Free
- LMDeploy vs LiteLLM
- LMDeploy vs LocalAI
- LMDeploy vs exo
- LMDeploy vs New API
- LMDeploy vs Jan
- LMDeploy vs FastChat
- LMDeploy vs One API
- LMDeploy vs Continue
- LMDeploy vs SGLang
- LMDeploy vs llamafile
- LMDeploy vs MLC LLM
- LMDeploy vs Guidance
- LMDeploy vs OpenLLM
- LMDeploy vs KoboldCpp
- LMDeploy vs Text Generation Inference
- LMDeploy vs Petals
- LMDeploy vs Xinference
- LMDeploy vs Llama Stack
- LMDeploy vs Page Assist
- LMDeploy vs MLX LM
- LMDeploy vs Enchanted
- LMDeploy vs Serge
- LMDeploy vs GPUStack
- LMDeploy vs LM Studio
- LMDeploy vs Text Embeddings Inference
- LMDeploy vs Harbor LLM Toolkit
- LMDeploy vs ik_llama.cpp
- LMDeploy vs Aphrodite Engine
Similar local llm runners apps
Ollama
Local LLM RunnersRun large language models locally with a simple CLI and API
Replaces ChatGPT, OpenAI API
llama.cpp
Local LLM RunnersHigh-performance LLM inference in plain C/C++
Replaces OpenAI API
vLLM
Local LLM RunnersHigh-throughput LLM serving engine with PagedAttention
Replaces OpenAI API
GPT4All
Local LLM RunnersPrivacy-first desktop chat with local language models
Replaces ChatGPT
LiteLLM
Local LLM RunnersUnified proxy and gateway for 100+ LLM APIs
Replaces OpenRouter
exo
Local LLM RunnersRun your own AI cluster across everyday devices
Replaces OpenAI API