Text Generation Inference
Hugging Face toolkit for production LLM serving
This project is archived. Upstream development has stopped, so expect no new features or security fixes. Consider an actively maintained local llm runners alternative.
Text Generation Inference is Hugging Face's production-grade toolkit for deploying and serving large language models. It offers optimized transformers code, continuous batching, and an OpenAI-compatible API.
Key features
- Production LLM serving
- Continuous batching
- Tensor parallelism
- OpenAI-compatible endpoint
Strengths
- Released under the Apache-2.0 license
- First-class Docker support for quick deployment
- Kubernetes-ready with Helm charts available
- Mature project with 10.9k GitHub stars
Text Generation Inference replaces
Compare Text Generation Inference
15 head-to-head comparisons.
- Text Generation Inference vs Ollama
- Text Generation Inference vs llama.cpp
- Text Generation Inference vs vLLM
- Text Generation Inference vs New API
- Text Generation Inference vs exo
- Text Generation Inference vs FastChat
- Text Generation Inference vs One API
- Text Generation Inference vs llamafile
- Text Generation Inference vs MLC LLM
- Text Generation Inference vs OpenLLM
- Text Generation Inference vs Petals
- Text Generation Inference vs LMDeploy
- Text Generation Inference vs ik_llama.cpp
- Text Generation Inference vs Aphrodite Engine
- Text Generation Inference vs Wllama
Similar local llm runners apps
Ollama
Local LLM RunnersRun large language models locally with a simple CLI and API
Replaces ChatGPT, OpenAI API
llama.cpp
Local LLM RunnersHigh-performance LLM inference in plain C/C++
Replaces OpenAI API
vLLM
Local LLM RunnersHigh-throughput LLM serving engine with PagedAttention
Replaces OpenAI API
GPT4All
Local LLM RunnersPrivacy-first desktop chat with local language models
Replaces ChatGPT
LiteLLM
Local LLM RunnersUnified proxy and gateway for 100+ LLM APIs
Replaces OpenRouter
New API
Local LLM RunnersNext-gen LLM gateway and AI asset management system
Replaces OpenRouter, OpenAI API