SHSelfHostIndex
TE

Text Generation Inference

Hugging Face toolkit for production LLM serving

Local LLM Runners Archived ★ 10.9k stars Hard setup Apache-2.0

This project is archived. Upstream development has stopped, so expect no new features or security fixes. Consider an actively maintained local llm runners alternative.

Text Generation Inference is Hugging Face's production-grade toolkit for deploying and serving large language models. It offers optimized transformers code, continuous batching, and an OpenAI-compatible API.

Official siteSource code

Key features

  • Production LLM serving
  • Continuous batching
  • Tensor parallelism
  • OpenAI-compatible endpoint

Strengths

  • Released under the Apache-2.0 license
  • First-class Docker support for quick deployment
  • Kubernetes-ready with Helm charts available
  • Mature project with 10.9k GitHub stars

Is Text Generation Inference secure?

1 known vulnerability affects Text Generation Inference's latest release (v3.3.7) according to OSV.dev; the most severe is rated medium.

Affecting v3.3.7

AdvisorySeveritySummary
CVE-2026-63086Mediumtext-generation-inference 3.3.7 SSRF via fetch_image in multimodal chat completions
  • Most recent fixed advisory CVE-2026-0599, published 2026-02-02 (254 days ago), fixed in 3.3.7

Source: OSV.dev, checked 2026-10-14. How this is measured.

Text Generation Inference replaces

Self-hosted OpenAI API alternatives

Health badge

Maintain Text Generation Inference? Add its health score to your README. The badge links back to this page.

SelfHostIndex: 15 · archived

Markdown
[![SelfHostIndex health score for Text Generation Inference](https://selfhostindex.com/badge/tgi.svg)](https://selfhostindex.com/app/tgi/)
HTML
<a href="https://selfhostindex.com/app/tgi/"><img src="https://selfhostindex.com/badge/tgi.svg" alt="SelfHostIndex health score for Text Generation Inference"></a>
reStructuredText
.. image:: https://selfhostindex.com/badge/tgi.svg
   :alt: SelfHostIndex health score for Text Generation Inference
   :target: https://selfhostindex.com/app/tgi/

The score is recomputed on every nightly rebuild from public repository data — see how it is calculated.

Compare Text Generation Inference

15 head-to-head comparisons.

Similar local llm runners apps