SHSelfHostIndex

New API vs Text Generation Inference

A side-by-side comparison of two self-hosted local llm runners options — licensing, setup difficulty, resource needs, and what each one replaces.

Not the right match-up?
NE New API Next-gen LLM gateway and AI asset management system VS TE Text Generation Inference Archived Hugging Face toolkit for production LLM serving
FeatureNew APIText Generation Inference
Deploy effortUnder-an-hour setupUnder-an-hour setup
Health score100 · Excellent15 · At risk
StatusActively maintainedArchived
CategoryLocal LLM RunnersLocal LLM Runners
LicenseMITApache-2.0
LanguageGoPython
Setup difficultyMediumHard
Min. RAM512 MB16,384 MB
Deploymentdockerdocker, kubernetes
GitHub stars★ 48,815★ 10,884
First released20242022
ReplacesOpenRouter, OpenAI APIOpenAI API

What are New API and Text Generation Inference?

New API

New API is a large-model interface management and distribution system derived from One API, providing an OpenAI-compatible gateway with richer billing, rendering and provider support. It is designed for running a self-hosted AI relay service.

  • OpenAI-compatible relay
  • Multi-provider channels
  • Detailed billing
  • Modern dashboard

Text Generation Inference

Text Generation Inference is Hugging Face's production-grade toolkit for deploying and serving large language models. It offers optimized transformers code, continuous batching, and an OpenAI-compatible API.

  • Production LLM serving
  • Continuous batching
  • Tensor parallelism
  • OpenAI-compatible endpoint

New API vs Text Generation Inference: key differences

The biggest difference is maintenance: Text Generation Inference's repository is archived and no longer developed, while New API is actively maintained. New API is written in Go, while Text Generation Inference is built with Python. Licensing differs — MIT for New API versus Apache-2.0 for Text Generation Inference. New API is the lighter option, starting around 512 MB of RAM against 16,384 MB for Text Generation Inference. Text Generation Inference is the more established project (first released 2022), while New API arrived in 2024. New API has the considerably larger community, at 48,815 GitHub stars versus 10,884.

Why pick each one

Choose New API if…

  • OpenAI-compatible gateway
  • Broad provider support
  • Built-in billing and quotas

Watch out for

  • Docs primarily Chinese
  • Niche relay use case
New API details

Choose Text Generation Inference if…

  • Released under the Apache-2.0 license
  • First-class Docker support for quick deployment
  • Kubernetes-ready with Helm charts available
  • Mature project with 10.9k GitHub stars
Text Generation Inference details

Frequently asked questions

Is New API or Text Generation Inference better?

New API is the stronger all-round pick: it has both the larger community and the simpler medium setup. Consider Text Generation Inference if its specific feature set fits your needs better.

Are New API and Text Generation Inference free and open-source?

Yes. New API is licensed under MIT and Text Generation Inference under Apache-2.0. Both can be self-hosted at no software cost.

Can I run New API and Text Generation Inference with Docker?

New API: yes. Text Generation Inference: yes.

Which is lighter on resources, New API or Text Generation Inference?

New API has the smaller minimum footprint at 512 MB of RAM, compared to about 16,384 MB for Text Generation Inference. Real-world usage depends on library size, user count, and enabled features.

Related comparisons