SHSelfHostIndex
EX

ExLlamaV2

Fast inference library for quantized LLMs on consumer GPUs

Self-Hosted AI ★ 4.6k stars Hard setup MIT

ExLlamaV2 is an inference library optimized for running quantized large language models efficiently on modern consumer GPUs. Its EXL2 quantization format allows flexible bitrates for the best speed-quality balance.

Official siteSource code

Key features

  • EXL2 flexible quantization
  • Fast single-GPU inference
  • Low memory footprint
  • Built-in server

Strengths

  • Released under the MIT license
  • Active community (4.6k GitHub stars)
  • Written in Python

Is ExLlamaV2 secure?

OSV.dev has no published security advisories for ExLlamaV2's repository. No known vulnerabilities is not proof of security: it means none have been publicly catalogued.

Source: OSV.dev, checked 2026-10-14. How this is measured.

ExLlamaV2 replaces

Self-hosted OpenAI API alternatives

Health badge

Maintain ExLlamaV2? Add its health score to your README. The badge links back to this page.

SelfHostIndex: 59 · fair

Markdown
[![SelfHostIndex health score for ExLlamaV2](https://selfhostindex.com/badge/exllamav2.svg)](https://selfhostindex.com/app/exllamav2/)
HTML
<a href="https://selfhostindex.com/app/exllamav2/"><img src="https://selfhostindex.com/badge/exllamav2.svg" alt="SelfHostIndex health score for ExLlamaV2"></a>
reStructuredText
.. image:: https://selfhostindex.com/badge/exllamav2.svg
   :alt: SelfHostIndex health score for ExLlamaV2
   :target: https://selfhostindex.com/app/exllamav2/

The score is recomputed on every nightly rebuild from public repository data — see how it is calculated.

Compare ExLlamaV2

10 head-to-head comparisons.

Similar self-hosted ai apps