SHSelfHostIndex
LM

LM Evaluation Harness

Unified framework to benchmark language models on many tasks

Self-Hosted AI ★ 14.2k stars Medium setup MIT

The Language Model Evaluation Harness from EleutherAI is a framework for evaluating generative language models on a large number of standardized benchmarks. It can be run locally to assess self-hosted models.

Official siteSource code

Key features

  • Hundreds of benchmarks
  • Many model backends
  • Reproducible evaluation
  • Standard in research

Pros & cons

Strengths

  • De facto research standard
  • Hundreds of tasks included
  • Supports local model backends

Trade-offs

  • Not a server or web app
  • Evaluations can take hours

Is LM Evaluation Harness secure?

OSV.dev has no published security advisories for LM Evaluation Harness's repository. No known vulnerabilities is not proof of security: it means none have been publicly catalogued.

Source: OSV.dev, checked 2026-10-14. How this is measured.

LM Evaluation Harness replaces

Self-hosted OpenAI Evals alternatives

Health badge

Maintain LM Evaluation Harness? Add its health score to your README. The badge links back to this page.

SelfHostIndex: 94 · excellent

Markdown
[![SelfHostIndex health score for LM Evaluation Harness](https://selfhostindex.com/badge/lm-evaluation-harness.svg)](https://selfhostindex.com/app/lm-evaluation-harness/)
HTML
<a href="https://selfhostindex.com/app/lm-evaluation-harness/"><img src="https://selfhostindex.com/badge/lm-evaluation-harness.svg" alt="SelfHostIndex health score for LM Evaluation Harness"></a>
reStructuredText
.. image:: https://selfhostindex.com/badge/lm-evaluation-harness.svg
   :alt: SelfHostIndex health score for LM Evaluation Harness
   :target: https://selfhostindex.com/app/lm-evaluation-harness/

The score is recomputed on every nightly rebuild from public repository data — see how it is calculated.

Similar self-hosted ai apps