DeepEval

Last reviewed Aug 26, 2026 · 869 words

The thing DeepEval gets right is a single decision: an LLM evaluation is a pytest test. It has a name, it asserts, it passes or fails, it runs in CI on every commit, and it lives next to your other tests instead of in a notebook or a dashboard. There is no server to host. What you run is pip install deepeval on the machine that already runs your test suite, and the "self-hosting" question is really about where the judge model lives.

What it is and where it sits

DeepEval is a Python library, Apache-2.0, 17,861 stars on GitHub, and it targets one layer of an AI stack: the gate between "the model produced something" and "we ship this prompt". You write a test, it runs your app against a set of inputs, and a set of metrics score the outputs. There are 14 or more built in: answer relevancy, faithfulness and contextual precision for RAG pipelines, hallucination, bias, toxicity, summarisation quality, plus a general-purpose GEval metric where you describe the criteria in plain English and let a judge model grade against them.

Almost all of those metrics are LLM-as-judge. That is the important architectural fact. DeepEval does not score anything itself; it sends the input, output and your criteria to a model and asks for a verdict. So the cost, the speed and the privacy of your evals are the cost, speed and privacy of whichever model you point it at.

A test looks like this

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric

def test_refund_answer():
    case = LLMTestCase(
        input="How do I get a refund?",
        actual_output=my_app("How do I get a refund?"),
    )
    assert_test(case, [AnswerRelevancyMetric(threshold=0.7)])

Run it with deepeval test run test_refunds.py, which wraps pytest and adds parallel execution and a results table. Plain pytest works too. A threshold below which the test fails is the whole control surface: start loose, tighten as the prompt improves, and a regression in the prompt shows up as a red build rather than a customer complaint three weeks later.

Pointing the judge at your own hardware

By default the metrics call OpenAI, which is fine for a team with a budget and no data-residency concerns and expensive for everyone else: every test case costs a judge call, some metrics make several, and a 200-case suite on every push adds up. The library ships a command to redirect it:

deepeval set-ollama llama3.1:8b

That routes every metric through a local Ollama instance, and there is an equivalent set-local-model for any OpenAI-compatible endpoint such as vLLM. Two honest caveats. An 8B judge is a worse grader than a frontier model and will be noisier on subtle criteria like faithfulness; I have found it good enough for relevancy and toxicity gates and unreliable for fine-grained GEval rubrics. And judge calls are slow on consumer GPUs, so a suite that took 2 minutes against an API can take 20 locally. The local LLM hardware guide covers what a judge-capable box looks like; a 70B-class model at 4-bit quantisation grades far better than 8B if you can fit it.

The cloud upsell is real but optional

The company behind DeepEval sells Confident AI, a hosted dashboard for test history, dataset management and tracing, and the CLI nudges you toward logging in. You do not need it. Results print to the terminal and can be written to JSON, datasets live as files in your repo, and if you want traces of production calls you can already run Langfuse for that. The synthetic dataset generator, which builds test inputs from your documents, runs locally with your chosen model and is genuinely useful for bootstrapping a RAG test set from a docs folder.

For a self-hoster the sensible split is: DeepEval in CI as the pass/fail gate, Langfuse in production as the trace store, and a small shared dataset that grows every time production surfaces a bad answer. The observability guide walks through the tracing half.

When Promptfoo is the better pick

Promptfoo covers similar ground from a YAML-and-CLI direction with a matrix view for comparing prompts and models side by side, and it needs no Python. If your question is "which of these 4 prompts on which of these 3 models", Promptfoo is quicker to answer it. If your question is "does the app still behave after this refactor", DeepEval's pytest shape wins, because it plugs into a workflow your team already has. Plenty of teams run both.

What I'd do

Add DeepEval to the existing test suite, start with 2 metrics (relevancy and either faithfulness or a single GEval rubric), and a dataset of 30 real inputs. Judge against a local Ollama model for the fast pre-commit run and against a stronger API model for the nightly run, so the expensive grader only fires once a day. Skip the cloud dashboard. Treat the thresholds like any other test assertion: tuned to catch real regressions, not to look green.