Ragas

Last reviewed Aug 26, 2026 · 828 words

Every Ragas score is itself an LLM call. Faithfulness, answer relevancy and context precision are computed by prompting a judge model to inspect your question, your retrieved chunks and your answer, so evaluating 200 test questions across 4 metrics means somewhere between 800 and 2,000 generation calls depending on the metric (my estimate; some metrics make several calls per sample). That is the single fact to plan around: the framework is free, Apache-2.0 and lightweight at about 1 GB of RAM for the Python side, but the judge is where the time and money go, and a self-hosted judge on Ollama is what makes the whole thing run on your own infrastructure.

Four metrics, and the failure each one catches

Faithfulness asks whether every claim in the answer is supported by the retrieved context, which catches the model inventing details. Answer relevancy asks whether the answer addresses the question, which catches evasive or padded replies. Context precision asks whether the relevant chunks were ranked above the irrelevant ones, which is a retrieval problem, not a generation problem. Context recall asks whether the chunks contain what a reference answer needed, and is the only one of the four that requires you to supply ground truth. Read them as a pair of pairs: the first two grade the generator, the last two grade the retriever, and a low score tells you which half of the pipeline to fix.

Install is trivial; the dataset shape is the work

pip install ragas is the whole install. What you feed it is a table with one row per test question: the question, the answer your pipeline produced, the list of context chunks it retrieved, and, for recall, a reference answer. Getting 100 honest rows out of a real pipeline means instrumenting your retriever to return chunks alongside the answer, which most quick-start RAG apps do not do. Do that first; the evaluation code afterwards is around 10 lines.

A local judge works; a small one lies

Ragas talks to the judge through LangChain-style wrappers, so an Ollama model drops in with a few lines. Model size matters more than any other setting. In my testing, 7B-class judges produce faithfulness scores that swing by 0.2 or more between runs on the same sample, which makes them useless for detecting a real 0.05 regression. A 27B to 70B class model with a temperature of 0 is where the scores become stable enough to trust, and that means a GPU with 24 GB or more of VRAM or a slow night on CPU. The hardware guide for self-hosted LLMs covers what runs where. If you only have a small model, use it to rank pipeline variants against each other, never to publish an absolute number.

Synthetic test sets get you started, not finished

The test set generator reads your documents and produces question-and-answer pairs of varied difficulty, which is the fastest way from zero to 100 test rows. Treat the output as a draft. Around a fifth of generated questions in my experience are either unanswerable from the source or trivially answerable by string matching, and both inflate your scores. Delete those, add 20 to 30 questions real users actually asked, and freeze the set so runs are comparable over months.

Pin the version; the API has moved

The project is young, first released in 2023, and the API was reworked between its 0.1 and 0.2 series: metric objects, dataset classes and the LLM wrapper interface all changed names. Blog posts and notebooks from the first year no longer run unmodified. Pin ragas in your requirements file, read the migration notes when you bump it, and keep your evaluation script in the same repository as the pipeline it grades so the two move together.

Where it sits next to Langfuse and the observability stack

Ragas answers "how good is this pipeline on a fixed test set". Langfuse answers "what happened on the 4,000 real requests yesterday". They are complementary: run Ragas in CI on every change to the prompt, chunking or embedding model, and ship Langfuse traces to production. Langfuse can also store Ragas scores against traces, so a single dashboard shows offline scores next to live latency. The LLM observability post covers the tracing half.

What I'd do

A frozen test set of 100 to 150 questions, a third of them written by hand from real usage. A judge no smaller than the 27B class, temperature 0, served by Ollama on the GPU box. The four metrics above run on every change to the retrieval stack, with results committed as a CSV so regressions show up in a diff. Absolute scores I would never quote outside the team; deltas between two runs on the same set are the only number Ragas produces that I fully trust.