promptfoo

Last reviewed Aug 26, 2026 · 860 words

Here is the whole workflow: a promptfooconfig.yaml holding your prompt, 3 providers and 20 test cases; npx promptfoo@latest eval; then promptfoo view on localhost and a matrix of which model passed which assertion. Everything runs on your machine, the only network traffic goes to the model endpoints you chose, and those endpoints can be an Ollama container on the same box. promptfoo (TypeScript, MIT, 24,574 stars, first released 2023) is the tool that makes "we tried a new prompt and it felt better" into a number, and it needs Node.js and about 1 GB of RAM to do it.

The config does the work

A minimal file that compares two local models against a hosted one on a summarisation task:

prompts:
  - "Summarise this support ticket in one sentence: {{ticket}}"
providers:
  - ollama:chat:llama3.1
  - ollama:chat:qwen2.5:14b
  - openai:gpt-4o-mini
tests:
  - vars:
      ticket: "My August invoice shows two charges for the same plan."
    assert:
      - type: icontains
        value: invoice
      - type: javascript
        value: output.length < 200
      - type: llm-rubric
        value: Mentions the duplicate charge and nothing unrelated
      - type: latency
        threshold: 3000

Each test runs each prompt against each provider, so 20 tests, 2 prompts and 3 providers is 120 calls, and promptfoo caches responses so re-running after editing one assertion costs nothing. The viewer shows the grid with pass rates per column; that column comparison is the feature, because it is the first honest answer to "can the 14B local model replace the API for this task". Put vars in a CSV instead when the test cases outgrow the YAML, and the tests: file://cases.csv line does the rest.

Deterministic assertions first, model-graded second

The assertion types split into two families. contains, regex, is-json, equals, javascript and python are deterministic, cost nothing, and never disagree with themselves; use them for structure, length, forbidden strings and every rule you can state precisely. llm-rubric, similar, factuality and model-graded-closedqa ask another model to judge, which is the only way to test tone or completeness but introduces a second model's opinion into your test suite. Two habits keep that sane. Pin the grader explicitly with defaultTest.options.provider rather than accepting the default, and make it a local model if you want the suite to run with no API key at all. And when a rubric test flips between runs, that is information about the rubric wording, not noise to suppress. The evals before vibes post covers how to build the case set itself.

Red-teaming runs from the same tool

promptfoo redteam init walks you through describing the application, then generates adversarial test cases across plugins for prompt injection, jailbreaks, PII leakage, harmful content and a long list of others, and promptfoo redteam run fires them at your target, which can be an HTTP endpoint for your actual app rather than a bare model. The generation step itself uses a model to write the attacks, so it wants a capable one; a small local model produces weaker attacks, which is a real limitation of the fully offline route. The report in the viewer groups findings by category and severity and is the first thing I would show a security reviewer who asks what testing an LLM feature has had.

Keeping it local, and what to switch off

Three settings for the privacy-minded. PROMPTFOO_DISABLE_TELEMETRY=1 in the environment turns off anonymous usage reporting. promptfoo share uploads results to the hosted viewer only when you run it, so simply do not; results otherwise stay in the local SQLite store under ~/.promptfoo. And if the team wants a shared results server, the same viewer runs as a self-hosted container behind your own proxy. In CI, the published GitHub Action or a plain npx promptfoo eval --no-cache in any pipeline fails the build when pass rate drops below the threshold you set, which is the point at which prompt changes start getting reviewed like code.

promptfoo is before deploy, Langfuse is after

People shopping for a LangSmith alternative often conflate two jobs. promptfoo tests prompts and models before you ship: fixed inputs, expected properties, pass or fail. Langfuse traces what real users did after you shipped: latency, cost, the actual outputs. You want both, they do not overlap, and the neat loop is exporting the worst production traces from Langfuse into promptfoo's test cases so next week's eval includes last week's failures. Local models for either come from Ollama or any OpenAI-compatible server.

What I'd do

Start with 20 real inputs from the feature you are building, 4 deterministic assertions each and 1 rubric, and run it against your two candidate local models plus the API you are trying to stop paying for. Wire promptfoo eval into CI with a pass-rate threshold. Add redteam before the feature faces the public, with a strong generator model for that one run even if the rest of the suite is offline. Node, a YAML file and an afternoon, and you never argue about prompts from feel again.