DeepEval

By Confident AI

Open-source "Pytest for LLMs" — unit-test your model and RAG output with metrics.

Website ↗ Repository ↗ Docs ↗

v4.1.32026-07-12PythonApache-2.0

#evaluation #testing #metrics #python

17.1k Open source Free tier Paid

Doc version 1.0 · Updated June 28, 2026

Related concepts

Harness Engineering Doc version 1.0 · Updated June 28, 2026 Wrapping an LLM in scaffolding — a control loop, tools, a sandbox, guardrails, and evaluation — so it behaves reliably in production.

Related tools

Overview

DeepEval is an open-source evaluation framework that treats LLM testing like unit testing — “Pytest for LLMs” — with a library of metrics for relevancy, faithfulness, hallucination, bias, and custom criteria.
It runs locally and in CI, and pairs with the Confident AI platform for hosted dashboards and regression tracking.

flowchart LR
  case[test case] --> m[metric]
  m --> judge[(LLM judge)]
  m --> verdict[score + pass/fail]

The Code samples tab shows scoring a single test case.

When to use it

Choose DeepEval when you want evaluation wired into your test workflow — writing metric-based assertions for model and RAG output, and gating CI on them, with an optional hosted platform on top.

Pricing

DeepEval is open source (Apache-2.0), free. The Confident AI platform adds a hosted dashboard.

Plan	Price	Notes
Starter	$0	Free, hosted dashboard
Premium	$19.99/user/mo	Higher limits
Growth	$49.99/user/mo	Team, more usage
Enterprise	Custom	SSO, SLA, support

Source ↗ · as of 2026-06-27

Example

Python · evaluate a case

When to use

Use DeepEval when you want LLM evaluation that feels like unit testing — assertions and metrics you run locally and in CI.

How it flows

Build LLMTestCases, attach metrics (relevancy, faithfulness, hallucination, and more), and evaluate scores each — pass/fail like a test suite.

Structure

flowchart LR
  case[test case] --> m[metric]
  m --> judge[(LLM judge)]
  m --> verdict[score + pass/fail]

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

case = LLMTestCase(
    input="What language is Qdrant written in?",
    actual_output="Qdrant is written in Rust.",
)
# Metrics use an LLM judge (set OPENAI_API_KEY or configure another).
evaluate(test_cases=[case], metrics=[AnswerRelevancyMetric()])

Example

DeepEval — implementation sample

A real, runnable mini-project. Download it and run with Docker.

View samples/deepeval_1 on GitHub ↗

June 28, 2026

DeepEval — implementation sample

A tiny DeepEval sample: it scores an answer with the Answer Relevancy metric (LLM-as-judge).

Configure

cd samples/deepeval_1
cp .env.sample .env
# edit .env: set OPENAI_API_KEY (used as the judge model)

Run with Docker

cd samples/deepeval_1
docker build -t aas-deepeval .
docker run --rm --env-file .env aas-deepeval

Run with Docker (in a devcontainer with DooD)

The foreground docker run may print nothing under Docker-outside-of-Docker — run detached and follow the logs:

docker logs -f "$(docker run -d --env-file .env aas-deepeval)"

Run locally

cd samples/deepeval_1
pip install -r requirements.txt
python app.py

Example run

Scores use an LLM judge, so the score and reason vary slightly run to run. One run with gpt-4o-mini:

answer_relevancy: 1.00
reason: The score is 1.00 because the response directly addresses the question about the programming language of Qdrant without any irrelevant statements.

Files

.env.sample

# Copy to `.env` and add your OpenAI API key (used as the judge model).
#   cp .env.sample .env
OPENAI_API_KEY=

MODEL=gpt-4o-mini

app.py

"""DeepEval: score an LLM answer with a metric (LLM-as-judge).

Builds one test case and runs the Answer Relevancy metric. The judge model uses
OPENAI_API_KEY. Output is a score (0-1) with a short reason.

Run:
    docker build -t aas-deepeval .
    docker run --rm --env-file .env aas-deepeval
"""

import os

from dotenv import load_dotenv
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

load_dotenv()

MODEL = os.environ.get("MODEL", "gpt-4o-mini")


def main() -> None:
    case = LLMTestCase(
        input="What language is Qdrant written in?",
        actual_output="Qdrant is written in Rust.",
    )
    metric = AnswerRelevancyMetric(model=MODEL)
    metric.measure(case)
    print(f"answer_relevancy: {metric.score:.2f}")
    print(f"reason: {metric.reason}")


if __name__ == "__main__":
    main()

Dockerfile

FROM python:3.13-slim

ENV PYTHONUNBUFFERED=1

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

ENTRYPOINT ["python", "app.py"]