Ragas

By Exploding Gradients

Open-source evaluation framework for RAG pipelines and LLM applications

Website ↗ Repository ↗ Docs ↗

No updates in 6+ monthsv0.4.32026-01-13PythonApache-2.0

#evaluation #rag #metrics #python

15.0k Completely free Open source

Doc version 1.0 · Updated June 28, 2026

Related concepts

Harness Engineering Doc version 1.0 · Updated June 28, 2026 Wrapping an LLM in scaffolding — a control loop, tools, a sandbox, guardrails, and evaluation — so it behaves reliably in production.

Related tools

Overview

Ragas is an open-source framework for evaluating RAG pipelines and LLM applications, with reference-free metrics like faithfulness, answer relevancy, and context precision.
It turns “does this RAG system actually work?” into numbers you can track in CI and compare across changes.

flowchart LR
  qa[question / answer / context] --> r[Ragas]
  r --> judge[(LLM judge)]
  r --> scores[faithfulness / relevancy]

The Code samples tab shows a minimal RAG evaluation.

When to use it

Choose Ragas when retrieval quality matters and you want objective, repeatable metrics for your RAG pipeline — to catch regressions and compare retrievers, chunking, or prompts.

Example

Python · evaluate RAG

When to use

Use Ragas to score RAG quality — does the answer stay faithful to the retrieved context, and is it relevant to the question?

How it flows

Provide questions, answers, and contexts; Ragas runs reference-free metrics (often LLM-judged) and returns per-metric scores.

Structure

flowchart LR
  qa[question / answer / context] --> r[Ragas]
  r --> judge[(LLM judge)]
  r --> scores[faithfulness / relevancy]

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
from datasets import Dataset

data = Dataset.from_dict({
    "question": ["What language is Qdrant written in?"],
    "answer": ["Qdrant is written in Rust."],
    "contexts": [["Qdrant is a vector database written in Rust."]],
})
# Metrics use an LLM judge (set OPENAI_API_KEY or configure another).
print(evaluate(data, metrics=[faithfulness, answer_relevancy]))

Example

Ragas — implementation sample

A real, runnable mini-project. Download it and run with Docker.

View samples/ragas_1 on GitHub ↗

June 28, 2026

Ragas — implementation sample

A tiny Ragas sample: it scores a RAG answer for faithfulness and response relevancy, judged by an OpenAI model + embeddings.

Configure

cd samples/ragas_1
cp .env.sample .env
# edit .env: set OPENAI_API_KEY (judge LLM + embeddings)

Run with Docker

cd samples/ragas_1
docker build -t aas-ragas .
docker run --rm --env-file .env aas-ragas

Run with Docker (in a devcontainer with DooD)

The foreground docker run may print nothing under Docker-outside-of-Docker — run detached and follow the logs:

docker logs -f "$(docker run -d --env-file .env aas-ragas)"

Run locally

cd samples/ragas_1
pip install -r requirements.txt
python app.py

Example run

Scores use an LLM judge, so they vary slightly run to run. One run with gpt-4o-mini:

{'faithfulness': 1.0000, 'answer_relevancy': 0.9701}

Files

.env.sample

# Copy to `.env` and add your OpenAI API key (judge LLM + embeddings).
#   cp .env.sample .env
OPENAI_API_KEY=

MODEL=gpt-4o-mini

app.py

"""Ragas: evaluate a RAG answer with reference-free metrics.

Scores one (question, answer, context) sample for faithfulness and response
relevancy, judged by an OpenAI model + embeddings (OPENAI_API_KEY).

Run:
    docker build -t aas-ragas .
    docker run --rm --env-file .env aas-ragas
"""

import os

from dotenv import load_dotenv
from ragas import EvaluationDataset, evaluate
from ragas.metrics import Faithfulness, ResponseRelevancy
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

load_dotenv()

MODEL = os.environ.get("MODEL", "gpt-4o-mini")


def main() -> None:
    dataset = EvaluationDataset.from_list(
        [
            {
                "user_input": "What language is Qdrant written in?",
                "response": "Qdrant is written in Rust.",
                "retrieved_contexts": ["Qdrant is a vector database written in Rust."],
            }
        ]
    )
    llm = LangchainLLMWrapper(ChatOpenAI(model=MODEL))
    embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

    result = evaluate(
        dataset,
        metrics=[Faithfulness(llm=llm), ResponseRelevancy(llm=llm, embeddings=embeddings)],
    )
    print(result)


if __name__ == "__main__":
    main()

Dockerfile

FROM python:3.13-slim

ENV PYTHONUNBUFFERED=1

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

ENTRYPOINT ["python", "app.py"]

requirements.txt

ragas>=0.2,<1
# Pin: ragas still imports langchain_community.chat_models.vertexai, which was
# removed in langchain-community 0.4. Stay on 0.3.x so the import resolves.
langchain-community>=0.3,<0.4
langchain-openai>=0.2,<1
python-dotenv>=1.2,<2