Documentation menu
On this page
Article 15v1.0.2pip install rag-benchmarking

RAG Benchmarking

Plug in any RAG system (LangChain, LlamaIndex, or custom) and benchmark it against classic and agentic-era metrics: faithfulness, answer relevancy, retrieval precision, and four agentic metrics for multi-step agents. The Python SDK sends samples to the evaluation server (`docker compose up`, port 5001), and the LLM-judge metrics need a judge model key such as GEMINI_API_KEY or OPENAI_API_KEY. Retrieval metrics run in process with no key.

View on GitHubPyPI ↗

Quick Start

bashpip install rag-benchmarking

Needs Python 3.11 or newer.

pythonfrom rag_benchmarking.harness.runner import EvaluationRunner
from rag_benchmarking.harness.schemas import EvalSample, RunConfig

# Retrieval metrics compare the document IDs your retriever returned
# with the ones that are relevant. No server, no judge model, no API key.
samples = [
    EvalSample(
        sample_id="q1",
        question="What is RAG?",
        contexts=["RAG stands for Retrieval-Augmented Generation."],
        answer="RAG combines retrieval with LLM generation.",
        retrieved_doc_ids=["doc-1", "doc-7", "doc-3"],
        relevant_doc_ids=["doc-1"],
    ),
]

report = EvaluationRunner(RunConfig(metric_group="retrieval", k=3)).evaluate(samples)
print(report.metrics)
# precision_at_k, recall_at_k, mrr, ndcg_at_k

# The LLM-judge metrics (faithfulness, answer relevancy) need the evaluation
# server and a judge key, and fail to import on a fresh 1.0.2 install
# (github.com/aiexponent/rag-benchmarking/issues/35).

Features

  • Framework-agnostic: works with LangChain, LlamaIndex, or any custom RAG system
  • Classic metrics: faithfulness, answer relevancy, context precision/recall
  • Retrieval metrics: Precision@K, Recall@K, MRR, NDCG
  • Agentic metrics: agent faithfulness, tool call accuracy, source attribution, retrieval necessity
  • REST API + Python SDK with LangChain and LlamaIndex adapters
  • Run history with comparison across configurations

Regulatory Foundation

Article 15Accuracy, robustness and cybersecurityUpcoming · 2 Dec 2027 · deferred

Read the full pillar: EU AI Act Article 15 explainer →

What the regulation requires

1. High-risk AI systems shall be designed and developed in such a way that they achieve an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently in those respects throughout their lifecycle. 3. The levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use. 4. High-risk AI systems shall be as resilient as possible regarding errors, faults or inconsistencies that may occur within the system or the environment in which the system operates, in particular due to their interaction with natural persons or other systems. Technical and organisational measures shall be taken in this regard. The robustness of high-risk AI systems may be achieved through technical redundancy solutions, which may include backup or fail-safe plans.
15(1)15(3)15(4)

What you face if you don't comply

Article 15 applies from 2 Dec 2027 for high-risk AI systems under Annex III, and from 2 Aug 2028 for those covered by Annex I. The original dates, 2 August 2026 and 2 August 2027, were moved by the Digital Omnibus, Regulation (EU) 2026/1744, published in the Official Journal on 24 July 2026 and in force since 27 July 2026. Providers must declare accuracy metrics in the instructions for use and demonstrate consistent performance across the lifecycle. Non-compliance via the Article 16 provider-obligation chain is sanctionable up to €15M or 3% of global annual turnover under Article 99(4). For RAG-based high-risk systems, "an appropriate level of accuracy" (Art. 15(1)) is not a self-asserted figure. It is a metric declared on the label and defensible against post-market evidence.

Up to €15M or 3% of global annual turnover, whichever is higher
Article 99(4) · Penalties

For SMEs, including start-ups, the fine is capped at whichever of the two is lower (Art. 99(6)). Since 27 Jul 2026, the same lower cap applies to small mid-cap enterprises (Art. 99(6a)).

How RAG Benchmarking addresses this

  • 15(1)Reproducible accuracy benchmarks for RAG pipelines (retrieval recall, answer faithfulness, citation precision) with versioned eval sets
  • 15(3)Generates the accuracy-metrics block for the Article 13 instructions for use, with confidence intervals and eval-set provenance
  • 15(4)Robustness suite: input perturbations, noisy-context, adversarial-passage, and OOD query stress tests with pass/fail thresholds
  • 15(4)Lifecycle drift monitoring: replays the declared eval set against the live system on a schedule and alerts on metric regression

Source: eur-lex.europa.eu/…/CELEX:32024R1689 · Retrieved

Frequently asked questions

Direct answers to common questions about RAG Benchmarking and Article 15. Regulatory citations reference EUR-Lex CELEX:32024R1689.

What does EU AI Act Article 15 require?
High-risk AI systems must achieve appropriate accuracy, robustness, and cybersecurity throughout their lifecycle. Accuracy metrics must be declared in the instructions for use (Article 15(3)), and the system must be resilient to errors, faults, and inconsistencies (Article 15(4)). Source: Regulation (EU) 2024/1689 Article 15(1), 15(3), 15(4).
When does Article 15 become enforceable?
Article 15 obligations apply from 2 Dec 2027 for high-risk AI systems listed in Annex III, and from 2 Aug 2028 for high-risk AI systems covered by Annex I. The Digital Omnibus on AI, Regulation (EU) 2026/1744, moved these dates from 2 Aug 2026 and 2 Aug 2027. It was published in the Official Journal on 24 Jul 2026 and has been in force since 27 Jul 2026. Source: Regulation (EU) 2024/1689 Article 113, third paragraph, point (c), as amended by Regulation (EU) 2026/1744.
Does RAG Benchmarking cover the cybersecurity leg of Article 15?
No. RAG Benchmarking covers the accuracy and robustness legs: faithfulness, retrieval precision, agentic metrics, and adversarial-passage stress tests. The cybersecurity leg (prompt injection resistance, jailbreak defence, model integrity) needs a runtime AI security control. Pair the two to cover both legs of Article 15.
What metrics does RAG Benchmarking measure?
Classic metrics (faithfulness, answer relevancy, context precision/recall), retrieval metrics (Precision@K, Recall@K, MRR, NDCG), and four agentic metrics (agent faithfulness, tool-call accuracy, source attribution, retrieval necessity).
Is RAG Benchmarking framework-agnostic?
Yes. RAG Benchmarking works with LangChain, LlamaIndex, or any custom RAG system that returns a sample with `question`, `contexts`, and `answer` fields. SDK adapters for LangChain and LlamaIndex are included; custom integrations use the JSONL schema directly.
What does a RAG Benchmarking run need?
The SDK sends samples to the evaluation server, so start it first with `docker compose up`; the API listens on port 5001. The LLM-judge metrics, such as faithfulness and answer relevancy, also need a judge model key on the server: `LLM_PROVIDER=gemini` with `GEMINI_API_KEY`, or `OPENAI_API_KEY`. Judge calls carry the provider's usage cost. Retrieval metrics (Precision@K, Recall@K, MRR, NDCG) run in process with no server and no key. On a fresh install of v1.0.2 the LLM-judge metrics fail to import (aiexponent/rag-benchmarking#35).
Can I bring my own evaluation dataset?
Yes. RAG Benchmarking accepts custom datasets in JSONL format with the expected schema. The bundled golden dataset is English-only; multilingual evaluation is not supported in v1.0.
Is RAG Benchmarking free?
Yes. Apache 2.0 licensed, with no telemetry. You run the harness and its evaluation server yourself. The LLM-judge metrics send the samples you evaluate to the judge model provider you configure, which has its own usage cost; retrieval metrics need no network.
What is the penalty for Article 15 non-compliance?
Up to €15M or 3% of global annual turnover, whichever is higher, under Article 99(4). The provider-obligation chain via Article 16 routes Article 15 failures through this penalty band. For SMEs, including start-ups, the fine is capped at whichever of the two is lower (Art. 99(6)). Since 27 Jul 2026, the same lower cap applies to small mid-cap enterprises (Art. 99(6a)).
How does drift monitoring work?
You declare an evaluation set version and a metric threshold. RAG Benchmarking replays the eval set against the live system on a schedule and alerts on metric regression, supporting the lifecycle-consistent-performance requirement of Article 15(1).

Known Limitations

  • Benchmark datasets are English-only; no multilingual evaluation support.
  • Custom dataset integration requires manual formatting to the expected JSONL schema.
  • Accuracy metrics only; latency and throughput are not measured.
  • LLM-as-judge metrics depend on the configured judge model quality.
  • Rate limiting is in-memory and resets on server restart.
  • On a fresh install of v1.0.2 the LLM-judge metrics fail to import (aiexponent/rag-benchmarking#35); retrieval metrics are not affected.

For the most current status, see GitHub issues.

Contributing

Contributions are welcome; everything is Apache 2.0 licensed. See the contributing guide and open issues.

License

Licensed under the Apache License 2.0. Not legal advice. Not a notified body.

AI Exponent LLC, a USA-registered company, also operates a separate advisory brand. The tool above is product, not advisory. How the two are kept apart.

Evidence flow

One tool covers one article. The full set covers your audit.

Each AiExponent tool emits a named artefact the next tool reads as input. Browse the full toolchain, from Article 5 screening through Article 72 post-market monitoring.

See all tools →

Content verified 2026-10-25