RAG Benchmarking

Quick Start

bashpip install rag-benchmarking

pythonfrom app.sdk.client import RagEval

client = RagEval(api_url="http://localhost:5001", api_key="your-key")

# Works with LangChain
result = my_chain.invoke({"query": "What is RAG?"})
sample = RagEval.from_langchain(result)

# Or any dict with question / contexts / answer
sample = {
    "question": "What is RAG?",
    "contexts": ["RAG stands for Retrieval-Augmented Generation."],
    "answer": "RAG combines retrieval with LLM generation.",
}

report = client.evaluate([sample], metrics=["faithfulness", "answer_relevancy"])
print(report["metrics"])
# {"faithfulness": 0.95, "answer_relevancy": 0.81}

Benchmark Results

Measured on the 50-sample golden dataset using gemini-2.5-flash as judge at temperature=0.0.

Features

• Framework-agnostic — works with LangChain, LlamaIndex, or any custom RAG system
• Classic metrics: faithfulness, answer relevancy, context precision/recall
• Retrieval metrics: Precision@K, Recall@K, MRR, NDCG
• Agentic metrics: agent faithfulness, tool call accuracy, source attribution, retrieval necessity
• REST API + Python SDK with LangChain and LlamaIndex adapters
• Run history with comparison across configurations

Regulatory Foundation

Frequently asked questions

Direct answers to common questions about RAG Benchmarking and Article 15. Regulatory citations reference EUR-Lex CELEX:32024R1689.

What does EU AI Act Article 15 require?

High-risk AI systems must achieve appropriate accuracy, robustness, and cybersecurity throughout their lifecycle. Accuracy metrics must be declared in the instructions for use (Article 15(3)), and the system must be resilient to errors, faults, and inconsistencies (Article 15(4)). Source: Regulation (EU) 2024/1689 Article 15(1), 15(3), 15(4).

When does Article 15 become enforceable?

Article 15 obligations for high-risk AI systems become enforceable on 2 August 2026, per Article 113. Source: Regulation (EU) 2024/1689 Article 113.

Does RAG-Bench cover the cybersecurity leg of Article 15?

No. RAG-Bench covers accuracy and robustness — measuring faithfulness, retrieval precision, agentic metrics, and adversarial-passage robustness. The cybersecurity leg (prompt injection resistance, jailbreak defence, model integrity) requires a runtime AI security control such as AgentShield. Pair the two for full Article 15 coverage.

What metrics does RAG-Bench measure?

Classic metrics (faithfulness, answer relevancy, context precision/recall), retrieval metrics (Precision@K, Recall@K, MRR, NDCG), and four agentic metrics (agent faithfulness, tool-call accuracy, source attribution, retrieval necessity).

Is RAG-Bench framework-agnostic?

Yes. RAG-Bench works with LangChain, LlamaIndex, or any custom RAG system that returns a sample with `question`, `contexts`, and `answer` fields. SDK adapters for LangChain and LlamaIndex are included; custom integrations use the JSONL schema directly.

What is the measured faithfulness on the golden dataset?

0.958 on the published 50-sample golden dataset (rated "Excellent"), with 0.810 answer relevancy ("Good"). These are the actual numbers from the v1.0.0-rc1 release benchmark — not aspirational targets.

Can I bring my own evaluation dataset?

Yes. RAG-Bench accepts custom datasets in JSONL format with the expected schema. The bundled golden dataset is English-only; multilingual evaluation is not supported in v1.0.

Is RAG-Bench free?

Yes. Apache 2.0 licensed. The harness itself runs locally; LLM-as-judge metrics depend on whichever judge model you configure (which may have its own usage cost).

What is the penalty for Article 15 non-compliance?

Up to €15M or 3% of global annual turnover, whichever is higher, under Article 99(4). The provider-obligation chain via Article 16 routes Article 15 failures through this penalty band.

How does drift monitoring work?

You declare an evaluation set version and a metric threshold. RAG-Bench replays the eval set against the live system on a schedule and alerts on metric regression — supporting the lifecycle-consistent-performance requirement of Article 15(1).

Known Limitations

• Benchmark datasets are English-only; no multilingual evaluation support.
• Custom dataset integration requires manual formatting to the expected JSONL schema.
• Accuracy metrics only — latency and throughput are not measured.
• LLM-as-judge metrics depend on the configured judge model quality.
• Rate limiting is in-memory and resets on server restart.

For the most current status, see GitHub issues.

Contributing

Contributions are welcome — Apache 2.0 licensed. See the contributing guide and open issues.

License

Licensed under the Apache License 2.0. Not legal advice. Not a notified body.