RAG Benchmarking
Version v1.0.2Plug in any RAG system (LangChain, LlamaIndex, or custom) and benchmark it against classic and agentic-era metrics: faithfulness, answer relevancy, retrieval precision, and four agentic metrics for multi-step agents. The Python SDK sends samples to the evaluation server (`docker compose up`, port 5001), and the LLM-judge metrics need a judge model key such as GEMINI_API_KEY or OPENAI_API_KEY. Retrieval metrics run in process with no key.
Install
bashpip install rag-benchmarkingNeeds Python 3.11 or newer.
Apache 2.0 · zero telemetry · source Regulation (EU) 2024/1689 (Article 15).
Why RAG Benchmarking exists
Accuracy, robustness and cybersecurity
Article 15 applies from 2 Dec 2027 for high-risk AI systems under Annex III, and from 2 Aug 2028 for those covered by Annex I. The original dates, 2 August 2026 and 2 August 2027, were moved by the Digital Omnibus, Regulation (EU) 2026/1744, published in the Official Journal on 24 July 2026 and in force since 27 July 2026. Providers must declare accuracy metrics in the instructions for use and demonstrate consistent performance across the lifecycle. Non-compliance via the Article 16 provider-obligation chain is sanctionable up to €15M or 3% of global annual turnover under Article 99(4). For RAG-based high-risk systems, "an appropriate level of accuracy" (Art. 15(1)) is not a self-asserted figure. It is a metric declared on the label and defensible against post-market evidence.
Other AiExponent tools