MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems (2025.naacl-long)
Copied to clipboard
| Challenge: | Traditional retrieval-augmented generation benchmarks use heuristics as the ground truth for evaluation, but require an expensive large language model (LLM) as a judge for a reliable evaluation. |
| Approach: | They propose to use large language models as a judge for retrieval-augmented generation benchmarks . they use heuristic metrics as input and a large language model as heuriistic input . |
| Outcome: | The proposed method couples heuristic features with large language models as judge for evaluation. |
Similar Papers
MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) systems are limited in their evaluation due to the intricate interplay between retrieval and generation components. |
| Approach: | They propose a Question Answering Question Answerer dataset specifically designed for RAG evaluation that integrates external, non-parametric knowledge retrieved by a retrieval pool of 37,800 entries. |
| Outcome: | The proposed dataset consists of 7,560 curated instances mapped to a retrieval pool of 37,800 entries, enabling an efficient evaluation of both retrieval and generation tasks. |
MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation (2025.acl-long)
Copied to clipboard
María Andrea Cruz Blandón, Jayasimha Talur, Bruno Charron, Dong Liu, Saab Mansour, Marcello Federico
| Challenge: | Existing benchmarks focus on English or use translated data, which fails to capture cultural nuances. |
| Approach: | They propose to use a multilingual end-to-end Meta-Evaluation RAG benchmark MEMERAG to assess accuracy and faithfulness of RAG systems. |
| Outcome: | The proposed benchmark can identify improvements offered by advanced prompting techniques and LLMs. |
Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models (2025.findings-acl)
Copied to clipboard
Shuliang Liu, Xinze Li, Zhenghao Liu, Yukun Yan, Cheng Yang, Zheni Zeng, Zhiyuan Liu, Maosong Sun, Ge Yu
| Challenge: | Existing evaluation metrics cannot fairly evaluate the outputs of RAG models during training and evaluation. |
| Approach: | They propose a method which prompts LLMs to generate different judgments based on various combinations of judgment dimensions and utilizes the judge-consistency to evaluate these judgments. |
| Outcome: | The proposed method generates more accurate evaluations for RAG models across different RAG model and datasets. |
Benchmarking Retrieval-Augmented Generation for Medicine (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have state-of-the-art performance on a wide range of medical question answering tasks, but they still face challenges with hallucinations and outdated knowledge. |
| Approach: | They propose a benchmark to evaluate medical RAG systems using large-scale experiments with over 1.8 trillion prompt tokens. |
| Outcome: | The proposed benchmark improves accuracy of six different LLMs by up to 18% over chain-of-thought prompting. |
BERGEN: A Benchmarking Library for Retrieval-Augmented Generation (2024.findings-emnlp)
Copied to clipboard
David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Stéphane Clinchant, Vassilina Nikoulina
| Challenge: | Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. |
| Approach: | They propose a library that allows to benchmark and standardize RAG experiments. |
| Outcome: | The proposed library is an end-to-end library for reproducible research standardizing RAG experiments. |
RAD-Bench: Evaluating Large Language Models’ Capabilities in Retrieval Augmented Dialogues (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing benchmarks assess LLMs' chat abilities in multi-turn dialogues or their use of retrieval for augmented responses in limited tasks such as knowledge QA or numeric reasoning. |
| Approach: | They propose a benchmark to evaluate LLMs' capabilities in multi-turn dialogues following retrievals. |
| Outcome: | The proposed benchmark evaluates LLMs' ability to perform in multi-turn dialogues following retrievals over 6 representative scenarios. |
CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmented Generation (2025.findings-naacl)
Copied to clipboard
Yiruo Cheng, Kelong Mao, Ziliang Zhao, Guanting Dong, Hongjin Qian, Yongkang Wu, Tetsuya Sakai, Ji-Rong Wen, Zhicheng Dou
| Challenge: | Existing research focuses on single-turn RAG, leaving a gap in addressing multi-turn conversations . a new benchmark is designed to assess RAG systems in realistic multi-turned conversations based on Wikipedia . |
| Approach: | They propose a large-scale benchmark to assess RAG systems in multi-turn contexts . CORAL includes diverse information-seeking conversations automatically derived from Wikipedia . authors propose unified framework to standardize various conversational RAG methods . |
| Outcome: | The proposed framework supports three core tasks of conversational RAG: passage retrieval, response generation, and citation labeling. |
RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation (2026.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) enhances large language models by integrating external knowledge retrieved at inference time. |
| Approach: | They evaluate RAG systems using MassiveDS, a large-scale datastore with mixture of knowledge. |
| Outcome: | The proposed approach improves performance on knowledge-intensive NLP tasks. |
RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation (2024.emnlp-demo)
Copied to clipboard
Xuanwang Zhang, Yun-Ze Song, Yidong Wang, Shuyun Tang, Xinfeng Li, Zhengran Zeng, Zhen Wu, Wei Ye, Wenyuan Xu, Yue Zhang, Xinyu Dai, Shikun Zhang, Qingsong Wen
| Challenge: | Existing research on Retrieval Augmented Generation (RAG) does not address the problem of hallucinations and real-time updating of knowledge. |
| Approach: | They propose a modular open-source library to equip LLMs with external knowledge. |
| Outcome: | The proposed approach reduces the need for expensive open-source tools and lacks fair comparisons between novel RAG algorithms. |
RAG over Tables: Hierarchical Memory Index, Multi-Stage Retrieval, and Benchmarking (2026.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) integrates knowledge from tables with an external knowledge base to improve the answer relevance and accuracy. |
| Approach: | They propose a table-corpora-aware RAG framework called T-RAG to integrate external knowledge into Large Language Models (LLMs) they then develop a multi-table question answering benchmark called MultiTableQA which spans 3 different task types, 57,193 tables, and 23,758 questions in total. |
| Outcome: | The proposed framework achieves state-of-the-art accuracy, recall, and runtime performance, with improvements of up to 9.4%. |