| Challenge: | RAGAs are a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines. |
| Approach: | They propose a framework for reference-free evaluation of Retrieval Augmented Generation pipelines. |
| Outcome: | RAGAs can be used to evaluate RAG pipelines without human annotations . the framework can be useful for faster evaluation cycles given the fast adoption of LLMs based on human annotation. |
Similar Papers
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2024.naacl-long)
Copied to clipboard
| Challenge: | Evaluating retrieval-augmented generation systems relies on hand annotations for input queries, passages to retrieve, and responses to generate. |
| Approach: | They propose an automated evaluation framework for retrieval-augmented generation (RAG) ARES fine tunes lightweight LLM judges on synthetically generated queries and answers . |
| Outcome: | The proposed framework evaluates RAG systems using only human annotations . it can be used to improve system understanding and create targeted solutions . |
RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generation (2026.eacl-demo)
Copied to clipboard
| Challenge: | Existing tools for evaluating RAG systems often collapse heterogeneous behaviors into single scores. |
| Approach: | They propose a diagnostic framework for automated, reference-free evaluation of RAG pipelines. |
| Outcome: | The proposed framework decomposes RAG behavior into retrieval quality,answer relevance and completeness, strictclaim-level faithfulness, and judge calibration. |
Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems (2026.eacl-long)
Copied to clipboard
Kin Kwan Leung, Mouloud Belbahri, Yi Sui, Alex Labach, Xueying Zhang, Stephen Anthony Rose, Jesse C. Cresswell
| Challenge: | Existing work on RAG errors has not accounted for the complexity of real-world RAG systems and their failure modes. |
| Approach: | They propose a taxonomy of error types that can occur in realistic RAG systems and an auto-evaluation method that can be used to track errors during development. |
| Outcome: | The proposed method can be used in practice to track and address errors during development. |
RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework (2025.acl-long)
Copied to clipboard
Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing evaluation metrics for RAG systems are lacking due to high costs of data construction and lack of factual accuracy. |
| Approach: | They propose a framework to evaluate RAG systems in specialized scenarios . they propose three new metrics to evaluate LLM-generated responses . |
| Outcome: | The proposed framework outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. |
DIRAS: Efficient LLM Annotation of Document Relevance for Retrieval Augmented Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | RAG systems leave out important relevant information (low recall) and excessively related but irrelevant information (high precision) authors propose a manual annotation-free schema that can be used for RAGs with limited performance. |
| Approach: | They propose a manual annotation-free schema that annotates unseen queries with calibrated relevance scores. |
| Outcome: | Evaluators show that DIRAS can achieve GPT-4-level performance on annotating and ranking unseen (query, document) pairs. |
Controlled Retrieval-augmented Context Evaluation for Long-form RAG (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) enhances large language models by incorporating context retrieved from external knowledge sources. |
| Approach: | They propose a Controlled Retrieval-aUgmented conteXt evaluation framework to directly assess retrieval-augmented contexts. |
| Outcome: | The proposed framework uses human-written summaries to control the information scope of knowledge. |
Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systems (2025.coling-main)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) models address fairness concerns with respect to sensitive attributes such as gender, geographic location, and other demographic factors. |
| Approach: | They propose a framework to evaluate fairness in RAG using scenario-based questions and analyzing disparities across demographic attributes. |
| Outcome: | The proposed framework analyzes disparities across demographic attributes and identifies fairness issues in retrieval and generation stages. |
RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation (2024.emnlp-demo)
Copied to clipboard
Xuanwang Zhang, Yun-Ze Song, Yidong Wang, Shuyun Tang, Xinfeng Li, Zhengran Zeng, Zhen Wu, Wei Ye, Wenyuan Xu, Yue Zhang, Xinyu Dai, Shikun Zhang, Qingsong Wen
| Challenge: | Existing research on Retrieval Augmented Generation (RAG) does not address the problem of hallucinations and real-time updating of knowledge. |
| Approach: | They propose a modular open-source library to equip LLMs with external knowledge. |
| Outcome: | The proposed approach reduces the need for expensive open-source tools and lacks fair comparisons between novel RAG algorithms. |
Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems (2025.coling-industry)
Copied to clipboard
Rafael Teixeira de Lima, Shubham Gupta, Cesar Berrospi Ramis, Lokesh Mishra, Michele Dolfi, Peter Staar, Panagiotis Vagenas
| Challenge: | Retrieval Augmented Generation (RAG) systems are widespread in the industry. |
| Approach: | They propose to use Q&A datasets to assess retrieval performance and label-targeted data generation to refine RAG datasets. |
| Outcome: | The proposed system can generate Q&A datasets with fine-tuned small LLMs. |
RAG+: Enhancing Retrieval-Augmented Generation with Application-Aware Reasoning (2025.emnlp-main)
Copied to clipboard
Yu Wang, Shiwan Zhao, Zhihu Wang, Ming Fan, Xicheng Zhang, Yubo Zhang, Zhengfan Wang, Heyuan Huang, Ting Liu
| Challenge: | Existing RAG paradigms often overlook the cognitive step of applying knowledge, leaving a gap between retrieved facts and task-specific reasoning. |
| Approach: | They introduce a module extension that integrates application-aware reasoning into the RAG pipeline. |
| Outcome: | Experiments show that RAG+ outperforms standard RAG variants and achieves gains of 3–5% in complex scenarios. |