RAG-Zeval: Enhancing RAG Responses Evaluator through End-to-End Reasoning and Ranking-Based Reinforcement Learning

Challenge:	Existing evaluation frameworks rely on direct prompting of resource-intensive models with complex multi-stage prompts, introducing significant computational cost and underutilizing models’ reasoning capabilities.
Approach:	They propose a framework that trains evaluators with reinforcement learning to generate comprehensive and sound assessments with detailed explanation in one-pass.
Outcome:	The proposed framework outperforms baseline evaluation frameworks that rely on LLMs with 10-100 more parameters and achieves the strongest correlation with human judgments.

Similar Papers

Overview

Challenge

Approach

Outcome

RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework (2025.acl-long)

Copied to clipboard

Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu, Maosong Sun

Text Generation

Challenge:	Existing evaluation metrics for RAG systems are lacking due to high costs of data construction and lack of factual accuracy.
Approach:	They propose a framework to evaluate RAG systems in specialized scenarios . they propose three new metrics to evaluate LLM-generated responses .
Outcome:	The proposed framework outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.

Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systems (2025.coling-main)

Copied to clipboard

Xuyang Wu, Shuowei Li, Hsin-Tai Wu, Zhiqiang Tao, Yi Fang

Responsible & Trustworthy NLP

Challenge:	Retrieval-Augmented Generation (RAG) models address fairness concerns with respect to sensitive attributes such as gender, geographic location, and other demographic factors.
Approach:	They propose a framework to evaluate fairness in RAG using scenario-based questions and analyzing disparities across demographic attributes.
Outcome:	The proposed framework analyzes disparities across demographic attributes and identifies fairness issues in retrieval and generation stages.

Eval-RAR: Evaluation-Driven Retrieval-Augmented Reasoning via Reinforcement Learning (2026.findings-acl)

Copied to clipboard

Heng Yu, Rui Li, Qi Liu, Wenjun Feng, Junfeng Kang, Yi Zhan

Information Retrieval

Challenge:	Existing methods for retrieval-augmented generation fail to provide explicit supervision for internal reasoning process.
Approach:	They propose an Evaluation-driven Retrieval-Augmented Reasoning framework that uses reinforcement learning and a fine-grained evaluation reward to optimize the process.
Outcome:	Eval-RAR outperforms existing methods on QA benchmarks on seven single-hop and multi-hop tasks.

ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2024.naacl-long)

Copied to clipboard

Jon Saad-Falcon, Omar Khattab, Christopher Potts, Matei Zaharia

Information Retrieval

Challenge:	Evaluating retrieval-augmented generation systems relies on hand annotations for input queries, passages to retrieve, and responses to generate.
Approach:	They propose an automated evaluation framework for retrieval-augmented generation (RAG) ARES fine tunes lightweight LLM judges on synthetically generated queries and answers .
Outcome:	The proposed framework evaluates RAG systems using only human annotations . it can be used to improve system understanding and create targeted solutions .

DecEx-RAG: Boosting Agentic Retrieval-Augmented Generation with Decision and Execution Optimization via Process Supervision (2025.emnlp-industry)

Copied to clipboard

Yongqi Leng, Yikun Lei, Xikai Liu, Meizhi Zhong, Bojian Xiong, Yurong Zhang, Yan Gao, null Yiwu, Yao Hu, Deyi Xiong

Green & Sustainable NLP

Challenge:	Recent advances in outcome-supervised reinforcement learning (RL) have shown strong performance, but this approach still suffers from inefficient exploration, sparse reward signals, and ambiguous global reward feedback.
Approach:	They propose a model that models RAG as a Markov Decision Process (MDP) and introduces an efficient pruning strategy to optimize data expansion.
Outcome:	The proposed model outperforms existing methods and achieves an average performance improvement of 6.2% across six datasets.

Test-time Corpus Feedback: From Retrieval to RAG (2026.findings-eacl)

Copied to clipboard

Mandeep Rathee, Venktesh V, Sean MacAvaney, Avishek Anand

Information Retrieval

Challenge:	Retrieval-augmented generation (RAG) pipelines treat retrieval and reasoning as isolated components, limiting performance on complex tasks.
Approach:	They propose to integrate large language models with retrieval to improve query quality . they also propose to use feedback to improve the query, retrieved context, or document pool .
Outcome:	The proposed methods bridge IR and NLP perspectives and highlight retrieval as a dynamic, learnable component of end-to-end RAG systems.

Reflective RAG: Self-Evaluation Driven Strategy Optimization in Agentic Retrieval-Augmented Generation (2026.findings-acl)

Copied to clipboard

Haiyan Wu, Chenchen Wang, Chaoqun Sun, Chengxiong Lu, Yan-Hong Chen, Zhiqiang Zhang, Xiaoqing Feng

Information Retrieval

Challenge:	Recent agentic RAG systems lack the capacity to evaluate the utility of retrieved information, leading to brittle reasoning and suboptimal decision-making.
Approach:	They propose a framework that integrates self-evaluation to dynamically optimize retrieval and generation strategy.
Outcome:	The proposed framework outperforms strong agentic baselines on five knowledge-intensive QA benchmarks and improves training stability and generalization to multi-hop reasoning tasks.

RAG-Critic: Leveraging Automated Critic-Guided Agentic Workflow for Retrieval Augmented Generation (2025.acl-long)

Copied to clipboard

Guanting Dong, Jiajie Jin, Xiaoxi Li, Yutao Zhu, Zhicheng Dou, Ji-Rong Wen

Information Retrieval

Challenge:	Recent advances in large language models (LLMs) have demonstrated remarkable performance across a wide range of downstream tasks.
Approach:	They propose a framework that leverages a critic-guided agentic workflow to improve RAG capabilities autonomously.
Outcome:	The proposed framework improves RAG capabilities autonomously by leveraging a critic-guided agentic workflow.

RAGAs: Automated Evaluation of Retrieval Augmented Generation (2024.eacl-demo)

Copied to clipboard

Shahul Es, Jithin James, Luis Espinosa Anke, Steven Schockaert

Information Retrieval

Challenge:	RAGAs are a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines.
Approach:	They propose a framework for reference-free evaluation of Retrieval Augmented Generation pipelines.
Outcome:	RAGAs can be used to evaluate RAG pipelines without human annotations . the framework can be useful for faster evaluation cycles given the fast adoption of LLMs based on human annotation.

Open-RAG: Enhanced Retrieval Augmented Reasoning with Open-Source Large Language Models (2024.findings-emnlp)

Copied to clipboard

Shayekh Islam, Md Asib Rahman, K S M Tozammel Hossain, Enamul Hoque, Shafiq Joty, Md Rizwan Parvez

Semantic Text Processing

Challenge:	Existing methods to integrate Large Language Models with external knowledge suffer from limited reasoning capabilities, especially when using open-source LLMs.
Approach:	They propose a framework that transforms an arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE) model capable of handling complex reasoning tasks.
Outcome:	The proposed framework transforms an arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE) model capable of handling complex reasoning tasks, including both single- and multi-hop queries.