Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, Fei Huang
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of natural language processing tasks. |
| Approach: | They propose a benchmark to assess the ability of LLMs to perform web traversal by using an explore-critic paradigm. |
| Outcome: | The proposed framework mimics human-like web navigation through an explore-critic paradigm and demonstrates the effectiveness of RAG combined with WebWalker in real-world scenarios. |
Similar Papers
Towards Reliable Agents: Benchmarking Customized LLM-Based Retrieval-Augmented Generation Frameworks with Deployment Validation (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing benchmarks for general-purpose RAG systems, such as CRAG, RGB, MultiHop-RAG, and CRUD-RAGG, are limited and lack a benchmark specifically tailored to evaluate frameworks. |
| Approach: | They evaluated OpenAI’s Assistants API versus a RAG assistant built with Langchain and deployed a system based on benchmark insights as a course assistant over a two-year span. |
| Outcome: | The proposed benchmarks show that domain-specific retrieval impacts response accuracy and highlight key challenges in real-world deployment. |
CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmented Generation (2025.findings-naacl)
Copied to clipboard
Yiruo Cheng, Kelong Mao, Ziliang Zhao, Guanting Dong, Hongjin Qian, Yongkang Wu, Tetsuya Sakai, Ji-Rong Wen, Zhicheng Dou
| Challenge: | Existing research focuses on single-turn RAG, leaving a gap in addressing multi-turn conversations . a new benchmark is designed to assess RAG systems in realistic multi-turned conversations based on Wikipedia . |
| Approach: | They propose a large-scale benchmark to assess RAG systems in multi-turn contexts . CORAL includes diverse information-seeking conversations automatically derived from Wikipedia . authors propose unified framework to standardize various conversational RAG methods . |
| Outcome: | The proposed framework supports three core tasks of conversational RAG: passage retrieval, response generation, and citation labeling. |
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2025.naacl-long)
Copied to clipboard
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, Manaal Faruqui
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly enhanced their capabilities across various cognitive tasks. |
| Approach: | They propose a high-quality evaluation dataset to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. |
| Outcome: | The proposed framework improves performance in end-to-end RAG scenarios. |
MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) systems are limited in their evaluation due to the intricate interplay between retrieval and generation components. |
| Approach: | They propose a Question Answering Question Answerer dataset specifically designed for RAG evaluation that integrates external, non-parametric knowledge retrieved by a retrieval pool of 37,800 entries. |
| Outcome: | The proposed dataset consists of 7,560 curated instances mapped to a retrieval pool of 37,800 entries, enabling an efficient evaluation of both retrieval and generation tasks. |
RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation (2026.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) enhances large language models by integrating external knowledge retrieved at inference time. |
| Approach: | They evaluate RAG systems using MassiveDS, a large-scale datastore with mixture of knowledge. |
| Outcome: | The proposed approach improves performance on knowledge-intensive NLP tasks. |
Is Agentic RAG worth it? An experimental comparison of RAG approaches (2026.acl-industry)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) systems have several limitations, including noisy or suboptimal retrieval, misuse of retrieval for out-of-scope queries, weak query–document matching, and variability or cost associated with the generator. |
| Approach: | They propose to use a "Enhanced" RAG to address weaknesses in the workflow . they propose to orchestrate the entire process, deciding which actions to perform, when to perform them, and whether to iterate . |
| Outcome: | The proposed models address shortcomings in the RAG workflow, and provide practical insights into the trade-offs between them. |
LLMs Know What They Need: Leveraging a Missing Information Guided Framework to Empower Retrieval-Augmented Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Existing solutions to improve the accuracy of RAG are based on retrieval-augmented generation . however, RAG still faces several challenges in tackling complex multi-hop queries . |
| Approach: | They propose a Missing Information Guided Retrieve-Extraction-Solving paradigm that leverages the identification of missing information to generate a targeted query. |
| Outcome: | The proposed method can extract information from retrieved knowledge and know what is still missing. |
AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning (2026.findings-acl)
Copied to clipboard
Jingbo Sun, Wenyue Chong, Songjun Tu, Qichao Zhang, Yaocheng Zhang, Jiajun Chai, Xiaohan Wang, Wei Lin, Guojun Yin, Dongbin Zhao
| Challenge: | Prior work limits search depth to reduce cost, but this often leads to underexploration of complex questions. |
| Approach: | They propose a reinforcement learning framework that evaluates each search step via self-generated intermediate answers. |
| Outcome: | Extensive experiments on multiple benchmarks show that AutoSearch achieves a superior accuracy-efficiency trade-off, alleviating over-searching while preserving search quality. |
Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systems (2025.coling-main)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) models address fairness concerns with respect to sensitive attributes such as gender, geographic location, and other demographic factors. |
| Approach: | They propose a framework to evaluate fairness in RAG using scenario-based questions and analyzing disparities across demographic attributes. |
| Outcome: | The proposed framework analyzes disparities across demographic attributes and identifies fairness issues in retrieval and generation stages. |
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. |
| Approach: | They propose a method that routes queries to RAG or LC based on model self-reflection. |
| Outcome: | The proposed method significantly reduces the computation cost while maintaining a comparable performance to RAG. |