Fedor Chernogorskii, Sergei Averkiev, Liliya Kudraleeva, Zaven Martirosian, Maria Tikhonova, Valentin Malykh, Alena Fenogenova
| Challenge: | Existing methods for evaluating RAG systems are labor-intensive and difficult to maintain. |
| Approach: | They propose a method to design a RAG benchmark on a regularly updated corpus. |
| Outcome: | The proposed method uses a regularly updated corpus to evaluate RAG models. |
Similar Papers
DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimization (2026.findings-eacl)
Copied to clipboard
Haiyang Shen, Hang Yan, Zhongshi Xing, Mugeng Liu, Yue Li, Zhiyang Chen, Yuxiang Wang, Jiuzheng Wang, Yun Ma
| Challenge: | Existing retrieval-augmented generation paradigms rely heavily on public knowledge . Existing RAGs reliant on public information and often falter when faced with domain-specific queries. |
| Approach: | They propose a framework that combines a data-construction modeling approach with a scalable synthetic data-generation pipeline to optimize domain-specific retrieval performance. |
| Outcome: | The proposed framework optimizes domain-specific retrieval performance and bolsters retriever robustness. |
Generating Q&A Benchmarks for RAG Evaluation in Enterprise Settings (2025.acl-industry)
Copied to clipboard
| Challenge: | DataMorgana generates synthetic Q&A benchmarks tailored to RAG applications . lexical, syntactic, and semantic diversity of generated benchmarks exceeds existing tools . |
| Approach: | They propose a tool for generating synthetic Q&A benchmarks tailored to RAG applications in enterprise settings. |
| Outcome: | The proposed tool surpasses existing tools in terms of lexical, syntactic, and semantic diversity while maintaining high quality. |
Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems (2025.coling-industry)
Copied to clipboard
Rafael Teixeira de Lima, Shubham Gupta, Cesar Berrospi Ramis, Lokesh Mishra, Michele Dolfi, Peter Staar, Panagiotis Vagenas
| Challenge: | Retrieval Augmented Generation (RAG) systems are widespread in the industry. |
| Approach: | They propose to use Q&A datasets to assess retrieval performance and label-targeted data generation to refine RAG datasets. |
| Outcome: | The proposed system can generate Q&A datasets with fine-tuned small LLMs. |
RAGthoven: A Configurable Toolkit for RAG-enabled LLM Experimentation (2025.coling-demos)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have significantly altered the landscape of Natural Language Processing (NLP), but their use as a baseline method has not been extensive. |
| Approach: | They propose a tool for automatic evaluation of RAG-based pipelines that provides a simple yet powerful abstraction. |
| Outcome: | The proposed tool provides an automatic evaluation of RAG-based pipelines. |
RAG over Tables: Hierarchical Memory Index, Multi-Stage Retrieval, and Benchmarking (2026.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) integrates knowledge from tables with an external knowledge base to improve the answer relevance and accuracy. |
| Approach: | They propose a table-corpora-aware RAG framework called T-RAG to integrate external knowledge into Large Language Models (LLMs) they then develop a multi-table question answering benchmark called MultiTableQA which spans 3 different task types, 57,193 tables, and 23,758 questions in total. |
| Outcome: | The proposed framework achieves state-of-the-art accuracy, recall, and runtime performance, with improvements of up to 9.4%. |
Test-time Corpus Feedback: From Retrieval to RAG (2026.findings-eacl)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) pipelines treat retrieval and reasoning as isolated components, limiting performance on complex tasks. |
| Approach: | They propose to integrate large language models with retrieval to improve query quality . they also propose to use feedback to improve the query, retrieved context, or document pool . |
| Outcome: | The proposed methods bridge IR and NLP perspectives and highlight retrieval as a dynamic, learnable component of end-to-end RAG systems. |
RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework (2025.acl-long)
Copied to clipboard
Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing evaluation metrics for RAG systems are lacking due to high costs of data construction and lack of factual accuracy. |
| Approach: | They propose a framework to evaluate RAG systems in specialized scenarios . they propose three new metrics to evaluate LLM-generated responses . |
| Outcome: | The proposed framework outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. |
ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios (2026.acl-long)
Copied to clipboard
António Loison, Quentin Macé, Antoine Edy, Victor Xing, Tom Balough, Gabriel de Souza P. Moreira, Bo Liu, Manuel Faysse, Celine Hudelot, Gautier Viaud
| Challenge: | Existing benchmarks focus on textual data, single-document comprehension, or evaluating retrieval and generation in isolation. |
| Approach: | They propose a multimodal RAG benchmark featuring multi-type queries over visually rich document corpora. |
| Outcome: | The proposed benchmark outperforms existing benchmarks in visual retrieval and human-verified queries. |
RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation (2024.emnlp-demo)
Copied to clipboard
Xuanwang Zhang, Yun-Ze Song, Yidong Wang, Shuyun Tang, Xinfeng Li, Zhengran Zeng, Zhen Wu, Wei Ye, Wenyuan Xu, Yue Zhang, Xinyu Dai, Shikun Zhang, Qingsong Wen
| Challenge: | Existing research on Retrieval Augmented Generation (RAG) does not address the problem of hallucinations and real-time updating of knowledge. |
| Approach: | They propose a modular open-source library to equip LLMs with external knowledge. |
| Outcome: | The proposed approach reduces the need for expensive open-source tools and lacks fair comparisons between novel RAG algorithms. |
RAGViz: Diagnose and Visualize Retrieval-Augmented Generation (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) lack domain-specific knowledge and can cause hallucinations. |
| Approach: | They propose a RAG diagnosis tool that visualizes the attentiveness of the generated tokens in retrieved documents. |
| Outcome: | RAGViz provides token and document-level attention visualization and generation comparison upon context document addition and removal. |