DIRAS: Efficient LLM Annotation of Document Relevance for Retrieval Augmented Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | RAG systems leave out important relevant information (low recall) and excessively related but irrelevant information (high precision) authors propose a manual annotation-free schema that can be used for RAGs with limited performance. |
| Approach: | They propose a manual annotation-free schema that annotates unseen queries with calibrated relevance scores. |
| Outcome: | Evaluators show that DIRAS can achieve GPT-4-level performance on annotating and ranking unseen (query, document) pairs. |
Similar Papers
RAGAs: Automated Evaluation of Retrieval Augmented Generation (2024.eacl-demo)
Copied to clipboard
| Challenge: | RAGAs are a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines. |
| Approach: | They propose a framework for reference-free evaluation of Retrieval Augmented Generation pipelines. |
| Outcome: | RAGAs can be used to evaluate RAG pipelines without human annotations . the framework can be useful for faster evaluation cycles given the fast adoption of LLMs based on human annotation. |
RE-RAG: Improving Open-Domain QA Performance and Interpretability with Relevance Estimator in Retrieval-Augmented Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to retrieval augmented generation (RAG) are based on parametric knowledge and external knowledge. |
| Approach: | They propose a weakly supervised method for training a relevance estimator (RE) that provides relative relevance between contexts as previous rerankers did, and provides confidence, which can be used to classify whether given context is useful for answering the given question. |
| Outcome: | The proposed framework improves previously unreferenced large language models and can be trained with a small generator without labels for correct contexts. |
Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation (2025.emnlp-main)
Copied to clipboard
Hengran Zhang, Minghao Tang, Keping Bi, Jiafeng Guo, Shihao Liu, Daiting Shi, Dawei Yin, Xueqi Cheng
| Challenge: | Existing studies on large language models for document utility annotations have shown that they improve retrieval performance and RAG outcomes compared to models trained on human annotations. |
| Approach: | They propose a model that maximizes their summed marginal likelihood to annotate document utility on multiple positive samples per query. |
| Outcome: | The proposed model maximizes the marginal likelihood of multiple positive samples per query. |
RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation (2026.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) enhances large language models by integrating external knowledge retrieved at inference time. |
| Approach: | They evaluate RAG systems using MassiveDS, a large-scale datastore with mixture of knowledge. |
| Outcome: | The proposed approach improves performance on knowledge-intensive NLP tasks. |
REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to extend knowledge scope of large language models (LLMs) lack internal parametric knowledge, resulting in misusing external knowledge. |
| Approach: | They propose a retrieval-augmented approach that provides LLMs with potentially relevant documents through a module. |
| Outcome: | The proposed approach outperforms existing methods on four open-domain QA tasks. |
RAG-Studio: Towards In-Domain Adaptation of Retrieval Augmented Generation Through Self-Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing RAG systems that use pre-trained LLMs and retrievers often fail in specialized domains and applications. |
| Approach: | They propose a self-aligned training framework that adapts general RAG models to specific domains solely through synthetic data. |
| Outcome: | Experiments on specialized domain corpus, general LLM, and general retriever show that the self-aligned training framework outperforms human-annotated training data in specialized fields. |
Probing-RAG: Self-Probing to Guide Language Models in Selective Document Retrieval (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to optimize retrieve-and-generate processes for real-world scenarios may not be optimal for large language models. |
| Approach: | They propose a Probing-RAG which utilizes hidden state representations from the intermediate layers of language models to adaptively determine the necessity of additional retrievals for a given query. |
| Outcome: | The proposed method outperforms previous methods while reducing the number of redundant retrieval steps. |
MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation (2025.acl-long)
Copied to clipboard
Chia-Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan, Chin-Chia Michael Yeh, Guanchu Wang, Mingzhi Hu, Zhichao Xu, Yan Zheng, Mahashweta Das, Na Zou
| Challenge: | Existing RAG systems struggle with the quality of retrieval documents, causing performance degradation and reducing performance. |
| Approach: | They propose a training-free RAG framework that leverages multiple LLM agents to collaboratively filter and score retrieved documents. |
| Outcome: | The proposed framework outperforms existing RAG frameworks in QA benchmarks and shows superior answer consistency and answer accuracy over baseline methods. |
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. |
| Approach: | They propose a method that routes queries to RAG or LC based on model self-reflection. |
| Outcome: | The proposed method significantly reduces the computation cost while maintaining a comparable performance to RAG. |
Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systems (2025.coling-main)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) models address fairness concerns with respect to sensitive attributes such as gender, geographic location, and other demographic factors. |
| Approach: | They propose a framework to evaluate fairness in RAG using scenario-based questions and analyzing disparities across demographic attributes. |
| Outcome: | The proposed framework analyzes disparities across demographic attributes and identifies fairness issues in retrieval and generation stages. |