Studying the Role of Input-Neighbor Overlap in Retrieval-Augmented Language Models Training Efficiency (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that retrieval-augmented language models can perform better than larger models, but the optimal degree of overlap remains unclear. |
| Approach: | They investigate how varying levels of query–context overlap affect model performance . they specifically generate synthetic context through paraphrasing queries . |
| Outcome: | The proposed model can improve test-time perplexity and training time by 40% without compromising performance. |
Similar Papers
More room for language: Investigating the effect of retrieval on language models (2024.naacl-short)
Copied to clipboard
| Challenge: | Retrieval-augmented language models are a promising alternative to standard pretraining, but little attention has been put into understanding what this type of training scheme does to the underlying language model when analyzed as a standalone -separated from the overall retrieval pipeline. |
| Approach: | They propose an ‘ideal retrieval’ methodology to study these models in a fully controllable setting and propose a retrieval augmentation methodology to examine their effects. |
| Outcome: | The proposed model saves substantially less world knowledge in their weights, but is worse at comprehending global context. |
Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Augmenting language models with a retrieval mechanism has been shown to improve performance while keeping the number of parameters low. |
| Approach: | They propose to augment language models with a retrieval mechanism by replacing semantic retrieval with BM25 . they find that the model's performance is better explained by surface-level similarities, they say . |
| Outcome: | The proposed method reduces perplexity and lowers the number of parameters while keeping the number low. |
Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that the quantity and quality of context affect retrieval-augmented generation models during training. |
| Approach: | They propose a method to mitigate overfitting to specific context quality by introducing bias to the cross-attention distribution. |
| Outcome: | The proposed method improves retrieval-augmented generation models on different context quality. |
In-Context Retrieval-Augmented Language Models (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing RALM methods focus on modifying the LM architecture to facilitate incorporation of external information, complicating deployment. |
| Approach: | They propose to condition a language model on relevant documents from a grounding corpus during generation by conditioning on external knowledge sources. |
| Outcome: | The proposed method significantly improves language modeling performance and provides natural source attribution mechanism. |
Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts (2024.findings-acl)
Copied to clipboard
| Challenge: | Retrieval Augmented Generation can be used to process long contexts in Open-Domain Question-Answering tasks. |
| Approach: | They propose a method to cover longer contexts in Open-Domain Question-Answering tasks by using a small encoder language model and cross-attention with origin inputs. |
| Outcome: | The proposed method can cover longer contexts while keeping the computing requirements close to the baseline. |
Tagging-Augmented Generation: Assisting Language Models in Finding Intricate Knowledge In Long Contexts (2025.emnlp-industry)
Copied to clipboard
Anwesan Pal, Karen Hovsepian, Tinghao Guo, Mengnan Zhao, Somendra Tripathi, Nikos Kanakaris, George Mihaila, Sumit Nigam
| Challenge: | Recent studies into effective context lengths of flagship large language models (LLMs) have revealed major limitations in effective question answering (QA) and reasoning over long and complex contexts for even the largest and most impressive cadre of models. |
| Approach: | They propose a lightweight data augmentation strategy that boosts LLM performance in long-context scenarios without degrading and altering the integrity and composition of retrieved documents. |
| Outcome: | The proposed strategy boosts performance in long-context scenarios without degrading and altering the integrity and composition of retrieved documents. |
Reimagining Retrieval Augmented Language Models for Answering Queries (2023.findings-acl)
Copied to clipboard
Wang-Chiew Tan, Yuliang Li, Pedro Rodriguez, Richard James, Xi Victoria Lin, Alon Halevy, Wen-tau Yih
| Challenge: | Large language models (LLMs) are expensive to train, deploy, and maintain, both financially and in terms of environmental impact. |
| Approach: | They present a reality check on large language models and compare their predictions to retrieval-augmented language models. |
| Outcome: | The proposed models fare better on question answering tasks and have become the foundation of impressive demos like Chat-GPT. |
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has proposed to improve relevance modeling by having large language models actively involved in retrieval, i.e., to guide retrieval with generation. |
| Approach: | They propose to have large language models actively involved in retrieval to guide retrieval with generation. |
| Outcome: | The proposed method synergizes retrieval and generation in an iterative manner, and can generate better results in subsequent iterations. |
Lexical Generalization Improves with Larger Models and Longer Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Excessive utilization of lexical overlap heuristics can lead to failure on challenging inputs. |
| Approach: | They analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension using a contrastive dataset. |
| Outcome: | The proposed model can be used to improve performance on a wide range of tasks, but it is often wrong. |
In-Context Reinforcement Learning with Retrieval-Augmented Generation for Text-to-SQL (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods of synthetic query generation generate mostly simple queries which might not be sufficiently representative of complex, real world queries. |
| Approach: | They propose to use large language models to fine tune query generation to produce complex queries that practitioners may pose during inference. |
| Outcome: | The proposed framework achieves 15-20% higher recall in database/table retrieval task compared to the existing state-of-the-art models for schema identification and upto 2% higher execution accuracy for SQL generation. |