Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriver + Flan-T5 are popular retriever-augmented language models for a variety of tasks. |
| Approach: | They evaluate the strengths and weaknesses of kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriver + Flan-T5 in reasoning over retrieved statements across different tasks. |
| Outcome: | The proposed models do not exhibit strong reasoning even when provided with only the required statements. |
Similar Papers
Retrieval-Augmented Retrieval: Large Language Models are Strong Zero-Shot Retriever (2024.findings-acl)
Copied to clipboard
Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Yibin Lei, Tianyi Zhou, Michael Blumenstein, Daxin Jiang
| Challenge: | Large-scale retrieval is indispensable in information-seeking tasks such as open-domain question answering and knowledgegrounded dialogue. |
| Approach: | They propose to use a large language model (LLM) to augment a query with its potential answers by prompting LLMs with a composition of the query and the query’s in-domain candidates. |
| Outcome: | The proposed method breaks brute-force combinations of retrievers with LLMs and lifts the performance of zero-shot retrieval to be very competitive on benchmark datasets. |
Unraveling and Mitigating Retriever Inconsistencies in Retrieval-Augmented Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-Augmented Large Language Models (RALMs) do not consistently outperform the original retrieval-free Language Model (LM). |
| Approach: | They propose a trainable framework that can adaptively retrieve from different knowledge sources and effectively decrease unpredictable reader errors. |
| Outcome: | The proposed framework significantly improves performance over the RALM with a single retriever by significantly reducing inconsistent behaviors. |
On Retrieval Augmentation and the Limitations of Language Model Training (2024.naacl-short)
Copied to clipboard
| Challenge: | Recent efforts to improve the performance of language models (LMs) have focused on scaling up model and training data size, though with steep accompanying energy and compute resource costs. |
| Approach: | They propose to augment a language model with k-nearest neighbors retrieval on its training data to reduce its perplexity. |
| Outcome: | The proposed model reduces storage costs by over 25x compared to traditional retrieval methods for GPT-2 and Mistral 7B . |
Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In (2023.acl-long)
Copied to clipboard
| Challenge: | Prior work on retrieval augmentation fine-tuned the retriever and the LM, making them closely coupled. |
| Approach: | They propose a generic retrieval plug-in that can be used to fine-tune retrieval augmentation and a LM to learn a user's preferences. |
| Outcome: | The proposed retriever improves the generalization of large language models on the MMLU and PopQA datasets by learning LM’s preferences from a known source LM . |
Reimagining Retrieval Augmented Language Models for Answering Queries (2023.findings-acl)
Copied to clipboard
Wang-Chiew Tan, Yuliang Li, Pedro Rodriguez, Richard James, Xi Victoria Lin, Alon Halevy, Wen-tau Yih
| Challenge: | Large language models (LLMs) are expensive to train, deploy, and maintain, both financially and in terms of environmental impact. |
| Approach: | They present a reality check on large language models and compare their predictions to retrieval-augmented language models. |
| Outcome: | The proposed models fare better on question answering tasks and have become the foundation of impressive demos like Chat-GPT. |
Great Memory, Shallow Reasoning: Limits of kNN-LMs (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing models trained on poor quality data have shown strong performance in language modeling and some downstream benchmarks. |
| Approach: | They evaluate kNN-LMs on a diverse set of tasks and evaluate their performance. |
| Outcome: | The proposed extension could improve on a variety of tasks, but it fails to perform on reasoning tasks that require integrating multiple pieces of information. |
ARL2: Aligning Retrievers with Black-box Large Language Models via Self-guided Adaptive Relevance Labeling (2024.acl-long)
Copied to clipboard
| Challenge: | Existing retrievers are misaligned with large language models due to separate training processes and inherent black-box nature of LLMs. |
| Approach: | They propose a retriever learning technique that harnesses LLMs as labelers to annotate and score adaptive relevance evidence. |
| Outcome: | Extensive experiments show that ARL2 improves accuracy and reduces the cost of API calls. |
Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LMs) excel in retrieving popular facts, but encounter difficulty with infrequent entity-relation pairs compared to retrievers. |
| Approach: | They propose to use a WiTQA dataset to explore the effects of combinations of entities and relations on LMs. |
| Outcome: | The proposed model can retain popular relations of less common entities while retaining the same popular relations. |
Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Retrieval-augmented large language models excel in various NLP tasks but are not always helpful when the knowledge required is absent in the model. |
| Approach: | They propose to determine whether the model is knowledgeable on a query via inspecting the (contextualized) pre-trained token embeddings of LLMs. |
| Outcome: | Experiments show that the proposed approach performs better than previous approaches on various benchmarks. |
In-Context Retrieval-Augmented Language Models (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing RALM methods focus on modifying the LM architecture to facilitate incorporation of external information, complicating deployment. |
| Approach: | They propose to condition a language model on relevant documents from a grounding corpus during generation by conditioning on external knowledge sources. |
| Outcome: | The proposed method significantly improves language modeling performance and provides natural source attribution mechanism. |