Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have demonstrated effectiveness in zero-shot retrieval tasks using large language models. |
| Approach: | They challenge this assumption by analyzing whether knowledge leakage in benchmarks contributes to performance gains. |
| Outcome: | The proposed methods have demonstrated significant performance gains across multiple benchmarks. |
Similar Papers
When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets (2024.findings-eacl)
Copied to clipboard
| Challenge: | Using large language models (LMs) for query or document expansion can improve generalization in information retrieval. |
| Approach: | They conduct the first comprehensive analysis of large language models (LMs) for query or document expansion. |
| Outcome: | The proposed expansions improve retrieval performance for weaker models but harm stronger models. |
Enabling LLM Knowledge Analysis via Extensive Materialization (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have majorly advanced NLP and AI, and a major success factor is their internalized factual knowledge. |
| Approach: | They propose a method to comprehensively materialize an LLM’s factual knowledge through recursive querying and result consolidation. |
| Outcome: | The proposed method provides constructive insights into the scope and structure of LLM knowledge (or beliefs) it provides scale, accuracy, bias, cutoff and consistency at the same time. |
Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Retrieval-augmented large language models excel in various NLP tasks but are not always helpful when the knowledge required is absent in the model. |
| Approach: | They propose to determine whether the model is knowledgeable on a query via inspecting the (contextualized) pre-trained token embeddings of LLMs. |
| Outcome: | Experiments show that the proposed approach performs better than previous approaches on various benchmarks. |
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2025.naacl-long)
Copied to clipboard
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, Manaal Faruqui
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly enhanced their capabilities across various cognitive tasks. |
| Approach: | They propose a high-quality evaluation dataset to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. |
| Outcome: | The proposed framework improves performance in end-to-end RAG scenarios. |
Data Doping or True Intelligence? Evaluating the Transferability of Injected Knowledge in LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a study shows that comprehension-intensive fine-tuning tasks retain knowledge longer . however, all models exhibit significant performance drops when applying injected knowledge in broader contexts . |
| Approach: | study: comprehension-intensive fine-tuning tasks achieve higher knowledge retention rates . larger models show improved retention across all task types, study finds . |
| Outcome: | a new study shows that comprehension-intensive fine-tuning tasks retain knowledge better than mapping-oriented tasks despite exposure to identical factual content. |
Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection (2026.acl-long)
Copied to clipboard
Yuwei Zhang, Wenhao Yu, Shangbin Feng, Yifan Zhu, Letian Peng, Jayanth Srinivasa, Gaowen Liu, Jingbo Shang
| Challenge: | Existing knowledge injection benchmarks for large language models lack standardized testing grounds. |
| Approach: | They propose a knowledge injection benchmark that leverages recently-added and expert-curated facts from Wikipedia’s “Did You Know...” entries. |
| Outcome: | The proposed framework improves reliability accuracy by 29.1%. |
When Do LLMs Need Retrieval Augmentation? Mitigating LLMs’ Overconfidence Helps Retrieval Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have difficulty knowing they do not possess certain knowledge and tend to provide specious answers in such cases. |
| Approach: | They propose to use Retrieval Augmentation to enhance LLMs' ability to perceive their knowledge boundaries to reduce overconfidence. |
| Outcome: | The proposed methods reduce overconfidence and improve accuracy in large language models with fewer retrieval calls. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
Investigating How Pre-training Data Leakage Affects Models’ Reproduction and Detection Capabilities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies do not examine how leaked instances in training datasets influence LLMs’ output and detection capabilities. |
| Approach: | They conduct an experimental survey to examine the relationship between data leakage in training datasets and its effects on the generation and detection by Large Language Models (LLMs). |
| Outcome: | The results show that enhancing leakage detection through few-shot learning can help mitigate the impact of the leakage rate in the training data on detection performance. |
Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have raised concerns about reliability and trustworthiness of the models. |
| Approach: | They analyze 134 papers and introduce a taxonomy of evidence-based text generation with LLMs. |
| Outcome: | The proposed methods highlight open challenges and outline promising directions for future work. |