Entropy-Based Decoding for Retrieval-Augmented Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Despite their success, retrieval-augmented LLMs still face the distractibility issue, where the generated responses are negatively influenced by noise from both external and intrinsic knowledge sources. |
| Approach: | They propose a entropy-based document-parallel ensemble decoding method that prioritizes low-entropies from retrieved documents and incorporates a contrastive decoding mechanism that contrasts the obtained low- and high-entropic ensemble distributions with the high-end internal knowledge across layers. |
| Outcome: | The proposed method improves on open-domain question answering datasets and shows that it is highly efficient. |
Similar Papers
Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts (2024.findings-emnlp)
Copied to clipboard
Youna Kim, Hyuhng Joon Kim, Cheonbok Park, Choonghyun Park, Hyunsoo Cho, Junyeob Kim, Kang Min Yoo, Sang-goo Lee, Taeuk Kim
| Challenge: | Recent research has been developed to amplify contextual knowledge over parametric knowledge of large language models (LLMs) in knowledge-intensive tasks such as open-domain question-answering . |
| Approach: | They propose to amplify contextual knowledge over parametric knowledge of large language models (LLMs) by contrastive decoding to leverage contextual influence effectively. |
| Outcome: | The proposed approach improves open-domain question answering tasks especially in robustness by remaining undistracted by noisy contexts in retrieval-augmented generation. |
Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit impressive natural language capabilities but suffer from hallucination – generating content that does not align with realworld facts. |
| Approach: | They propose to extrapolate critical token probabilities beyond the last layer to improve decoding by manipulating the predicted distributions at inference time. |
| Outcome: | The proposed methods surpass state-of-the-art on multiple datasets by large margins. |
Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Retrieval-augmented large language models excel in various NLP tasks but are not always helpful when the knowledge required is absent in the model. |
| Approach: | They propose to determine whether the model is knowledgeable on a query via inspecting the (contextualized) pre-trained token embeddings of LLMs. |
| Outcome: | Experiments show that the proposed approach performs better than previous approaches on various benchmarks. |
Retrieval-Augmented Retrieval: Large Language Models are Strong Zero-Shot Retriever (2024.findings-acl)
Copied to clipboard
Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Yibin Lei, Tianyi Zhou, Michael Blumenstein, Daxin Jiang
| Challenge: | Large-scale retrieval is indispensable in information-seeking tasks such as open-domain question answering and knowledgegrounded dialogue. |
| Approach: | They propose to use a large language model (LLM) to augment a query with its potential answers by prompting LLMs with a composition of the query and the query’s in-domain candidates. |
| Outcome: | The proposed method breaks brute-force combinations of retrievers with LLMs and lifts the performance of zero-shot retrieval to be very competitive on benchmark datasets. |
Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models (2025.acl-long)
Copied to clipboard
Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, Dong Yu
| Challenge: | Large language models have demonstrated remarkable performance across a wide range of language tasks due to their remarkable ability in context modeling. |
| Approach: | They propose to use parallel context encoding to reduce attention entropy by incorporating attention sinks and selective mechanisms to reduce irregular attention . they also propose to incorporate attention sink mechanisms into the parallel encoded context to reduce the irregular attention. |
| Outcome: | The proposed methods lower irregular attention entropy and narrow performance gaps. |
Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have limited generalization capabilities and performance challenges. |
| Approach: | They evaluate 15 different backbone LLMs and non-LLMs to evaluate their performance . larger models and extensive pre-training consistently enhance in-domain accuracy and data efficiency . |
| Outcome: | The results show that larger models and extensive pre-training enhance in-domain accuracy and data efficiency. |
Not All Citations Are Equal:Entropy-Guided Citation Selection for Noise-Resistant Medical LLM (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models have demonstrated extensive potential in medical applications . however, their practical deployment in healthcare faces significant challenges . |
| Approach: | They propose a training-free multi-turn reasoning framework and a post-training methodology that provides external knowledge support for large language models. |
| Outcome: | The proposed framework elicits internal thought, external thought, and fusion thought, with an entropy-based reward that encourages selective citation of beneficial external knowledge while penalizing noisy citations. |
DVD: Dynamic Contrastive Decoding for Knowledge Amplification in Multi-Document Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate information with hallucinations due to uneven retrieval quality and irrelevant contents. |
| Approach: | They propose a decoding strategy which dynamically amplifies knowledge from selected documents during the generation phase. |
| Outcome: | The proposed method outperforms other decoding strategies on ALCE-ASQA, NQ, TQA and PopQA benchmarks. |
RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods rely on separate retrievers to fetch top-k text chunks for generating evidence, and they lack joint optimization. |
| Approach: | They propose a framework that integrates retrieval and generation into a single, auto-regressive process, enabling LLMs to directly generate fine-grained evidence from the corpus with constrained decoding. |
| Outcome: | Extensive experiments on five open-domain QA datasets demonstrate the proposed framework’s superior performance across both in-domain and out-of-domain tasks. |
Lexicon-Enhanced Self-Supervised Training for Multilingual Dense Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent multilingual pre-trained models perform poorly on multilingual retrieval tasks due to lack of multilingual training data. |
| Approach: | They propose to mine and generate self-supervised training data based on large-scale unlabeled corpus and introduce query generator to generate more queries in target languages for unlabed passages. |
| Outcome: | The proposed method performs better than baselines on a Mr. TYDI dataset and an industrial dataset from a commercial search engine. |