Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive prowess in solving a wide range of tasks with world knowledge, but it remains unclear how well they perceive their factual knowledge boundaries. |
| Approach: | They propose to use a retrieval augmentation approach to enhance LLMs' awareness of factual knowledge boundaries to analyze factual and factual information in open-domain question answering (QA) |
| Outcome: | The proposed method improves LLMs’ QA and judgemental capabilities by integrating supporting documents with the questions. |
Similar Papers
When Do LLMs Need Retrieval Augmentation? Mitigating LLMs’ Overconfidence Helps Retrieval Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have difficulty knowing they do not possess certain knowledge and tend to provide specious answers in such cases. |
| Approach: | They propose to use Retrieval Augmentation to enhance LLMs' ability to perceive their knowledge boundaries to reduce overconfidence. |
| Outcome: | The proposed methods reduce overconfidence and improve accuracy in large language models with fewer retrieval calls. |
Knowledge Boundary of Large Language Models: A Survey (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) store vast amount of knowledge in their parameters, but they still have limitations in the memorization and utilization of certain knowledge. |
| Approach: | They propose a comprehensive definition of the LLM knowledge boundary and introduce a formalized taxonomy categorizing knowledge into four distinct types. |
| Outcome: | The proposed definition of the LLM knowledge boundary and taxonomy categorizes knowledge into four distinct types . aims to offer a comprehensive overview, facilitate access to key issues, and inspire further advancements in LLM research. |
Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LMs) excel in retrieving popular facts, but encounter difficulty with infrequent entity-relation pairs compared to retrievers. |
| Approach: | They propose to use a WiTQA dataset to explore the effects of combinations of entities and relations on LMs. |
| Outcome: | The proposed model can retain popular relations of less common entities while retaining the same popular relations. |
KBM: Delineating Knowledge Boundary for Adaptive Retrieval in Large Language Models (2025.findings-emnlp)
Copied to clipboard
Zhen Zhang, Xinyu Wang, Yong Jiang, Zile Qiao, Zhuo Chen, Guangyu Li, Feiteng Mu, Mengting Hu, Pengjun Xie, Fei Huang
| Challenge: | Retrieval-augmented generation (RAG) is employed to tackle these challenges . a Knowledge Boundary Model (KBM) is used to express the known/unknown of a given question . |
| Approach: | They propose a Knowledge Boundary Model to express the known/unknown of a given question . they find that not all questions need to trigger RAG to improve performance . |
| Outcome: | The proposed model reduces time and computational costs by retrieving parts of unknown knowledge . the proposed model can express the known/unknown of a given question and determine whether a RAG needs to be triggered . |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are useful interfaces to factual knowledge, but their usefulness is limited by their tendency to deliver inconsistent answers to semantically equivalent questions. |
| Approach: | They evaluate the effectiveness of up-scaling and augmenting the LM with a passage retrieval database to reduce inconsistency. |
| Outcome: | The proposed models reduce inconsistency but retrieval augmentation is more efficient. |
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models struggle with tasks requiring rich world knowledge, implying the difficulty of encoding a wealth of world knowledge in their parameters. |
| Approach: | They propose a retrieval-augmentation method that improves performance and reduces inference costs by only retrieving non-parametric memories when necessary. |
| Outcome: | The proposed method improves performance and reduces inference costs by only retrieving non-parametric memories when necessary. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
LLMs are Biased Evaluators But Not Biased for Fact-Centric Retrieval Augmented Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models (LLMs) exhibit significant biases in evaluation tasks, especially in preferentially rating and favoring self-generated content. |
| Approach: | They propose to simulate two critical phases of retrieval-augmented generation (RAG) frameworks where keyword extraction and factual accuracy take precedence over stylistic elements. |
| Outcome: | The proposed model emulates two critical phases of the retrieval-augmented generation framework. |
Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have improved performance across tasks . however, the sensitivity of LLMs to prompt leads to unreliability of evaluation results . |
| Approach: | They propose a new concept to evaluate language models with a fixed question or limited paraphrases as the query. |
| Outcome: | The proposed method outperforms existing benchmarks on multiple language models . it avoids prompt sensitivity, rendering models more reliable and robust . |