Permitted Knowledge Boundary: Evaluating the Knowledge-Constrained Responsiveness of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has raised concerns about the controllability of large language models. |
| Approach: | They propose to define a "boundary bias" to depict KCR in large language models . they propose to quantify the boundary bias of LLMs and assess the KCR . |
| Outcome: | The proposed model is based on two new datasets to assess its performance. |
Similar Papers
Knowledge Boundary of Large Language Models: A Survey (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) store vast amount of knowledge in their parameters, but they still have limitations in the memorization and utilization of certain knowledge. |
| Approach: | They propose a comprehensive definition of the LLM knowledge boundary and introduce a formalized taxonomy categorizing knowledge into four distinct types. |
| Outcome: | The proposed definition of the LLM knowledge boundary and taxonomy categorizes knowledge into four distinct types . aims to offer a comprehensive overview, facilitate access to key issues, and inspire further advancements in LLM research. |
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are impressive in solving tasks, but they can quickly be outdated after deployment. |
| Approach: | They provide a review of recent advances in aligning deployed large language models with the ever-changing world knowledge. |
| Outcome: | The proposed models can be used to perform various tasks directly through in-context learning or for further fine-tuning for domain-specific uses. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Evaluating Large Language Models via Linguistic Profiling (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) undergo extensive evaluation against various benchmarks collected in established leaderboards to assess their performance across multiple tasks. |
| Approach: | They propose a new evaluation methodology to test LLMs' sentence generation abilities under specific linguistic constraints. |
| Outcome: | The proposed evaluation methodology is based on the 'linguistic profiling' approach and is not intended to be a task-oriented evaluation. |
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Large Language Model Performance. (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Structured generation is used to extract key output information from large language models (LLMs). |
| Approach: | They examine whether constraints on generation space impact LLMs’ abilities, including reasoning and domain knowledge comprehension. |
| Outcome: | The proposed model is based on a few-shot in-context learning and instruction-following capabilities. |
Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have improved performance across tasks . however, the sensitivity of LLMs to prompt leads to unreliability of evaluation results . |
| Approach: | They propose a new concept to evaluate language models with a fixed question or limited paraphrases as the query. |
| Outcome: | The proposed method outperforms existing benchmarks on multiple language models . it avoids prompt sensitivity, rendering models more reliable and robust . |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
Assessing the Capabilities of Large Language Models in Coreference: An Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are a new approach to coreference resolution, but their performance is not yet fully understood. |
| Approach: | They propose that future efforts should improve scope, data, and evaluation methods of traditional coreference research to adapt to the development of LLMs. |
| Outcome: | The proposed methods improve scope, data, and evaluation methods of traditional coreference research to adapt to the development of LLMs. |
Are Your LLMs Capable of Stable Reasoning? (2025.findings-acl)
Copied to clipboard
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, Kai Chen
| Challenge: | Existing evaluation protocols and metrics do not capture the full spectrum of LLM capabilities, especially in complex reasoning tasks. |
| Approach: | They propose a new evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model’s potential capabilities and operational consistency. |
| Outcome: | The proposed evaluation metric measures model performance across multiple sampling attempts and provides comprehensive insights into their potential capabilities and operational consistency. |
Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive prowess in solving a wide range of tasks with world knowledge, but it remains unclear how well they perceive their factual knowledge boundaries. |
| Approach: | They propose to use a retrieval augmentation approach to enhance LLMs' awareness of factual knowledge boundaries to analyze factual and factual information in open-domain question answering (QA) |
| Outcome: | The proposed method improves LLMs’ QA and judgemental capabilities by integrating supporting documents with the questions. |