Memorization ≠ Understanding: Do Large Language Models Have the Ability of Scenario Cognition? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive performance across NLP tasks. |
| Approach: | They propose a framework to assess LLMs’ scenario cognition . they examine the ability to link semantic scenario elements with their arguments in context . |
| Outcome: | The proposed framework assesses large language models’ scenario cognition . it shows that current models rely on superficial memorization, failing to achieve robust semantic scenario cognition even in simple cases. |
Similar Papers
Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on metaphor processing have focused on single datasets and specific task settings, often using artificially constructed data through lexical replacement. |
| Approach: | They propose to evaluate the capabilities of Large Language Models (LLMs) in metaphor interpretation across multiple datasets, tasks, and prompt configurations. |
| Outcome: | The proposed frameworks are more realistic and efficient than current models and are more efficient than existing models. |
Do Large Language Models Know How Much They Know? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models are highly capable systems, but their capabilities and limitations are unclear. |
| Approach: | They develop a benchmark that challenges LLMs to recall all information they possess on specific topics. |
| Outcome: | The proposed model can recall excessive, insufficient, or the precise amount of information they possess on a given topic, indicating their awareness of how much they know about the given topic. |
How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models exhibit increasingly sophisticated linguistic capabilities, yet the extent to which these models reflect human-like cognition versus advanced pattern recognition remains an open question. |
| Approach: | They conduct a series of targeted experiments to assess whether LLMs construct semantic representations and pragmatic inferences in a human-like manner. |
| Outcome: | The proposed framework can be used to assess the cognitive and linguistic capabilities of large language models (LLMs). |
FAC2E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are evaluated by overall performance on various text understanding and generation tasks. |
| Approach: | They propose a framework for Fine-grAined and Cognition-grounded LLMs’ Capability Evaluation that dissociates the language-related capabilities from cognition-related ones. |
| Outcome: | The proposed framework dissociates the language-related capabilities from cognition-related ones and breaks down the process of applying a specific capability into three sub-steps: recalling relevant knowledge, utilizing knowledge, and solving problems. |
A Multi-Perspective Analysis of Memorization in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can generate the same sequences contained in the pre-train corpus, known as memorization. |
| Approach: | They analyze the relationship between memorization and outputs from Large Language Models (LLMs) they show a sudden drop and increase in the frequency of input tokens when generating memorized/unmemorized sequences . |
| Outcome: | The proposed model can generate the same sequences contained in the pre-train corpus, and it can predict unmemorized tokens. |
Language Models Struggle to Use Representations Learned In-Context (2026.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that large language models are capable of inducing rich representations of data that are seen in-context . a novel task, adaptive world modeling, shows that even the most performant LLMs cannot reliably leverage novel semantics defined in-constitut. |
| Approach: | They propose to use in-context representations to induce rich representations of data . they also propose to probe models using a novel task to enable flexible deployment . |
| Outcome: | The proposed model can use in-context representations to complete simple downstream tasks. |
From Remembering to Metacognition: Do Existing Benchmarks Accurately Evaluate LLMs? (2025.findings-emnlp)
Copied to clipboard
Geng Zhang, Yizhou Ying, Sihang Jiang, Jiaqing Liang, Guanglei Yue, Yifei Fu, Hailin Hu, Yanghua Xiao
| Challenge: | Existing benchmark datasets focus on low-level cognitive tasks while providing limited coverage of higher-level reasoning skills. |
| Approach: | They analyze the cognitive depth of popular LLM benchmarks using Bloom’s Taxonomy to evaluate both the cognitive and knowledge dimensions. |
| Outcome: | The results show that incorporating higher-level cognitive instructions into the current instruction fine-tuning process improves model performance. |
Can Large Language Models Understand Context? (2024.findings-eacl)
Copied to clipboard
Yilun Zhu, Joel Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, Bo-Hsiang Tseng
| Challenge: | Existing evaluation methodologies for Large Language Models (LLMs) have been inadequate to evaluate their ability to understand contextual features. |
| Approach: | They propose a benchmark to assess large language models' ability to understand context by adapting existing datasets to suit their evaluation. |
| Outcome: | The proposed model performs better under the in-context learning pretraining scenario than state-of-the-art models. |
Memorization or Reasoning? Exploring the Idiom Understanding of LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | idioms have long posed a challenge due to their unique linguistic properties, which set them apart from other common expressions. |
| Approach: | They propose to use a large-scale dataset of idioms in six languages to evaluate LLMs' idiomatic processing ability. |
| Outcome: | The proposed model integrates contextual cues and reasoning to improve idiom understanding in LLMs, suggesting that their performance is influenced by memorization and reasoning. |
Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using multilingual models, we find that treating languages in isolation obscures the true patterns of memorization. |
| Approach: | They propose a graph-based correlation metric that incorporates language similarity to analyze cross-lingual memorization. |
| Outcome: | The proposed model incorporates language similarity to analyze cross-lingual memorization in 95 languages. |