Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models' (LLMs) performance is greatly inflated by memorization, a study finds . authors propose a framework to analyze how LLMs reason under different degrees of memory access. |
| Approach: | They propose a framework that allows large language models to reason under different degrees of memory access. |
| Outcome: | Evaluating GPT-4o, LLaMA3.3-70B, and DeepSeek V3 on character-centric story understanding benchmarks, they find up to a 45.2% accuracy drop under the Restrictive Setting. |
Similar Papers
On LLM-Based Scientific Inductive Reasoning Beyond Equations (2025.emnlp-main)
Copied to clipboard
Brian S. Lin, Jiaxin Yuan, Zihan Zhou, Shouli Wang, Shuo Wang, Cunliang Kong, Qi Shi, Yuxuan Li, Liner Yang, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing research on inductive reasoning models emphasizes rule design without grounding them in specific scenarios. |
| Approach: | They propose to use LLMs to learn underlying patterns from limited examples in entirely new environments. |
| Outcome: | The proposed benchmark evaluates the inductive reasoning abilities of large language models in scientific settings. |
Out-of-Context Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lightweight technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
| Approach: | They propose a lightweight technique that trains only new token embeddings on axioms . they train only new embeddables and evaluate them on unseen tasks . |
| Outcome: | The proposed technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
Working Memory Identifies Reasoning Limits in Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models, we examine the limitations of their cognitive capabilities and their working memory. |
| Approach: | They examine the limitations of large language models from a scaling perspective . they also assess various prompting strategies, revealing their diverse impacts on LLM performance. |
| Outcome: | The proposed models perform poorly on n-back tasks and on prompting strategies. |
Demystifying Verbatim Memorization in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that Large Language Models (LLMs) memorize long sequences verbatim, with serious copyright and privacy implications. |
| Approach: | They develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences. |
| Outcome: | The proposed framework creates a control model M () and a treatment model M with injected sequences. |
From Remembering to Metacognition: Do Existing Benchmarks Accurately Evaluate LLMs? (2025.findings-emnlp)
Copied to clipboard
Geng Zhang, Yizhou Ying, Sihang Jiang, Jiaqing Liang, Guanglei Yue, Yifei Fu, Hailin Hu, Yanghua Xiao
| Challenge: | Existing benchmark datasets focus on low-level cognitive tasks while providing limited coverage of higher-level reasoning skills. |
| Approach: | They analyze the cognitive depth of popular LLM benchmarks using Bloom’s Taxonomy to evaluate both the cognitive and knowledge dimensions. |
| Outcome: | The results show that incorporating higher-level cognitive instructions into the current instruction fine-tuning process improves model performance. |
LlmLink: Dual LLMs for Dynamic Entity Linking on Long Narratives with Collaborative Memorisation and Prompt Optimisation (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods focus on supervised fine-tuning or limited to one-off prediction, which poses a challenge where the context is long. |
| Approach: | They propose a dynamic approach to CoREFerence resolution in chunked long narratives by deploying dual Large Language Models. |
| Outcome: | The proposed model achieves performance gains over existing models and fine-tuning approaches on long narrative datasets, significantly reducing the resources required for inference and training. |
An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint (2025.emnlp-main)
Copied to clipboard
Yi Sun, Han Wang, Jiaqiang Li, Jiacheng Liu, Xiangyu Li, Hao Wen, Yizhen Yuan, Huiwen Zheng, Yan Liang, Yuanchun Li, Yunxin Liu
| Challenge: | Large Language Models (LLMs) are a powerful tool for test-time scaling, but they are often used under time constraints. |
| Approach: | They propose to use LLMs to make models think before answering questions . they also use self-correction and best-of-N decoding to encourage deeper thinking . |
| Outcome: | The proposed models are able to achieve higher inference accuracy with extra inference computation under time constraints. |
Answer When Needed, Forget When Not: Language Models Pretend to Forget via In-Context Knowledge Unlearning (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly required to selectively unlearn specific information. |
| Approach: | They propose a method which fine-tunes pre-trained LLMs to enable prompt unlearning of target knowledge within the context while preserving unrelated information. |
| Outcome: | The proposed method achieves up to 95% forget accuracy while retaining 80% of unrelated knowledge, significantly outperforming baselines in both in-domain and out-of-domain scenarios. |
Memorization or Reasoning? Exploring the Idiom Understanding of LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | idioms have long posed a challenge due to their unique linguistic properties, which set them apart from other common expressions. |
| Approach: | They propose to use a large-scale dataset of idioms in six languages to evaluate LLMs' idiomatic processing ability. |
| Outcome: | The proposed model integrates contextual cues and reasoning to improve idiom understanding in LLMs, suggesting that their performance is influenced by memorization and reasoning. |
Controllable Memorization in LLMs via Weight Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on mitigating memorization, but the deliberate control of memorisation has been underexplored. |
| Approach: | They propose a gradient-based weight pruning framework to control memorization rates in large language models by fine-grained control over pruning parameters. |
| Outcome: | The proposed framework enables models to suppress or enhance memorization based on application-specific requirements. |