MEMORIZATION

Personal Information Parroting in Language Models (2026.findings-eacl)

Challenge:	Recent work suggests that success stems from memorizing single prototypical words for each relation.
Approach:	They propose a neural paraphrasing approach that maps NCs to paraphrases that express the relation between constituent words.
Outcome:	The proposed method performs better when memorization is not possible.

Copied to clipboard

Nishant Subramani, Kshitish Ghate, Mona T. Diab

Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs (2024.emnlp-main)

Challenge:	Modern language models memorize millions of PI instances, increasing privacy risks.
Approach:	They develop a model that parrots 13.6% of PI verbatim on a manually curated set of 483 instances . they recommend that pretraining datasets be aggressively filtered and anonymized to minimize PI parroting.
Outcome:	The proposed model outperforms the best regex-based PI detectors on a manually curated set of 483 instances of PI.

Copied to clipboard

Kanishka Misra, Kyle Mahowald

Evaluating LLMs for Quotation Attribution in Literary Texts: A Case Study of LLaMa3 (2025.naacl-short)

Challenge:	Language models learn rare syntactic phenomena by generalization vs. memorization, a study finds . aannalysis experiments show that humans learn rare grammatical structures by generalizing from less rare phenomena.
Approach:	They iteratively trained transformer language models on a systematically manipulated corpus and evaluated their learning of a rare grammatical phenomenon.
Outcome:	The results show that language models learn rare grammatical phenomena by generalization vs. memorization . human-scale corpora are used to train the models and compare their learning to counterfactual corpors .

Copied to clipboard

Gaspard Michel, Elena V. Epure, Romain Hennequin, Christophe Cerisara

Challenge:	Large Language Models (LLMs) have shown promising results in literary tasks . however, quotation attribution remains a challenging task and methods that generalize across writing styles are lacking analysis regarding book memorization and annotation contamination.
Approach:	They evaluate the ability of Llama-3 to attribute utterances of direct-speech to their speaker in novels by assessing the impact of book memorization and annotation contamination.
Outcome:	The proposed model outperforms existing models on a corpus of 28 novels and shows that book memorization and annotation contamination do not explain the performance gain.

ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks (2024.findings-naacl)

Copied to clipboard

Xiaodong Yu, Hao Cheng, Xiaodong Liu, Dan Roth, Jianfeng Gao

Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time (2025.emnlp-main)

Challenge:	Existing static benchmarks do not guarantee that models can use the provided evidence for answering, which is essential to avoid hallucination when the required knowledge is new or private.
Approach:	They propose to automatically perturb existing static one for dynamic evaluation by using a chatGPT framework and a set of open-domain QA datasets.
Outcome:	The proposed framework generates new test cases on two open-domain QA datasets and is human-readable and useful to trigger hallucination in LLMs.

Copied to clipboard

Huihan Li, You Chen, Siyuan Wang, Yixin He, Ninareh Mehrabi, Rahul Gupta, Xiang Ren

Memory Dial: A Training Framework for Controllable Memorization in Language Models (2026.findings-acl)

Challenge:	Large Language Models (LLMs) perform well on reasoning benchmarks but often fail when inputs alter slightly, raising concerns about overreliance on memorization.
Approach:	They propose a framework for Source-aware Token-level Identification of Memorization which attributes each token in a reasoning chain to one of multiple memorization sources based on their statistical co-occurrence with the token in the pretraining corpus.
Outcome:	The proposed framework attributes each token in a reasoning chain to one of multiple memorization sources based on their statistical co-occurrence with the token in the pretraining corpus.

Copied to clipboard

Xiangbo Zhang, Ali Emami

AgentTuning: Enabling Generalized Agent Abilities for LLMs (2024.findings-acl)

Challenge:	Existing approaches to memorization detection are post-hoc . large language models can reproduce training data verbatim, complicating accuracy estimates .
Approach:	They propose a training framework that makes memorization an explicit variable.
Outcome:	The proposed framework produces models identical in architecture, data, and optimization, but varying in memorization pressure.

Copied to clipboard

Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, Jie Tang

GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021.findings-emnlp)

Challenge:	Open large language models (LLMs) with great performance in various tasks are far inferior to commercial models such as ChatGPT and GPT-4 when acting as agents to tackle complex tasks in the real world.
Approach:	They propose a method to enhance the agent capabilities of LLMs while maintaining their general abilities.
Outcome:	The AgentLM-70B is comparable to GPT-3.5-turbo on unseen agent tasks, demonstrating generalized agent capabilities.

Copied to clipboard

Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, Woomyoung Park

Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning (2024.findings-emnlp)

Challenge:	Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability.
Approach:	They propose a data augmentation technique that leverages large-scale language models to generate real text samples from a mixture of real samples.
Outcome:	The proposed method outperforms existing methods on diverse classification tasks.

Copied to clipboard

Akshara Prabhakar, Thomas Griffiths, R. McCoy

Span Selection Pre-training for Question Answering (2020.acl-main)

Challenge:	Chain-of-Thought (CoT) prompting has been shown to enhance the multi-step reasoning capabilities of Large Language Models (LLMs).
Approach:	They propose to use CoT prompting to analyze a symbolic reasoning task where letters are shifted forward some number of steps in the alphabet.
Outcome:	The proposed model performs well on a symbolic reasoning task, with three LLMs performing the task using CoT prompts.

Copied to clipboard

Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, G P Shrivatsa Bhargav, Dinesh Garg, Avi Sil

Natural Language Interfaces

Challenge:	Pre-trained BERTs provide large gains across many language understanding tasks, achieving a new state-of-the-art (SOTA).
Approach:	They propose a new pre-training task inspired by reading comprehension to better align the pre- training from memorization to understanding.
Outcome:	The proposed model outperforms BERT-BASE and BERT LARGE on a new dataset and improves answer prediction F1 by 4 points and supporting fact prediction F1.

Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs (2026.eacl-long)

Copied to clipboard

Yuxuan Jiang, Francis Ferraro

Does it Really Generalize Well on Unseen Data? Systematic Evaluation of Relational Triple Extraction Methods (2022.naacl-main)

Challenge:	Large Language Models' (LLMs) performance is greatly inflated by memorization, a study finds . authors propose a framework to analyze how LLMs reason under different degrees of memory access.
Approach:	They propose a framework that allows large language models to reason under different degrees of memory access.
Outcome:	Evaluating GPT-4o, LLaMA3.3-70B, and DeepSeek V3 on character-centric story understanding benchmarks, they find up to a 45.2% accuracy drop under the Restrictive Setting.

Copied to clipboard

Juhyuk Lee, Min-Joong Lee, June Yong Yang, Eunho Yang

Information Extraction & Text Mining

Challenge:	Existing extraction models memorize and recall already seen triples but cannot generalize effectively for unseen triples.
Approach:	They propose a method to generalize existing extraction models by rearranging datasets and augmenting test sets.
Outcome:	The proposed method can significantly increase the generalization performance of existing models.

Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teaching (2025.findings-acl)

Copied to clipboard

Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Yipeng Zhang, Haitao Mi, Helen M. Meng

Quantifying Adaptability in Pre-trained Language Models with 500 Tasks (2022.naacl-main)

Challenge:	Existing approaches to keeping large language models current involve continued pre-training on new documents.
Approach:	They propose a learning framework that augments documents with knowledge-intensive tasks created in a self-supervised manner, focusing on memorization, comprehension, and self-reflection.
Outcome:	The proposed learning framework improves an LLM’s ability to acquire new knowledge from unseen raw documents through self-teaching.

Copied to clipboard

Belinda Li, Jane Yu, Madian Khabsa, Luke Zettlemoyer, Alon Halevy, Jacob Andreas

Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks (2022.findings-emnlp)

Challenge:	a recent study examines the features and limits of LM adaptability to new tasks . many questions about the nature and limits remain unanswered .
Approach:	They evaluate adaptability to new tasks using a new benchmark, TaskBench500 . they find adaptation procedures differ dramatically in their ability to memorize small datasets .
Outcome:	The proposed benchmark compares 500 procedurally generated sequence modeling tasks to a new benchmark.

Copied to clipboard

Vikas Raunak, Arul Menezes

Text Generation

Challenge:	Memorization presents a challenge for constrained Natural Language Generation tasks . previous studies focused on counterfactual memorization, linking it to hallucinations .
Approach:	They propose an algorithm for extractive memorization in constrained sequence generation tasks . they propose to elicit non-memorized translations of memorized samples from the same model .
Outcome:	The proposed algorithm could be leveraged to mitigate memorization in the model through finetuning.

When and how to paraphrase for named entity recognition? (2023.acl-long)

Copied to clipboard

Saket Sharma, Aviral Joshi, Yiyun Zhao, Namrata Mukhija, Hanoz Bhathena, Prateek Singh, Sashank Santhanam

Information Extraction & Text Mining

Challenge:	Named entity recognition (NER) is a key component underpinning many industrial pipelines for a variety of downstream applications.
Approach:	They propose to use back translation to annotate entity spans in generations and propose a paraphraser with a larger dataset.
Outcome:	The proposed method improves NER performance across different datasets with gold annotations and paraphrasing strength.

ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models (2023.acl-long)

Copied to clipboard

Jonas Belouadi, Steffen Eger

Investigating Memorization of Conspiracy Theories in Text Generation (2021.findings-acl)

Challenge:	End-to-end models learn to complete a task by directly learning all steps, without intermediary algorithms such as hand-crafted rules or post-processing.
Approach:	They propose to train end-to-end poetry generation conditioned on styles such as rhyme, meter, and alliteration . they pre-train ByGPT5, a new token-free decoder-only language model, and fine-tune it on a custom corpus of English and German quatrains .
Outcome:	The proposed model outperforms other models on a large custom corpus of English and German quatrains while being more parameter efficient and performing favorably compared to humans.

Copied to clipboard

Sharon Levy, Michael Saxon, William Yang Wang

Efficient Continue Training of Temporal Language Model with Structural Information (2023.findings-emnlp)

Challenge:	Existing studies examine conspiracy theories in social media, but they have not evaluated their presence in generative language models.
Approach:	They examine the ability of generative language models to generate conspiracy theory text . they highlight the difficulties of this task and discuss the drawbacks .
Outcome:	The proposed model can generate conspiracy theories without access to training data.

Copied to clipboard

Zhaochen Su, Juntao Li, Zikang Zhang, Zihan Zhou, Min Zhang

ALPACA AGAINST VICUNA: Using LLMs to Uncover Memorization of LLMs (2025.naacl-long)

Challenge:	Existing temporal language models are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components.
Approach:	They propose a method that captures syntactically changed tokens and captures the relationship between the time prefix and tokens.
Outcome:	The proposed method outperforms existing temporal language models on two datasets and three tasks.

Copied to clipboard

Aly M. Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, Santu Rana

An Empirical Study of Memorization in NLP (2022.acl-long)

Challenge:	Existing studies have shown that pre-trained LLMs emit training data up to 150 more often than in regular operation.
Approach:	They propose a black-box prompt optimization method where an attacker LLM agent uncovers higher levels of memorization in a victim agent .
Outcome:	The proposed method shows 23.7% more overlap with training data compared to state-of-the-art baselines.

Copied to clipboard

Xiaosen Zheng, Jing Jiang

Linguistic Theories

Challenge:	Existing studies see memorization as hindering generalization in deep learning models.
Approach:	They propose a long-tail theory to explain the memorization behavior of deep learning models . they use three different NLP tasks to test whether the theory holds .
Outcome:	The proposed long-tail theory is validated in three NLP tasks and shows it is faithful.

On the Reliability of Large Language Models for Causal Discovery (2025.acl-long)

Copied to clipboard

Tao Feng, Lizhen Qu, Niket Tandon, Zhuang Li, Xiaoxi Kang, Gholamreza Haffari

Analyzing Memorization in Large Language Models through the Lens of Model Attribution (2025.naacl-long)

Challenge:	Existing statistical methods to identify causal relationships from observational data remain elusive.
Approach:	They examine the impact of memorization for accurate causal relation prediction, the influence of incorrect causal relations in pre-training data and the contextual nuances that influence LLMs’ understanding of causal relations.
Outcome:	The proposed models are effective in recognizing causal relations that occur frequently in pre-training data, but their ability to generalize to new or rare causal relations is limited.

Copied to clipboard

Tarun Ram Menta, Susmit Agrawal, Chirag Agarwal

Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? (2024.acl-long)

Challenge:	Existing research has focused on extracting memorized content from LLMs or developing memorization metrics without exploring the underlying architectural factors that contribute to memorizing.
Approach:	They analyze how attention modules at different layers impact its memorization and generalization performance by using attribution techniques.
Outcome:	The proposed model can be used to mitigate memorization while keeping other components like layer normalization and MLP transformations intact.

Copied to clipboard

Nishant Balepur, Abhilasha Ravichander, Rachel Rudinger

A Multi-Perspective Analysis of Memorization in Large Language Models (2024.emnlp-main)

Challenge:	Multiple-choice question answering (MCQA) is often used to evaluate large language models . a recent study found that LLMs perform MCQA with choices-only prompts .
Approach:	They investigate whether LLMs can perform multiple-choice question answering (MCQA) with choices-only prompts . they find no evidence that the choices- only accuracy stems from memorization alone .
Outcome:	The results show that LLMs perform MCQA with choices-only prompts with 0.33 accuracy gain.

Copied to clipboard

Bowen Chen, Namgi Han, Yusuke Miyao

Powerful Training-Free Membership Inference Against Fine-Tuned Autoregressive Language Models (2026.acl-long)

Challenge:	Large Language Models (LLMs) can generate the same sequences contained in the pre-train corpus, known as memorization.
Approach:	They analyze the relationship between memorization and outputs from Large Language Models (LLMs) they show a sudden drop and increase in the frequency of input tokens when generating memorized/unmemorized sequences .
Outcome:	The proposed model can generate the same sequences contained in the pre-train corpus, and it can predict unmemorized tokens.

Copied to clipboard

David Ilić, David Stanojević, Kostadin Cvejoski

Controllable Memorization in LLMs via Weight Pruning (2025.emnlp-main)

Challenge:	Existing methods for auditing fine-tuned language models have limited detection rates . membership inference attacks aim to determine if a specific record was in a model's training set .
Approach:	They propose a membership inference attack that exploits memorization at error positions . EZ-MIA achieves 3.8 higher detection than previous state-of-the-art .
Outcome:	The proposed attack achieves 3.8 higher detection than previous state-of-the-art models . EZ-MIA achieves 8 higher detectability than prior work, requiring no model training .

Copied to clipboard

Chenjie Ni, Zhepeng Wang, Runxue Bao, Shangqian Gao, Yanfu Zhang

Challenge:	Existing studies have focused on mitigating memorization, but the deliberate control of memorisation has been underexplored.
Approach:	They propose a gradient-based weight pruning framework to control memorization rates in large language models by fine-grained control over pruning parameters.
Outcome:	The proposed framework enables models to suppress or enhance memorization based on application-specific requirements.

Enhancing Emotion Recognition in Conversation via Multi-view Feature Alignment and Memorization (2023.findings-emnlp)

Copied to clipboard

Guiyang Hou, Yongliang Shen, Wenqi Zhang, Wei Xue, Weiming Lu

Multimodality

Challenge:	Emotion recognition in conversation (ERC) is an advanced capability of conversational AI systems.
Approach:	They propose a semi-parametric paradigm for Emotion Recognition in conversation that uses supervised contrastive learning to align semantic-view and context-view features.
Outcome:	The proposed model achieves state-of-the-art on four widely used benchmarks.

Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards (2026.acl-long)

Copied to clipboard

Youliang Yuan, Qiuyang Mang, Jingbang Chen, Hong Wan, Xiaoyuan Liu, Junjielong Xu, Jen-tse Huang, Wenxuan Wang, Wenxiang Jiao, Pinjia He

Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets (2024.findings-emnlp)

Challenge:	Existing models are susceptible to reward hacking, leading to a substantial overestimation of a model's reasoning ability.
Approach:	They propose a Rubric Reward Model that rewards the entire reasoning trajectory against problem-specific rubrics.
Outcome:	The proposed model outperforms outcome-only supervision on four math benchmarks and boosts Verified Pass@1024 from 26.7% to 62.6% and reduces the incidence of Miracle Steps by 71%.

Copied to clipboard

Melanie Walsh, Maria Antoniak, Anna Preus

Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities (2025.emnlp-main)

Challenge:	a task evaluates how well LLMs recognize poetry, but performance varies by poetic form . performance varying by poetic forms; models struggle to identify unfixed poetic forms .
Approach:	They use a benchmark dataset to evaluate how well LLMs recognize poetry . they find that the models can identify fixed poetic forms with high accuracy .
Outcome:	The proposed task evaluates how well LLMs recognize poetry features . performance varies significantly by poetic form; models struggle to identify unfixed forms . authors urge more work that builds nuance and ambiguity into humanistic benchmarks .

Copied to clipboard

Xiaoyu Luo, Yiyi Chen, Johannes Bjerva, Qiongxiu Li

Multilinguality

Challenge:	Using multilingual models, we find that treating languages in isolation obscures the true patterns of memorization.
Approach:	They propose a graph-based correlation metric that incorporates language similarity to analyze cross-lingual memorization.
Outcome:	The proposed model incorporates language similarity to analyze cross-lingual memorization in 95 languages.

LLM-Symbolic Integration for Robust Temporal Tabular Reasoning (2025.findings-acl)

Copied to clipboard

Atharv Kulkarni, Kushagra Dixit, Vivek Srikumar, Dan Roth, Vivek Gupta

Memorization or Reasoning? Exploring the Idiom Understanding of LLMs (2025.emnlp-main)

Challenge:	Existing methods for temporal tabular question answering are inconsistent and fail to provide the variability needed to thoroughly evaluate models.
Approach:	TEMPTABQA-C uses a synthetic dataset and symbolic representation to generate and execute SQL queries.
Outcome:	TEMPTABQA-C improves on previous methods for temporal tabular question answering . incorporating adaptive fewshot prompting with tailored examples improves performance . lack of robustness, scalability, and interpretable solutions is key obstacle .

Copied to clipboard

Jisu Kim, Youngwoo Shin, Uiji Hwang, Jihun Choi, Richeng Xuan, Taeuk Kim

Task-Stratified Knowledge Scaling Laws for Post-Training Quantized Large Language Models (2026.findings-acl)

Challenge:	idioms have long posed a challenge due to their unique linguistic properties, which set them apart from other common expressions.
Approach:	They propose to use a large-scale dataset of idioms in six languages to evaluate LLMs' idiomatic processing ability.
Outcome:	The proposed model integrates contextual cues and reasoning to improve idiom understanding in LLMs, suggesting that their performance is influenced by memorization and reasoning.

Copied to clipboard

Chenxi Zhou, Pengfei Cao, Jiang Li, Bohan Yu, Jinyu Ye, Jun Zhao, Kang Liu

Interpretable Mnemonic Generation for Kanji Learning via Expectation-Maximization (2025.emnlp-main)

Challenge:	Existing scaling laws focus on general performance, overlooking crucial fine-grained factors and how quantization differentially impacts diverse knowledge capabilities.
Approach:	They propose a framework that unifies model size, bit-width, and fine-grained factors into memorization, application, and reasoning.
Outcome:	The proposed framework shows strong fit and cross-architecture consistency on 293 different PTQ configurations.

Copied to clipboard

Jaewook Lee, Alexander Scarlatos, Andrew Lan

Explainability & Interpretability in NLP

Challenge:	Existing methods for mnemonic generation in Japanese are limited in their interpretability due to script differences.
Approach:	They propose a method that models the mnemonic construction process as driven by common rules.
Outcome:	The proposed method performs well in the cold-start setting for new learners while providing insight into the mechanisms behind effective mnemonic creation.

Exposing the Achilles’ Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning (2025.acl-long)

Copied to clipboard

Joykirat Singh, Akshay Nambi, Vibhav Vineet

Memorizing is Not Enough: Deep Knowledge Injection Through Reasoning (2025.acl-long)

Challenge:	Existing evaluations focus on final accuracy, neglecting the critical aspect of reasoning capabilities.
Approach:	They propose to evaluate LLMs’ abilities to detect and correct reasoning mistakes by using rule-based methods and smaller language models.
Outcome:	The proposed model outperforms existing models such as GPT-4o and GPT4 in both accuracy and accuracy, but lacks data contamination and memorization concerns.

Copied to clipboard

Ruoxi Xu, Yunjie Ji, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Ben He, Yingfei Sun, Xiangang Li, Le Sun

Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision (2025.emnlp-main)

Challenge:	Existing knowledge injection frameworks focus on knowledge memorization and retrieval, but static nature of large language models leads to outdated information as the real world evolves or when adapting to domain-specific knowledge.
Approach:	They propose a four-tier knowledge injection framework that defines the levels of knowledge injection: memorization, retrieval, reasoning, and association.
Outcome:	The proposed framework defines the levels of knowledge injection: memorization, retrieval, reasoning, and association.

Copied to clipboard

Xingwei Tan, Marco Valentino, Mahmud Elahi Akhter, Maria Liakata, Nikolaos Aletras