Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using the counterfactual memorisation metric, we find that when training neural networks, models will memorise some inputs but not others. |
| Approach: | They use the counterfactual memorisation metric to build a resource that places 5M NMT datapoints on a memorisations-generalisation map and describe how the datapoint’s surface-level characteristics and a models’ per-datum training signals are predictive of memorising in NMT. |
| Outcome: | The proposed model places 5M NMT datapoints on a memorisation-generalisation map and shows how their surface-level characteristics and models’ per-datum training signals are predictive of memorising in NMT. |
Similar Papers
Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Memorisation in neural models is concerned due to overfitting and privacy concerns . a dominant hypothesis based on image classification is that lower layers learn generalisable features and deeper layers specialise and memorise. |
| Approach: | They apply 4 techniques to localise and edit models' memories. |
| Outcome: | The proposed method shows that memorisation is a gradual process rather than a localised one. |
On Compositional Generalization of Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Modern neural machine translation models have shown competitive performance in benchmarks such as WMT, but there are significant issues such as robustness, domain generalization, etc. |
| Approach: | They propose a benchmark dataset for NMT models from the perspective of compositional generalization and quantitatively analyze the results. |
| Outcome: | The proposed model performs well under traditional metrics, but is low in out-of-domain and low-resource conditions. |
Domain adapted machine translation: What does catastrophic forgetting forget and why? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) models can be specialized by domain adaptation, often fine-tuning on a dataset of interest. |
| Approach: | They propose a novel approach to understanding catastrophic forgetting during NMT adaptation by investigating the relationship between the data and the in-domain vocabulary coverage. |
| Outcome: | The proposed model can be specialized by fine-tuning on a domain of interest, but can fail to achieve the predicted quality of the target domain. |
Neuron-Level Differentiation of Memorization and Generalization in Large Language Models (2025.emnlp-main)
Copied to clipboard
Ko-Wei Huang, Yi-Fu Fu, Ching-Yu Tsai, Yu-Chieh Tu, Tzu-ling Cheng, Cheng-Yu Lin, Yi-Ting Yang, Heng-Yi Liu, Keng-Te Liao, Da-Cheng Juan, Shou-De Lin
| Challenge: | Existing models exhibit memorization and generalization behaviors in ways that are not easily interpretable or controllable. |
| Approach: | They propose to use a GPT-2 and LLaMA-3.2 model to identify distinct neuron subsets responsible for each behavior to steer the model toward memorization or generalization. |
| Outcome: | The proposed models show that inference-time interventions on these neurons can steer the model’s behavior toward memorization or generalization. |
Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies have revealed a number of pathologies of neural machine translation systems. |
| Approach: | They propose to use maximum a posteriori decoding to identify the highest-scoring translation, i.e. the mode problem, to validate the model and its training algorithm. |
| Outcome: | The proposed model reproduces the statistical data well, but the beam search strays from the statistics. |
Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Memorization presents a challenge for constrained Natural Language Generation tasks . previous studies focused on counterfactual memorization, linking it to hallucinations . |
| Approach: | They propose an algorithm for extractive memorization in constrained sequence generation tasks . they propose to elicit non-memorized translations of memorized samples from the same model . |
| Outcome: | The proposed algorithm could be leveraged to mitigate memorization in the model through finetuning. |
Memorisation versus Generalisation in Pre-trained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art pre-trained language models have been shown to memorise facts and perform well with limited amounts of training data. |
| Approach: | They propose to extend pre-trained language models to generalise and memorise facts in noisy and low-resource scenarios. |
| Outcome: | The proposed extension improves performance in low-resource named entity recognition tasks. |
Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Prior work treats all types of mismatches between source and target as noise . Consequently, it remains unclear how noisy parallel training samples impact NMT training. |
| Approach: | They propose a divergent-aware NMT framework that uses factors to help NMT recover from the degradation caused by naturally occurring divergences. |
| Outcome: | The proposed framework improves translation quality and model calibration on EN-FR tasks. |
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Neural machine translation models suffer from catastrophic forgetting during continual training . models tend to overfit to frequent observations in the in-domain data but forget previously learned knowledge. |
| Approach: | They investigated the causes of catastrophic forgetting in NMT models by examining their parameters and modules. |
| Outcome: | The proposed model forgets previously learned knowledge and swings to fit new data . the results show that some parameters are important for both the general-domain and in-domain translation and the great change of them during continual training brings about the performance decline in general- domain. |
Categorizing Semantic Representations for Neural Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | Modern neural machine translation models suffer limitation in compositional generalization, resulting in weakened translation performance on unseen compounds. |
| Approach: | They propose to introduce categorization to the contextualized representations to improve generalization by reducing sparsity and overfitting. |
| Outcome: | The proposed method reduces compositional generalization error rates by 24% on a dedicated MT dataset. |