| Challenge: | Neural machine translation models are data-driven and require large-scale training corpus . continual learning remains a big challenge for artificial intelligence systems and models . |
| Approach: | They propose a continual learning framework for NMT models that incorporates multiple stages of training to alleviate catastrophic forgetting problem. |
| Outcome: | The proposed framework achieves superior performance compared to baseline models in all settings. |
Similar Papers
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Neural machine translation models suffer from catastrophic forgetting during continual training . models tend to overfit to frequent observations in the in-domain data but forget previously learned knowledge. |
| Approach: | They investigated the causes of catastrophic forgetting in NMT models by examining their parameters and modules. |
| Outcome: | The proposed model forgets previously learned knowledge and swings to fit new data . the results show that some parameters are important for both the general-domain and in-domain translation and the great change of them during continual training brings about the performance decline in general- domain. |
Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions (2022.emnlp-main)
Copied to clipboard
| Challenge: | Currently, continuous learning methods suffer from catastrophic forgetting problem, causing model to forget previous knowledge while learning new knowledge. |
| Approach: | They propose a two-stage continuous learning method based on local features of the real loss to avoid catastrophic forgetting problem. |
| Outcome: | The proposed method achieves significant improvements on domain adaptation and more challenging language adaptation tasks. |
Continual Knowledge Distillation for Neural Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Current parallel corpora are not publicly accessible but trained models are more readily available. |
| Approach: | They propose a method to take advantage of existing translation models to improve one model of interest. |
| Outcome: | The proposed method improves on Chinese-English and German-English datasets and is robust to malicious models. |
ICL: Iterative Continual Learning for Multi-domain Neural Machine Translation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on learning domain knowledge from multiple domains, but task-specific parameters hinder mutual transfer of knowledge between new domains. |
| Approach: | They propose an iterative Continual learning framework for multi-domain neural machine translation that leverages previously acquired domain knowledge. |
| Outcome: | The proposed model outperforms baseline models on UM-Corpus and OPUS datasets. |
Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Neural networks tend to gradually forget the previously learned knowledge when learning multiple tasks sequentially from dynamic data distributions. |
| Approach: | They propose a method that iteratively provides complementary knowledge to student models by dynamically updating teacher models trained on specific data orders. |
| Outcome: | The proposed method improves on multiple machine translation tasks and improves performance over baseline systems. |
Iterative Dual Domain Adaptation for Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Empirical results on Chinese-English and English-German translation tasks demonstrate the effectiveness of our proposed framework. |
| Approach: | They propose an iterative dual domain adaptation framework for neural machine translation that uses multiple corpora to perform bidirectional translation knowledge transfer. |
| Outcome: | Empirical results on Chinese-English and English-German translation tasks demonstrate the effectiveness of the proposed framework. |
Continual Learning with Semi-supervised Contrastive Distillation for Incremental Neural Machine Translation (2024.acl-long)
Copied to clipboard
| Challenge: | Multi-domain learning is a good solution for solving domain tasks but it requires retraining when adding a new domain. |
| Approach: | They propose to exploit unlabeled data from the same distributions of the older domains to avoid catastrophic forgetting. |
| Outcome: | The proposed framework exploits unlabeled data from the same distributions of the older domains to avoid catastrophic forgetting. |
A Survey of Domain Adaptation for Neural Machine Translation (C18-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) is a deep learning based approach for machine translation. |
| Approach: | They propose to use a deep learning approach to train machine translation in scenarios where large-scale parallel corpora are available. |
| Outcome: | The proposed approach yields the state-of-the-art translation performance in resource rich scenarios. |
Self-Paced Learning for Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that the training of neural machine translation (NMT) rely on the quality of artificial schedule drawn up with the handcrafted features, e.g. sentence length or word rarity. |
| Approach: | They propose to train NMT model using a self-paced learning approach that allows it to quantify the learning confidence over training examples and flexibly govern its learning via regulating the loss in each iteration step. |
| Outcome: | The proposed model outperforms baseline models and those trained with human-designed curricula on translation quality and convergence speed. |
Domain adapted machine translation: What does catastrophic forgetting forget and why? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) models can be specialized by domain adaptation, often fine-tuning on a dataset of interest. |
| Approach: | They propose a novel approach to understanding catastrophic forgetting during NMT adaptation by investigating the relationship between the data and the in-domain vocabulary coverage. |
| Outcome: | The proposed model can be specialized by fine-tuning on a domain of interest, but can fail to achieve the predicted quality of the target domain. |