Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | a lack of data in low-resource languages has limited the performance of a multilingual pre-trained model. |
| Approach: | They propose a continuous pre-training framework to adapt mBART to unseen languages . they construct noisy mixed-language text from the monolingual corpus of the target language . |
| Outcome: | The proposed framework improves finetuning performance on low-resource translation pairs . the proposed framework also improves on translation pairs where both languages are seen . |
Similar Papers
Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation? (2022.findings-acl)
Copied to clipboard
En-Shiun Lee, Sarubi Thillainathan, Shravan Nayak, Surangika Ranathunga, David Adelani, Ruisi Su, Arya McCarthy
| Challenge: | Pre-trained multilingual sequence-to-sequence models like mBART and mT5 can be used to translate low-resource languages, but their practical application is unclear. |
| Approach: | They conduct an empirical experiment in 10 languages to determine what can pre-trained multilingual sequence-to-sequence models like mBART do to translate low-resource languages? |
| Outcome: | The proposed models are robust to domain differences, but translations for unseen and typologically distant languages remain below 3.0 BLEU. |
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence, but training them from scratch is prohibitively expensive. |
| Approach: | They propose to continuously pre-train LLMs from existing pre-trained LLM models by using a set of parameters instead of randomly initializing them. |
| Outcome: | The proposed approach saves significant resources and accelerates convergence and performance. |
Multilingual Neural Machine Translation (2020.coling-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we will cover the latest advances in NMT to enhance low-resource translation. |
| Approach: | They will cover the latest advances in NMT approaches that leverage multilingualism . they will focus on topics such as language divergence, transfer learning and pivoting . |
| Outcome: | This tutorial will cover the latest advances in NMT to enhance low-resource translation models. |
Disentangling Pretrained Representation to Leverage Low-Resource Languages in Multilingual Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Multilingual neural machine translation requires an enormous dataset, leaving the low-resource language (LRL) underdeveloped. |
| Approach: | They evaluated five languages using a parallel corpus of 1,000 instances each and found a zero-shot improvement of 7.4 from the baseline score of 7.1 to a score of 15.5 at best. |
| Outcome: | The proposed model improves performance in the linguistically diverse country of Indonesia by 7.4 from baseline score of 7.1 to 15.5 at best. |
Multilingual Denoising Pre-training for Neural Machine Translation (2020.tacl-1)
Copied to clipboard
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer
| Challenge: | Existing approaches to pre-train models focus on only English corpora, but this is not common in machine translation. |
| Approach: | They propose a sequence-to-sequence denoising auto-encoder pre-trained on monolingual corpora . they show that it produces significant performance gains across MT tasks . |
| Outcome: | The proposed model can achieve significant performance gains across a wide variety of MT tasks. |
Efficient Continual Pre-training of LLMs for Low-resource Languages (2025.naacl-industry)
Copied to clipboard
| Challenge: | Open-source large language models (LLMs) are a promising tool for low-resource languages . however, there is still a substantial performance gap between high-resourced languages and LRLs . |
| Approach: | They develop an algorithm to select a subset of texts from a larger corpus and use it to select tokens for LLMs. |
| Outcome: | The proposed algorithm reduces the cost of continual pre-training (CPT) with large amounts of language-specific data. |
Towards Effective and Efficient Continual Pre-training of Large Language Models (2025.acl-long)
Copied to clipboard
Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Xin Zhao, Zhicheng Dou, Jiaxin Mao, Yankai Lin, Ruihua Song, Jun Xu, Xu Chen, Rui Yan, Zhewei Wei, Di Hu, Wenbing Huang, Ji-Rong Wen
| Challenge: | Continual pre-training (CPT) has been an important approach for adapting language models to specific domains or tasks. |
| Approach: | They propose a Continual pre-training method that can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
| Outcome: | The proposed method can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
Continued Pretraining and Interpretability-Based Evaluation for Low-Resource Languages: A Galician Case Study (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to remarkable improvements in language understanding and text generation. |
| Approach: | They propose a framework to evaluate large language models for underrepresented languages . they examine CPT strategies for languages with limited representation in multilingual models . |
| Outcome: | The proposed evaluation framework is based on the case of Galician language . it assesses trade-offs between linguistic enrichment and task-solving capabilities . |
Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages (D19-1)
Copied to clipboard
| Challenge: | Using parallel corpora, we train a single, direct NMT model for non-English language pairs. |
| Approach: | They propose three ways to increase the relation among source, pivot, and target languages in pre-training . they use additional adapter component to smoothly connect pre-trained encoder and decoder . |
| Outcome: | The proposed methods outperform multilingual models up to +2.6% BLEU in WMT 2019 French-German and German-Czech tasks. |
Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation (2020.acl-main)
Copied to clipboard
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, Yonghui Wu
| Challenge: | Existing multilingual NMT approaches do not utilize the abundance of monolingual data, especially in low-resource languages. |
| Approach: | They propose to combine monolingual data with self-supervision to pre-train translation models and fine-tune on small amounts of supervised data. |
| Outcome: | The proposed approach improves translation quality of low-resource languages and zero-shot translation quality. |