| Challenge: | a recent study shows that large language models perform well in low-resource languages . a vast majority of languages don't have comparable data as compared to English . |
| Approach: | They propose to use Translationese as synthetic data for pre-training language models for low-resource languages. |
| Outcome: | The proposed method reduces performance of LMs trained on clean data in Indian languages . the proposed model performs better in English than in other languages, but is not comparable to English. |
Similar Papers
Not Enough Data to Pre-train Your Language Model? MT to the Rescue! (2023.findings-acl)
Copied to clipboard
| Challenge: | In recent years, transformer-based language models (LMs) have become the default approach for many NLP tasks. |
| Approach: | They compare the performance of transformer-based language models with machine-translated corpora. |
| Outcome: | The proposed model can be improved with real data, but further research is needed. |
Synthetic Pre-Training Tasks for Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | toxicity and bias can be addressed by pre-training with synthetic resources . BLEU scores are used to compare methods with real-world data . |
| Approach: | They propose several ways to generate obfuscated data from large parallel corpus and concatenating phrase pairs from small word-aligned corpus with synthetic parallel data without real human language corpora. |
| Outcome: | The proposed methods can be used to generate obfuscated data or synthetic parallel data without real human language corpora even with high levels of oblication. |
Leveraging Synthetic Targets for Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Using synthetic target data, training models on synthetic targets outperforms training on actual ground-truth data. |
| Approach: | They propose a recipe for training machine translation models on synthetic target data by leveraging a large pre-trained model. |
| Outcome: | The proposed model outperforms training on real-world translation datasets. |
How Can Synthetic Data Improve Multilingual Language Model Pretraining? A Data Quality Perspective (2026.acl-long)
Copied to clipboard
| Challenge: | Low-resource languages are a long-tail problem for multilingual LLMs due to limited high-quality training data. |
| Approach: | They propose a method that translates high-quality, knowledge-rich English data into low-resource languages . they propose SynRank, which leverages synthetic data as positive samples to train a classifier . |
| Outcome: | The proposed method matches handcrafted rule-based filtering by human experts and significantly improves knowledge-intensive tasks with less data. |
A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation (2022.naacl-main)
Copied to clipboard
David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anuoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Derguene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, Sam Manthalu
| Challenge: | Low-resource languages are left out of large-scale pretraining datasets . authors explore how to leverage existing pre-trained models to create low-resourced translation systems for 16 African languages. |
| Approach: | They investigate how large-scale pre-trained models can be used to create low-resource translation systems for 16 African languages. |
| Outcome: | The proposed models can translate between hundreds of languages even though there is little parallel data available for training. |
Pre-training via Leveraging Assisting Languages for Neural Machine Translation (2020.acl-srw)
Copied to clipboard
| Challenge: | Sequence-to-sequence (S2S) pre-training with large monolingual data is not always available for the languages of interest (LOI). |
| Approach: | They propose to use monolingual corpora of other languages to complement the scarce monolingual LOI by script mapping (Chinese to Japanese) . Using only Chinese and French monolinguals, they improve Japanese-English translation quality by up to 8.5 BLEU in low-resource scenarios. |
| Outcome: | The proposed approach improves Japanese-English translation quality by up to 8.5 BLEU in low-resource scenarios. |
Disentangling Pretrained Representation to Leverage Low-Resource Languages in Multilingual Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Multilingual neural machine translation requires an enormous dataset, leaving the low-resource language (LRL) underdeveloped. |
| Approach: | They evaluated five languages using a parallel corpus of 1,000 instances each and found a zero-shot improvement of 7.4 from the baseline score of 7.1 to a score of 15.5 at best. |
| Outcome: | The proposed model improves performance in the linguistically diverse country of Indonesia by 7.4 from baseline score of 7.1 to 15.5 at best. |
Multilingual Data Filtering using Synthetic Data from Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that effective filters can be created by utilising Large Language Models to synthetically label data, which is then used to train smaller neural models for filtering purposes. |
| Approach: | They extend this approach to languages beyond English to train neural models for filtering purposes. |
| Outcome: | The proposed approach is effective at filtering parallel text for translation quality and filtering for domain specificity. |
Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection (2023.findings-eacl)
Copied to clipboard
| Challenge: | AfriBERTa shows that training transformer models from scratch on 1GB of data from many unrelated African languages outperforms massively multilingual models on downstream NLP tasks. |
| Approach: | They propose that training on smaller amounts of data but from related languages could match the performance of models trained on large, unrelated data. |
| Outcome: | The proposed model outperforms models trained on large, unrelated datasets on downstream NLP tasks. |
Multilingual Denoising Pre-training for Neural Machine Translation (2020.tacl-1)
Copied to clipboard
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer
| Challenge: | Existing approaches to pre-train models focus on only English corpora, but this is not common in machine translation. |
| Approach: | They propose a sequence-to-sequence denoising auto-encoder pre-trained on monolingual corpora . they show that it produces significant performance gains across MT tasks . |
| Outcome: | The proposed model can achieve significant performance gains across a wide variety of MT tasks. |