How Transferable are Attribute Controllers on Pretrained Multilingual Translation Models? (2024.eacl-long)
Copied to clipboard
| Challenge: | Pretrained multilingual translation models with massive coverage are becoming of the backbone of many translation systems. |
| Approach: | They propose to use a gradient-based inference-time controller to control a pretrained multilingual model by using a model with attribute annotations. |
| Outcome: | The proposed model performs well on pretrained multilingual models and is attribute- rather than language-specific. |
Similar Papers
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on zero-shot cross-lingual transfer . mBERT, mBART and mT5 provide high-quality representations for texts in various languages . |
| Approach: | They propose to use mBART and NLLB-200 to finetune a multilingual pretrained language model on input-output pairs in one language and use it to make task predictions for inputs in other languages. |
| Outcome: | The proposed approach significantly reduces generation in the wrong language with full finetuning and can be competitive in some cases. |
Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained multilingual models enable zero-shot learning even for unseen languages . current multilingual model covers only a small subset of the world's languages - due to data sparsity, they are not likely to obtain good results for many lowresource languages. |
| Approach: | They ask: how does the number of pretraining languages influence zero-shot learning for unseen languages? do the findings change if the languages used for pretraining are all related? |
| Outcome: | The results show that pretrained models can zero-shot learn for unseen languages even for limited amounts even for low-resource languages. |
T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual text classification leverage text classifiers trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning. |
| Approach: | They propose to combine a neural machine translator and a text classifier trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning. |
| Outcome: | The proposed approach significantly improves over a baseline approach. |
A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation (2022.naacl-main)
Copied to clipboard
David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anuoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Derguene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, Sam Manthalu
| Challenge: | Low-resource languages are left out of large-scale pretraining datasets . authors explore how to leverage existing pre-trained models to create low-resourced translation systems for 16 African languages. |
| Approach: | They investigate how large-scale pre-trained models can be used to create low-resource translation systems for 16 African languages. |
| Outcome: | The proposed models can translate between hundreds of languages even though there is little parallel data available for training. |
Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection (2023.findings-eacl)
Copied to clipboard
| Challenge: | AfriBERTa shows that training transformer models from scratch on 1GB of data from many unrelated African languages outperforms massively multilingual models on downstream NLP tasks. |
| Approach: | They propose that training on smaller amounts of data but from related languages could match the performance of models trained on large, unrelated data. |
| Outcome: | The proposed model outperforms models trained on large, unrelated datasets on downstream NLP tasks. |
mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for improving multilingual models only use entity information in pretraining and do not explicitly use entities in downstream tasks. |
| Approach: | They propose to leverage Wikipedia entity representations for downstream tasks . they train a multilingual language model with 24 languages with entity representation . |
| Outcome: | The proposed model outperforms word-based models in cross-lingual transfer tasks. |
Multilingual Translation from Denoising Pre-Training (2021.findings-acl)
Copied to clipboard
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, Angela Fan
| Challenge: | Recent work shows potential of training one model for multilingual machine translation . but little has been explored on the potential to combine denoising pretraining with multilingual translation in a single model. |
| Approach: | They propose to combine denoising pretraining with multilingual machine translation in a single model. |
| Outcome: | The proposed model improves over models trained from scratch and bilingually for translation into English. |
Disentangling Pretrained Representation to Leverage Low-Resource Languages in Multilingual Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Multilingual neural machine translation requires an enormous dataset, leaving the low-resource language (LRL) underdeveloped. |
| Approach: | They evaluated five languages using a parallel corpus of 1,000 instances each and found a zero-shot improvement of 7.4 from the baseline score of 7.1 to a score of 15.5 at best. |
| Outcome: | The proposed model improves performance in the linguistically diverse country of Indonesia by 7.4 from baseline score of 7.1 to 15.5 at best. |
Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that results from high-resource languages cannot be easily transferred to realistic, low-resourced scenarios. |
| Approach: | They analyse performance of multilingual transformer models using available resources for Hausa, isiXhosa and NER and topic classification. |
| Outcome: | The proposed models can achieve with as little as 10 or 100 labeled sentences the same performance as baselines with much more supervised training data. |