From Monolingual to Multilingual FAQ Assistant using Multilingual Co-training (D19-61)
Copied to clipboard
Mayur Patidar, Surabhi Kumari, Manasi Patwardhan, Shirish Karande, Puneet Agarwal, Lovekesh Vig, Gautam Shroff
| Challenge: | Recent research on cross-lingual transfer shows state-of-the-art results on benchmark datasets using pre-trained language representation models like BERT. |
| Approach: | They propose a method to augment an annotated dataset with machine translations in target languages and fine-tune the PLRM jointly. |
| Outcome: | The proposed approach provides consistent gains on multiple benchmark datasets while requiring a single model for multiple languages. |
Similar Papers
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
How Multilingual is Multilingual BERT? (P19-1)
Copied to clipboard
| Challenge: | Existing studies have shown that deep, contextualized language models can encode syntactic and named entity information, but they have focused on what models trained on English capture about English. |
| Approach: | They propose a multilingual model pre-trained from monolingual Wikipedia corpora . they show that multilingual BERT is surprisingly good at zero-shot cross-lingual model transfer . |
| Outcome: | The proposed model can find translation pairs, but it exhibits systematic deficiencies affecting certain language pairs. |
Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models that pretrain for cross-lingual tasks do not improve cross-linguistic learning. |
| Approach: | They propose to employ machine translation as a continued training objective to enhance language representation learning by bridging multilingual pretraining and cross-lingual applications. |
| Outcome: | The proposed model performance is compared with existing models and their latent representations. |
On the Cross-lingual Transferability of Monolingual Representations (2020.acl-main)
Copied to clipboard
| Challenge: | State-of-the-art unsupervised multilingual models generalize in zero-shot cross-lingual setting . generalization ability attributed to shared subword vocabulary and joint training across multiple languages . |
| Approach: | They propose an approach that transfers a monolingual model to new languages at the lexical level. |
| Outcome: | The proposed approach is competitive with multilingual BERT on cross-lingual classification benchmarks and on a new cross-linguistic question answering dataset. |
Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)
Copied to clipboard
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks. |
| Approach: | They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains . |
| Outcome: | The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks. |
Cross-lingual Transfer of Monolingual Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual learning using multilingual models cast doubt on shared vocabulary and joint pre-training . et al. (2005) show that model knowledge learned in the source language enhances the learning of the target language independently of language proximity. |
| Approach: | They propose a method for transferring monolingual models to other languages through continuous pre-training and investigate their results in English. |
| Outcome: | The proposed method outperforms a model trained from scratch in the GLUE benchmark for English . it shows that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in english. |
Multilingual BERT Post-Pretraining Alignment (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work improves on the success of monolingual pretrained language models by adding cross-lingual tasks that always involve English. |
| Approach: | They propose a method to align multilingual contextual embeddings as a post-pretraining step for improved cross-lingual transferability of pretrained language models. |
| Outcome: | The proposed model outperforms XLM-R_Base on translation-train tasks while using less parallel data and fewer parameters. |
The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining (2026.acl-long)
Copied to clipboard
Jiandong Shao, Raphael Tang, Crystina Zhang, Karin Sevegnani, Pontus Stenetorp, Jianfei Yang, Yao Lu
| Challenge: | Existing research suggests that multilingual large language models can achieve impressive cross-lingual understanding despite largely monolingual pretraining. |
| Approach: | They compare a monolingual-only corpus with a standard web corpus that removes all multilingual documents and then retrain the models from scratch under controlled conditions. |
| Outcome: | The results show that removing bilingual data causes translation performance to drop 56% in BLEU, whereas code-switching contributes minimally. |
T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual text classification leverage text classifiers trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning. |
| Approach: | They propose to combine a neural machine translator and a text classifier trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning. |
| Outcome: | The proposed approach significantly improves over a baseline approach. |
Multilingual Translation from Denoising Pre-Training (2021.findings-acl)
Copied to clipboard
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, Angela Fan
| Challenge: | Recent work shows potential of training one model for multilingual machine translation . but little has been explored on the potential to combine denoising pretraining with multilingual translation in a single model. |
| Approach: | They propose to combine denoising pretraining with multilingual machine translation in a single model. |
| Outcome: | The proposed model improves over models trained from scratch and bilingually for translation into English. |