| Challenge: | Multilingual pre-trained language models have demonstrated impressive (zero-shot) cross-lingual transfer abilities, however, their performance is hindered when the target language has distant typology from the source language or when pre-training data is limited in size. |
| Approach: | They propose a method that contextually retrieves prompts as flexible guidance for encoding instances conditionally. |
| Outcome: | The proposed method improves on the XTREME task and also for low-resource languages in unsupervised sentence retrieval. |
Similar Papers
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2022.acl-long)
Copied to clipboard
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, Furu Wei
| Challenge: | ELECTRA-style tasks are used to pretrain cross-lingual models for NLP tasks . masked language modeling tasks require massive computation resources, rendering such models quite expensive . |
| Approach: | They propose to use ELECTRA-style tasks to pre-train a cross-lingual language model . they propose to pretrain the model on multilingual and parallel corpora . |
| Outcome: | The proposed model outperforms baseline models on cross-lingual understanding tasks with much less computation cost. |
XLM-D: Decorate Cross-lingual Pre-training Model as Non-Autoregressive Neural Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-training language models have been successful in natural language understanding and autoregressive generation tasks, but non-autoregressive models have not been sufficiently successful. |
| Approach: | They propose a pre-trained masked language model (MLM) and a non-autoregressive generation model with a lightweight decorator. |
| Outcome: | The proposed model outperforms the previous mask-predict model on translation datasets by 19.9x. |
Multi-Granularity Contrasting for Cross-Lingual Pre-Training (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to pre-training focus on embedding alignment, but they neglect the modeling of bidirectional contexts. |
| Approach: | They propose a framework to learn languageuniversal representations using multi-granularity contrasting framework . they encode semantic equivalents from different languages into similar representations . |
| Outcome: | The proposed framework can achieve significant performance gains in machine translation and cross-lingual language understanding. |
Multilingual Pre-training with Self-supervision from Global Co-occurrence Information (2023.findings-acl)
Copied to clipboard
| Challenge: | Empirical studies show multilinguality and crosslinguality emerge from MLM pretraining without supervision. |
| Approach: | They propose to use global co-occurrence information as a source of structural information on multilingual corpora. |
| Outcome: | Empirical studies show that MLM-GC pre-training outperforms MLM pre- training for 4 downstream cross-lingual tasks and 1 additional monolingual task. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |
Large Language Models are good multi-lingual learners : When LLMs meet cross-lingual prompts (2025.coling-main)
Copied to clipboard
| Challenge: | Experimental results show that Large Language Models can generate rule-based data in long contexts without following all specified rules. |
| Approach: | They propose a novel prompting strategy Multi-Lingual Prompt which automatically translates the error-prone rule that an LLM struggles to follow into another language, thus drawing greater attention to it. |
| Outcome: | The proposed framework outperforms state-of-the-art prompting methods on public datasets across various tasks, with a specific case study in text-to-MIP instances. |
T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual text classification leverage text classifiers trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning. |
| Approach: | They propose to combine a neural machine translator and a text classifier trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning. |
| Outcome: | The proposed approach significantly improves over a baseline approach. |
Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual natural language inference lack annotated parallel corpora. |
| Approach: | They propose a new prompt learning framework with the Multilingual Verbalizer for XNLI that uses a multilingual verbalizer to align the representations of original and augmented multilingual questions into a unified semantic space with consistency regularization. |
| Outcome: | The proposed framework outperforms existing methods under few-shot and full-shot cross-lingual transfer settings. |
Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on multilingual models have focused on their cross-lingual transfer behavior . a recent study examined multilingual model learning from the multilingual pretraining signal . |
| Approach: | They analyze checkpoints during multilingual pretraining to identify when models acquire in-language and cross-lingual abilities. |
| Outcome: | The proposed model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones. |
Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)
Copied to clipboard
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks. |
| Approach: | They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains . |
| Outcome: | The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks. |