XNLI

From Monolingual to Multilingual FAQ Assistant using Multilingual Co-training (D19-61)

Challenge:	Recent studies have focused on zero-shot cross-lingual transfer of pretrained languages.
Approach:	They propose to use few-shot cross-lingual transfer to improve zero-shot performance of multilingual pretrained language models.
Outcome:	The proposed model can be scaled to high-quality samples and improves on zero-shot performance.

Copied to clipboard

Mayur Patidar, Surabhi Kumari, Manasi Patwardhan, Shirish Karande, Puneet Agarwal, Lovekesh Vig, Gautam Shroff

Practical Transformer-based Multilingual Text Classification (2021.naacl-industry)

Challenge:	Recent research on cross-lingual transfer shows state-of-the-art results on benchmark datasets using pre-trained language representation models like BERT.
Approach:	They propose a method to augment an annotated dataset with machine translations in target languages and fine-tune the PLRM jointly.
Outcome:	The proposed approach provides consistent gains on multiple benchmark datasets while requiring a single model for multiple languages.

Copied to clipboard

Cindy Wang, Michele Banko

Multilingual BERT Post-Pretraining Alignment (2021.naacl-main)

Challenge:	XNLI does not reflect the data availability and task variety of industry applications.
Approach:	They compare transformer-based text classification methods to multilingual models in five different languages . they use a task- and domain-adaptive pretraining and data augmentation technique .
Outcome:	The proposed methods outperform monolingual models on two tasks in five languages . the results show that practical modifications can improve model performance without labeling .

Copied to clipboard

Lin Pan, Chung-Wei Hang, Haode Qi, Abhishek Shah, Saloni Potdar, Mo Yu

Inducing Language-Agnostic Multilingual Representations (2021.starsem-1)

Challenge:	Recent work improves on the success of monolingual pretrained language models by adding cross-lingual tasks that always involve English.
Approach:	They propose a method to align multilingual contextual embeddings as a post-pretraining step for improved cross-lingual transferability of pretrained language models.
Outcome:	The proposed model outperforms XLM-R_Base on translation-train tasks while using less parallel data and fewer parameters.

Copied to clipboard

Wei Zhao, Steffen Eger, Johannes Bjerva, Isabelle Augenstein

Translation Errors Significantly Impact Low-Resource Languages in Cross-Lingual Learning (2024.eacl-short)

Challenge:	Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world, but they currently require large pretraining corpora or access to typologically similar languages.
Approach:	They propose to remove language identity signals from multilingual embeddings by re-aligning vector spaces of target languages to a pivot source language and removing language-specific means and variances.
Outcome:	The proposed approaches reduce cross-lingual transfer gap by 8.9 points (m-BERT) and 18.2 points (XLM-R) on average across all tasks and languages.

Copied to clipboard

Ashish Agrawal, Barah Fazili, Preethi Jyothi

Don’t Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of Contextual Embeddings (2020.emnlp-main)

Challenge:	XNLI benchmarks use parallel versions of English evaluation sets in multiple target languages . a recent study found that translation errors exist in some low-resource languages resulting in incorrect estimates of cross-lingual transfer .
Approach:	They propose to measure the gap in performance between zero-shot evaluations on human-translated and machine-transcribed target text across multiple target languages.
Outcome:	The proposed benchmarks show that translation errors exist for Hindi and Urdu . the results corroborate previous studies that found translation errors in Hindi and urdu despite translation errors.

Copied to clipboard

Phillip Keung, Yichao Lu, Julian Salazar, Vikas Bhardwaj

S4-Tuning: A Simple Cross-lingual Sub-network Tuning Method (2022.acl-short)

Challenge:	Multilingual contextual embeddings have demonstrated state-of-the-art performance in zero-shot cross-lingual transfer learning.
Approach:	They show that English dev accuracy makes it difficult to obtain reproducible results . they recommend providing oracle scores alongside zero-shot results if possible .
Outcome:	mBERT and XLM have shown strong performance on cross-lingual recognition, text classification, dependency parsing, and other tasks.

Copied to clipboard

Runxin Xu, Fuli Luo, Baobao Chang, Songfang Huang, Fei Huang

Cross-Lingual Transfer

Challenge:	Existing multilingual pre-trained language models allow to adapt to target languages with only few labeled examples.
Approach:	They propose a simple cross-lingual sub-network tuning method that detects the most essential sub-netzwork for each target language and updates it during fine-tuning.
Outcome:	The proposed method improves on three multi-lingual tasks involving 37 different languages.

T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification (2023.tacl-1)

Copied to clipboard

Inigo Jauregi Unanue, Gholamreza Haffari, Massimo Piccardi

Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer (2023.findings-acl)

Challenge:	Existing approaches to cross-lingual text classification leverage text classifiers trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning.
Approach:	They propose to combine a neural machine translator and a text classifier trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning.
Outcome:	The proposed approach significantly improves over a baseline approach.

Copied to clipboard

Shuang Li, Xuming Hu, Aiwei Liu, Yawen Yang, Fukun Ma, Philip S. Yu, Lijie Wen

A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference (2022.emnlp-main)

Challenge:	Existing approaches to cross-lingual natural language inference lack annotated parallel corpora.
Approach:	They propose a new prompt learning framework with the Multilingual Verbalizer for XNLI that uses a multilingual verbalizer to align the representations of original and augmented multilingual questions into a unified semantic space with consistency regularization.
Outcome:	The proposed framework outperforms existing methods under few-shot and full-shot cross-lingual transfer settings.

Copied to clipboard

Kerem Zaman, Yonatan Belinkov

Reasoning

Challenge:	Existing evaluations of attribution methods focus on the English language . plausibility and faithfulness are two main criteria for plausible and faithful attributions .
Approach:	They propose a cross-lingual strategy to measure faithfulness based on word alignments.
Outcome:	The proposed approach eliminates drawbacks of erasure-based evaluations and provides a multilingual dataset with highlights to support future studies.

EnerGIZAr: Leveraging GIZA++ for Effective Tokenizer Initialization (2025.findings-acl)

Copied to clipboard

Pranaydeep Singh, Eneko Agirre, Gorka Azkune, Orphee De Clercq, Els Lefever

Enhancing Cross-lingual Natural Language Inference by Prompt-learning from Cross-lingual Templates (2022.acl-long)

Challenge:	Continual pre-training has long been considered the default strategy for adapting models to non-English languages, but struggles with initializing new embeddings, especially for non-Latin scripts.
Approach:	They propose a method that leverages statistical word alignment techniques to improve continual pre-training by leveraging word alignment matrix between source and target tokens.
Outcome:	The proposed method outperforms existing methods on key NLP tasks including POS tagging, Sentiment Analysis, NLI, and NER in Hindi, Basque, Arabic and Korean.

Copied to clipboard

Kunxun Qi, Hai Wan, Jianfeng Du, Haolan Chen

C-XNLI: Croatian Extension of XNLI Dataset (2023.findings-acl)

Challenge:	Existing methods for enhancing pre-trained cross-lingual language models with additional data are rare in practice, especially for low-resource languages.
Approach:	They propose a prompt-learning framework for enhancing cross-lingual natural language inference by constructing cloze-style questions through cross-linguistic templates.
Outcome:	The proposed framework significantly outperforms existing models under cross-lingual transfer settings.

Copied to clipboard

Leo Obadić, Andrej Jertec, Marko Rajnović, Branimir Dropuljić

Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model (2022.emnlp-main)

Challenge:	Existing parallel datasets limit multilingual evaluations due to the nature of linguistic annotation, which is tedious, subjective, and costly.
Approach:	They extend the Cross-lingual Natural Language Inference corpus with Croatian and use Facebook's 1.2B parameter m2m_100 model to analyze the train set and compare its quality with the existing machine-translated German set.
Outcome:	The proposed model is consistent with other XNLI dubs and is compared with the existing machine-translated German train set.

Copied to clipboard

Mingqi Li, Fei Ding, Dan Zhang, Long Cheng, Hongxin Hu, Feng Luo

A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets (2024.findings-naacl)

Challenge:	Existing methods for improving multilingual models did not focus on learning the semantic structure of representation.
Approach:	They propose a method to improve multilingual language models by aligning parallel sentences . they propose token-, word-, sentence- and structure-level alignment objectives .
Outcome:	The proposed method outperforms baseline models on XNLI, PAWS-X, and XQuAD . it obtains comparable performance on low-resource languages, the authors show .

Copied to clipboard

Tanja Samardzic, Ximena Gutierrez, Christian Bentz, Steven Moran, Olga Pelloni

Subword Mapping and Anchoring across Languages (2021.findings-emnlp)

Challenge:	a new study aims to assess linguistic diversity of multilingual data sets against a reference language sample . linguistic diversity is typically measured as the number of languages included in the data set . but such measures do not consider structural properties of the included languages .
Approach:	They propose to measure linguistic diversity against a reference language sample to maximise linguistic diversity.
Outcome:	The proposed measure can be used to identify the types of languages that are not represented in a data set.

Copied to clipboard

Giorgos Vernikos, Andrei Popescu-Belis

XNLIeu: a dataset for cross-lingual NLI in Basque (2024.naacl-long)

Challenge:	State-of-the-art multilingual systems rely on shared vocabularies that cover all considered languages.
Approach:	They propose a method to construct bilingual subword vocabularies by mapping and anchoring subwords together over multiple languages.
Outcome:	The proposed method improves zero-shot transfer to an unseen language without task-specific data, but only by sharing subword embeddings.

Copied to clipboard

Maite Heredia, Julen Etxaniz, Muitze Zulaika, Xabier Saralegi, Jeremy Barnes, Aitor Soroa

Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (D19-1)

Challenge:	XNLI is a popular benchmark used to evaluate cross-lingual Natural Language Understanding (NLU) in languages such as English, Basque and other low-resource languages.
Approach:	They expand XNLI to include Basque, a low-resource language that can benefit from transfer-learning approaches.
Outcome:	The proposed dataset includes Basque, a low-resource language that can benefit from transfer-learning approaches.

Copied to clipboard

Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Ming Zhou

XNLI: Evaluating Cross-lingual Sentence Representations (D18-1)

Challenge:	Existing models that can handle cross-lingual tasks with limited or no training data are insensitive to different languages.
Approach:	They propose to use Unicoder to train models in one language and apply it to other languages.
Outcome:	Experiments show that Unicoder learns the mappings among different languages from more perspectives.

Copied to clipboard

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, Veselin Stoyanov

Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots (2021.naacl-main)

Challenge:	State-of-the-art natural language processing systems rely on annotated data to learn competent models.
Approach:	They extend the development and test sets of the Multi-Genre Natural Language Inference Corpus to 14 languages, including Swahili and Urdu.
Outcome:	The proposed evaluation set extends the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 14 languages including low-resource languages such as Swahili and Urdu.

Copied to clipboard

Samson Tan, Shafiq Joty

Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings (2021.findings-emnlp)

Challenge:	Multilingual models have demonstrated impressive cross-lingual transfer abilities.
Approach:	They propose two strong adversarial attacks that target multilingual models that can handle code-mixed sentences using bilingual dictionaries.
Outcome:	The proposed model has a success rate of 89.75% against XLM-R-large, bringing its average accuracy of 79.85 down to 8.18 on XNLI.

Copied to clipboard

Sawsan Alqahtani, Garima Lalwani, Yi Zhang, Salvatore Romeo, Saab Mansour

A Deep Transfer Learning Method for Cross-Lingual Natural Language Inference (2022.lrec-1)

Challenge:	Recent studies suggest different methods to improve multilingual word representations in contextualized settings including techniques that align between source and target embedding spaces.
Approach:	They propose to use Optimal Transport as an alignment objective during fine-tuning to improve multilingual contextualized representations for downstream cross-lingual transfer.
Outcome:	The proposed method achieves better performance on two tasks (XNLI and XQuAD) and is competitive with existing methods.

Copied to clipboard

Dibyanayan Bandyopadhyay, Arkadipta De, Baban Gain, Tanik Saikh, Asif Ekbal

Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference (2021.findings-acl)

Challenge:	Natural Language Inference (NLI) is a crucial task in AI and natural language processing.
Approach:	They propose an effective transfer learning approach for cross-lingual NLI . they perform experiments on English-Hindi language pairs in cross-linguistic setting .
Outcome:	The proposed model improves the baseline model by 10% over the state-of-the-art model.

Copied to clipboard

Hai Hu, He Zhou, Zuoyu Tian, Yiwen Zhang, Yina Patterson, Yanting Li, Yixin Nie, Kyle Richardson

Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training (2023.findings-acl)

Challenge:	Multilingual transformers have been shown to have remarkable transfer skills in zero-shot settings.
Approach:	They investigate cross-lingual transfer abilities of XLM-R for Chinese and English natural language inference using a large scale Chinese dataset.
Outcome:	The proposed model trains on Chinese and English natural language inference datasets.

Copied to clipboard

Kelly Marchisio, Patrick Lewis, Yihong Chen, Mikel Artetxe

Zero-shot Cross-lingual Alignment for Embedding Initialization (2024.findings-acl)

Challenge:	Existing approaches to pretrain Masked Language Models (MLMs) are expensive and require a full forward and backward pass over the entire model.
Approach:	They propose to learn a shallow mini-model from a fraction of a large model's parameters and plug it into a larger model for rapid cross-lingual transfer.
Outcome:	Experiments on XNLI, MLQA and PAWS-X show that mini-model adaptation matches the standard approach using up to 2.3x less compute on average.

Copied to clipboard

Xi Ai, Zhiyong Huang

Identifying Elements Essential for BERT’s Multilinguality (2020.emnlp-main)

Challenge:	CrossInit initializes embeddings into similar geometrical structures across languages in unsupervised manner.
Approach:	They propose a method that initializes embeddings into similar geometrical structures across languages in an unsupervised manner.
Outcome:	The proposed method demostrates similar patterns in low-resource and dissimilar languages.

Copied to clipboard

Philipp Dufter, Hinrich Schütze

Improving Multilingual Models with Language-Clustered Vocabularies (2020.emnlp-main)

Challenge:	Multilingual BERT (mBERT) does not use any crosslingual signal during training.
Approach:	They propose a multilingual pretraining setup that modifies the masking strategy using VecMap to allow for fast experimentation.
Outcome:	The proposed setup with pretrained models with three languages shows that it works well.

Copied to clipboard

Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, Jason Riesa

Zero-Shot Cross-Lingual Transfer with Meta Learning (2020.emnlp-main)

Challenge:	State-of-the-art multilingual models depend on vocabularies that cover all languages . but the methods for generating those vocalaries are not ideal for massively multilingual applications.
Approach:	They propose a procedure for multilingual vocabulary generation that combines separately trained vocabularies of several automatically derived language clusters.
Outcome:	The proposed procedure shows improvements across languages on multilingual benchmark tasks . the proposed procedure reduces out-of-vocabulary rate by a factor of 8 .

Copied to clipboard

Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, Isabelle Augenstein

Learning Task Sampling Policy for Multitask Learning (2021.findings-emnlp)

Challenge:	There are more than 7,000 languages spoken in the world, over 90 of which have more than 10 million native speakers each.
Approach:	They propose to use meta-learning to train a model on multiple languages at the same time . they use standard supervised, zero-shot cross-lingual, and few-shot crosses-lingual settings for different natural language understanding tasks.
Outcome:	The proposed setup improves on the state-of-the-art for a total of 15 languages.

Copied to clipboard

Dhanasekar Sundararaman, Henry Tsai, Kuang-Huei Lee, Iulia Turc, Lawrence Carin

Targeted Adversarial Training for Natural Language Understanding (2021.naacl-main)

Challenge:	Existing methods to train multi-task models with auxiliary tasks are limited by the number of combinations and the importance of each auxiliary task is not known a priori.
Approach:	They propose a search method that automatically assigns importance weights to auxiliary tasks to improve the target task quality.
Outcome:	The proposed method outperforms uniform sampling and the corresponding single-task baseline on XNLI and GLUE.

Copied to clipboard

Lis Pereira, Xiaodong Liu, Hao Cheng, Hoifung Poon, Jianfeng Gao, Ichiro Kobayashi

Robustness in NLP

Challenge:	Existing adversarial training approaches focus on making adversarials less expensive or regularizing rather than replacing the standard training objective.
Approach:	They propose an algorithm to introspect current mistakes and prioritize adversarial training steps to where the model errs the most.
Outcome:	The proposed algorithm improves adversarial training for natural language understanding by introspecting mistakes and prioritizing training steps to where the model errs the most.

AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages (2022.acl-long)

Copied to clipboard

Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, Katharina Kann

ALBETO and DistilBETO: Lightweight Spanish Language Models (2022.lrec-1)

Challenge:	Pretrained multilingual models can perform cross-lingual transfer in a zero-shot setting, even for unseen languages.
Approach:	They propose to extend XNLI to 10 indigenous languages of the Americas and test multiple zero-shot and translation-based approaches.
Outcome:	The proposed model can perform cross-lingual transfer in a zero-shot setting even for languages unseen during pretraining.

Copied to clipboard

José Cañete, Sebastian Donoso, Felipe Bravo-Marquez, Andrés Carvallo, Vladimir Araujo

Translation Artifacts in Cross-lingual Transfer Learning (2020.emnlp-main)

Challenge:	Recent advances in pre-trained language models have made them more popular . however, there are still limited versions of these models for other languages .
Approach:	They present ALBETO and DistilBETO which are versions of ALBERT and DistillBERT pre-trained exclusively on Spanish corpora.
Outcome:	The proposed models outperform BETO and ALBERT on Spanish datasets . the models outpace BETO on MLDoc, PAWS-X, XNLI, MLQA, SQAC and XQuAD datasets.

Copied to clipboard

Mikel Artetxe, Gorka Labaka, Eneko Agirre

Enhancing Cross-lingual Prompting with Dual Prompt Augmentation (2023.findings-acl)

Challenge:	Existing cross-lingual transfer learning techniques involve human and machine translations.
Approach:	They propose to use machine translation to translate test set or training set to introduce subtle artifacts that have a notable impact in existing cross-lingual models.
Outcome:	The proposed translation process reduces the lexical overlap between the premise and hypothesis by 4.3 and 2.8 points . the proposed translation-test and zero-shot approaches improve on previous work .

Copied to clipboard

Meng Zhou, Xin Li, Yue Jiang, Lidong Bing

Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)

Challenge:	a recent study shows that prompting is superior for multilingual/cross-lingual problems . despite its effectiveness on English tasks, its potential for cross-lingual problem is under-explored .
Approach:	They propose a framework for prompting that can be used to augment cross-lingual prompts.
Outcome:	The proposed framework achieves 46.54% with only 16 English training examples per class, significantly better than fine-tuning.

Copied to clipboard

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov

Improving Cross-lingual Transfer with Contrastive Negative Learning and Self-training (2024.lrec-main)

Challenge:	Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks.
Approach:	They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains .
Outcome:	The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks.

Copied to clipboard

Guanlin Li, Xuechen Zhao, Amir Jafari, Wenhao Shao, Reza Farahbakhsh, Noel Crespi

Cross-Lingual Transfer

Challenge:	Recent studies improve cross-lingual transfer learning by better aligning the internal representations within the multilingual model or exploring the information of the target language using self-training.
Approach:	They propose to use negative pairs to align the multilingual model and self-train the model to converge on the obtained clean pseudo-labels.
Outcome:	The proposed method improves upon the baseline models and can serve as a beneficial complement to the alignment-based methods.

XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models (2023.emnlp-main)

Copied to clipboard

Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, Madian Khabsa

New Datasets for Automatic Detection of Textual Entailment and of Contradictions between Sentences in French (2024.lrec-main)

Challenge:	Large multilingual models rely on a single vocabulary shared across 100+ languages . this vocabulary bottleneck limits the representational capabilities of multilingual model XLM-R .
Approach:	They propose a new approach for scaling to large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language.
Outcome:	The proposed model outperforms XLM-R on all language tasks and is particularly effective on low-resource tasks.

Copied to clipboard

Maximos Skandalis, Richard Moot, Christian Retoré, Simon Robillard

Textual Inference

Challenge:	DACCORD is a dataset for automatic detection of contradictions between sentences . it is the first dataset exclusively dedicated to this task .
Approach:	They introduce DACCORD, a dataset in French for automatic detection of contradictions between sentences.
Outcome:	The proposed datasets are more challenging than existing datasets for the mainstream task in French.

False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models (2025.findings-emnlp)

Copied to clipboard

Julie Kallini, Dan Jurafsky, Christopher Potts, Martijn Bartelds

Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text (2026.acl-long)

Challenge:	Prior work has shown that token overlap facilitates cross-lingual transfer or introduces interference between languages?
Approach:	They devised a controlled experiment where they train bilingual autoregressive models on multiple language pairs under systematically varied vocabulary overlap settings.
Outcome:	The proposed model outperforms models with disjointed vocabularies on XNLI and XQuAD and shows that token overlap is beneficial for multilingual tokenizers.

Copied to clipboard

Amr Mohamed, Yang Zhang, Michalis Vazirgiannis, Guokan Shang