DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer (2023.eacl-main)
Copied to clipboard
| Challenge: | Zero-shot cross-lingual transfer has been shown to be sub-optimal across low-resource languages due to the skew in resource distribution in languages. |
| Approach: | They propose to jointly reduce feature incongruity between the source and target language and increase generalization capabilities of pre-trained multilingual transformers. |
| Outcome: | Empirical results show that the proposed approach outperforms the standard zero-shot fine-tuning method on multiple datasets across all languages using only unlabeled instances in the target language. |
Similar Papers
A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning (2022.coling-1)
Copied to clipboard
Kunbo Ding, Weijie Liu, Yuejian Fang, Weiquan Mao, Zhe Zhao, Tao Zhu, Haoyan Liu, Rong Tian, Yiren Chen
| Challenge: | Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries . however, its effect is limited by the gap between embedding clusters of different languages . |
| Approach: | They propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embedders without semantic loss. |
| Outcome: | Experimental results show that the proposed method outperforms existing methods on cross-lingual tasks and can achieve a better multilingual alignment. |
Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual generalisability of large pre-trained models use English training data and test data in unseen languages. |
| Approach: | They propose to use multilingual pre-trained models to model cross-lingual transfer in a selection of target languages. |
| Outcome: | The proposed model can be used to improve cross-lingual transfer performance in low-resource languages with no labeled training data. |
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that multilingual transformers are less effective in resource-lean scenarios and for distant languages. |
| Approach: | They propose to use massively multilingual transformers to pretrain languages . they show that MMTs are less effective in resource-lean scenarios and distant languages if they are pre-trained via language modeling . |
| Outcome: | The proposed model is less effective in resource-lean scenarios and for distant languages than cross-lingual word embeddings. |
Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on zero-shot cross-lingual transfer . mBERT, mBART and mT5 provide high-quality representations for texts in various languages . |
| Approach: | They propose to use mBART and NLLB-200 to finetune a multilingual pretrained language model on input-output pairs in one language and use it to make task predictions for inputs in other languages. |
| Outcome: | The proposed approach significantly reduces generation in the wrong language with full finetuning and can be competitive in some cases. |
Model and Data Transfer for Cross-Lingual Sequence Labelling in Zero-Resource Settings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have proposed data-based cross-lingual transfer as an effective technique for cross-linguistic sequence labelling, but they have failed to perform well. |
| Approach: | They propose to use data-based cross-lingual transfer to train supervised models from a source language to unlabelled target languages. |
| Outcome: | The proposed techniques outperform data-based cross-lingual transfer approaches in a zero-shot setting. |
SLICER: Sliced Fine-Tuning for Low-Resource Cross-Lingual Transfer for Named Entity Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large multilingual models fail to successfully transfer to low-resource languages for zero-shot cross-lingual transfer . sliced fine-tuning for named entity recognition (SLICER) forces stronger token contextualization in the Transformer. |
| Approach: | They propose a simple yet highly effective approach for improving zero-shot cross-lingual transfer for named entity recognition to low-resource languages. |
| Outcome: | The proposed approach reduces decontextualization of token representations and classifiers . it yields consistent transfer gains for low-resource languages, the authors show . |
Unifying Cross-Lingual Transfer across Scenarios of Resource Scarcity (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to deal with resource scarcity have not been developed to deal effectively with the problem. |
| Approach: | They propose to use a set of tools to harness data from one or more high-resource "source" languages to compensate for a shortage of data in low-resourced "target" languages. |
| Outcome: | The proposed technique can be easily adapted to unseen languages, extending the range of the proposed technique and translation-based transfer more broadly. |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing cross-lingual transfer methods that use labeled data and linguistic resources would consume excessive resources for a large number of languages. |
| Approach: | They propose a parameter-efficient cross-lingual transfer learning framework that utilizes a translation-based alignment method to mitigate multilingual disparities. |
| Outcome: | The proposed framework reduces disparities among languages and improves cross-lingual transfer results in low-resource scenarios while keeping and fine-tuning only a small number of parameters. |
Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained multilingual language encoders do not precisely align words and phrases across languages. |
| Approach: | They propose a learning strategy for training robust models by drawing connections between adversarial examples and failure cases of zero-shot cross-lingual transfer. |
| Outcome: | The proposed model can achieve good performance even if representations of different languages are not aligned well. |