Por Qué Não Utiliser Alla Språk? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer (2022.findings-naacl)
Copied to clipboard
| Challenge: | a lack of labeled data for low-resource languages leads to the need for effective cross-lingual transfer learning. |
| Approach: | They propose a mixed training method that trains on both source and target data with stochastic gradient surgery, a novel gradient-level optimization. |
| Outcome: | The proposed method outperforms current methods on all tasks and escapes overfitting issues. |
Similar Papers
Cross-lingual Few-Shot Learning on Unseen Languages (2022.aacl-main)
Copied to clipboard
| Challenge: | Large pre-trained language models have demonstrated the ability to obtain good performance on downstream tasks with limited examples in resource-rich languages. |
| Approach: | They propose to use a downstream sentiment analysis task to analyze the effectiveness of several few-shot learning strategies across 12 languages, including 8 unseen languages, to compare results. |
| Outcome: | The proposed model, XLM-R, gives the best performance on a task with few examples in resource-rich languages. |
Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual generalisability of large pre-trained models use English training data and test data in unseen languages. |
| Approach: | They propose to use multilingual pre-trained models to model cross-lingual transfer in a selection of target languages. |
| Outcome: | The proposed model can be used to improve cross-lingual transfer performance in low-resource languages with no labeled training data. |
Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained multilingual language encoders do not precisely align words and phrases across languages. |
| Approach: | They propose a learning strategy for training robust models by drawing connections between adversarial examples and failure cases of zero-shot cross-lingual transfer. |
| Outcome: | The proposed model can achieve good performance even if representations of different languages are not aligned well. |
Don’t Stop Fine-Tuning: On Training Regimes for Few-Shot Cross-Lingual Transfer with Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work highlights the fallacies of zero-shot cross-lingual transfer with large multilingual models. |
| Approach: | They propose to replace sequential fine-tuning with joint fine-uning on source and target language instances. |
| Outcome: | The proposed techniques yield improved and more stable FS-XLT across the board. |
A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning (2022.coling-1)
Copied to clipboard
Kunbo Ding, Weijie Liu, Yuejian Fang, Weiquan Mao, Zhe Zhao, Tao Zhu, Haoyan Liu, Rong Tian, Yiren Chen
| Challenge: | Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries . however, its effect is limited by the gap between embedding clusters of different languages . |
| Approach: | They propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embedders without semantic loss. |
| Outcome: | Experimental results show that the proposed method outperforms existing methods on cross-lingual tasks and can achieve a better multilingual alignment. |
Zero-Shot Cross-Lingual Transfer with Meta Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | There are more than 7,000 languages spoken in the world, over 90 of which have more than 10 million native speakers each. |
| Approach: | They propose to use meta-learning to train a model on multiple languages at the same time . they use standard supervised, zero-shot cross-lingual, and few-shot crosses-lingual settings for different natural language understanding tasks. |
| Outcome: | The proposed setup improves on the state-of-the-art for a total of 15 languages. |
Nearest Neighbour Few-Shot Learning for Cross-lingual Classification (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained models can cause over-fitting when limited data are available. |
| Approach: | They propose to use a nearest-neighbor few-shot technique to improve cross-lingual adaptation using 16 distinct languages across two NLP tasks. |
| Outcome: | The proposed approach improves fine-tuning using only a handful of labeled samples in target locales and also generalizes across tasks. |
How to Translate Your Samples and Choose Your Shots? Analyzing Translate-train & Few-shot Cross-lingual Transfer (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have focused on zero-shot cross-lingual transfer of pretrained languages. |
| Approach: | They propose to use few-shot cross-lingual transfer to improve zero-shot performance of multilingual pretrained language models. |
| Outcome: | The proposed model can be scaled to high-quality samples and improves on zero-shot performance. |
Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing cross-lingual transfer methods that use labeled data and linguistic resources would consume excessive resources for a large number of languages. |
| Approach: | They propose a parameter-efficient cross-lingual transfer learning framework that utilizes a translation-based alignment method to mitigate multilingual disparities. |
| Outcome: | The proposed framework reduces disparities among languages and improves cross-lingual transfer results in low-resource scenarios while keeping and fine-tuning only a small number of parameters. |
Saliency-based Multi-View Mixed Language Training for Zero-shot Cross-lingual Classification (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent multilingual pre-trained models have been demonstrated effective in many cross-lingual tasks. |
| Approach: | They propose a framework that leverages code-switched data with multi-view learning to fine-tune XLM-R. |
| Outcome: | The proposed model achieves state-of-the-art on zero-shot cross-lingual sentiment classification and dialogue state tracking tasks. |