Zero-Shot Dependency Parsing with Worst-Case Aware Automated Curriculum Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Large multilingual pretrained language models such as mBERT and XLM-RoBERTa have been found to be effective for cross-lingual transfer of syntactic parsing models but only between related languages. |
| Approach: | They propose to use multi-task learning to dynamically optimize for parsing performance on outlier languages by using a multi-level learning approach. |
| Outcome: | The proposed method significantly outperforms uniform and size-proportional sampling in the zero-shot setting. |
Similar Papers
Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representations (D19-61)
Copied to clipboard
| Challenge: | Pretrained sentence representations have set the new state of the art in many language understanding tasks. |
| Approach: | They propose to use a multilingual corpus to train deep bidirectional sentence representations that are fully lexicalized to allow for the development of an unsupervised universal dependency parser. |
| Outcome: | The proposed approach outperforms the best CoNLL 2018 systems in all of the shared task’s six truly low-resource languages while using a single system. |
A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning (2022.coling-1)
Copied to clipboard
Kunbo Ding, Weijie Liu, Yuejian Fang, Weiquan Mao, Zhe Zhao, Tao Zhu, Haoyan Liu, Rong Tian, Yiren Chen
| Challenge: | Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries . however, its effect is limited by the gap between embedding clusters of different languages . |
| Approach: | They propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embedders without semantic loss. |
| Outcome: | Experimental results show that the proposed method outperforms existing methods on cross-lingual tasks and can achieve a better multilingual alignment. |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models (2022.acl-long)
Copied to clipboard
| Challenge: | Massively Multilingual Transformer based Language Models have been shown to be effective on zero-shot transfer across languages, though performance varies from language to language depending on pivot language(s) used for fine-tuning. |
| Approach: | They propose to combine multi-task learning problems with multi-lingual Transformers to model zero-shot transfer across languages. |
| Outcome: | The proposed model can predict zero-shot transfer across languages with a multi-task learning problem with pretraining data in very few languages. |
Zero- and Few-Shot NLP with Pretrained Language Models (2022.acl-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to introduce NLP researchers to the latest techniques for learning from little-to-no data . aims at bringing interested researchers up to speed about the latest and ongoing techniques . |
| Approach: | They aim to introduce techniques for learning from little-to-no data using pretrained language models. |
| Outcome: | This tutorial aims to bring interested NLP researchers up to speed about recent techniques . it will cover methods from manual engineering, better inference algorithms to better tuning methods . |
Frustratingly Simple but Surprisingly Strong: Using Language-Independent Features for Zero-shot Cross-lingual Semantic Parsing (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing training data is limited for languages other than English, so is the performance of the developed parsers. |
| Approach: | They propose to apply a pre-trained multilingual model to Italian, German and Dutch parsers where only a small number of manually annotated parses are available. |
| Outcome: | The proposed model improves on six parsers in English and Italian, German and Dutch, with the addition of universal dependency relations and universal POS tags as model-agnostic features. |
Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on zero-shot cross-lingual transfer . mBERT, mBART and mT5 provide high-quality representations for texts in various languages . |
| Approach: | They propose to use mBART and NLLB-200 to finetune a multilingual pretrained language model on input-output pairs in one language and use it to make task predictions for inputs in other languages. |
| Outcome: | The proposed approach significantly reduces generation in the wrong language with full finetuning and can be competitive in some cases. |
Massively Multilingual Transfer for NER (P19-1)
Copied to clipboard
| Challenge: | Existing approaches for cross-lingual transfer use a single source language, but there are exceptions. |
| Approach: | They propose two techniques for modulating the transfer, suitable for zero-shot or few-shot learning, respectively. |
| Outcome: | The proposed methods are much more effective than baseline models and rival oracle selection of the single best individual model. |
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that multilingual transformers are less effective in resource-lean scenarios and for distant languages. |
| Approach: | They propose to use massively multilingual transformers to pretrain languages . they show that MMTs are less effective in resource-lean scenarios and distant languages if they are pre-trained via language modeling . |
| Outcome: | The proposed model is less effective in resource-lean scenarios and for distant languages than cross-lingual word embeddings. |
Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual generalisability of large pre-trained models use English training data and test data in unseen languages. |
| Approach: | They propose to use multilingual pre-trained models to model cross-lingual transfer in a selection of target languages. |
| Outcome: | The proposed model can be used to improve cross-lingual transfer performance in low-resource languages with no labeled training data. |