| Challenge: | Existing methods for crosslingual transfer use multilingual word embeddings, but contextual word representations are not yet available. |
| Approach: | They propose a method to produce multilingual contextual word representations by training a single language model on text from multiple languages. |
| Outcome: | The proposed method compares model models to monolingual and non-contextual variants and shows that polyglot learning can be beneficial for multilingual representations. |
Similar Papers
Cross-lingual Transfer of Monolingual Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual learning using multilingual models cast doubt on shared vocabulary and joint pre-training . et al. (2005) show that model knowledge learned in the source language enhances the learning of the target language independently of language proximity. |
| Approach: | They propose a method for transferring monolingual models to other languages through continuous pre-training and investigate their results in English. |
| Outcome: | The proposed method outperforms a model trained from scratch in the GLUE benchmark for English . it shows that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in english. |
Unsupervised Cross-Lingual Representation Learning (P19-4)
Copied to clipboard
| Challenge: | a comprehensive survey of cutting-edge weakly-supervised and unsupervised cross-lingual word representations is presented . |
| Approach: | This tutorial provides a comprehensive survey of recent work on weakly-supervised and unsupervised cross-lingual word representations. |
| Outcome: | This tutorial provides a comprehensive survey of cutting-edge weakly-supervised and unsupervised word representations. |
Sources of Transfer in Multilingual Named Entity Recognition (2020.acl-main)
Copied to clipboard
| Challenge: | naive training of named-entity recognition models using annotated data from multiple languages consistently underperforms monolingual models. |
| Approach: | They propose a polyglot named-entity recognition model where one model is trained using annotated data drawn from multiple languages. |
| Outcome: | The proposed model outperforms models trained on monolingual data despite more training data . the proposed model shares many parameters across languages and fine-tunes them to outperFORM monolingual models. |
Cross-lingual Parsing with Polyglot Training and Multi-treebank Learning: A Faroese Case Study (D19-61)
Copied to clipboard
| Challenge: | Cross-lingual dependency parsing involves transferring syntactic knowledge from one language to another. |
| Approach: | They compare two approaches to cross-lingual dependency parsing using monolingual source models and a polyglot model which is trained on the combination of all source languages. |
| Outcome: | The proposed methods improve low-resource dependency parsers by transferring syntactic knowledge from one language to another. |
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |
KIT-Multi: A Translation-Oriented Multilingual Embedding Corpus (L18-1)
Copied to clipboard
| Challenge: | Cross-lingual word embeddings are representations of words across languages in a shared continuous vector space. |
| Approach: | They propose a multilingual word embedding corpus which is acquired by neural machine translation and is based on monolingual data. |
| Outcome: | The proposed method is competitive with existing methods but on the cross-lingual document classification task, it obtains the best figures. |
Language Embeddings for Typology and Cross-lingual Transfer Learning (2021.acl-long)
Copied to clipboard
| Challenge: | Recent efforts to leverage multilingual datasets highlight potential of multilingual models that can perform well across various languages. |
| Approach: | They propose to generate language representations that capture relationships among languages and evaluate them using WALS and two extrinsic tasks. |
| Outcome: | The proposed model can be leveraged in cross-lingual tasks without parallel data . the proposed model is based on the World Atlas of Language Structures (WALS) and two extrinsic tasks . |
Joint Representation Learning of Cross-lingual Words and Entities via Attentive Distant Supervision (D18-1)
Copied to clipboard
| Challenge: | Existing methods for learning word and entity representations in monolingual settings are limited. |
| Approach: | They propose a method for joint representation learning of cross-lingual words and entities that captures mutually complementary knowledge and enables cross-linguistic inferences. |
| Outcome: | The proposed method captures mutually complementary knowledge and enables cross-lingual inferences among knowledge bases and texts. |
Polyglot Semantic Role Labeling (P18-2)
Copied to clipboard
| Challenge: | Existing approaches to multilingual semantic dependency parsing treat languages independently, without exploiting similarities between semantic structures across languages. |
| Approach: | They propose to combine resources from different languages in a CoNLL 2009 shared task to build a single polyglot semantic dependency parser. |
| Outcome: | The proposed model outperforms monolingual training on a CoNLL 2009 dataset with training data from multiple languages and representations using multilingual word vectors. |
Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies show that multilingual generative models exhibit a strong language bias toward high-resource languages. |
| Approach: | They propose a cross-lingual alignment framework exploiting pairs of translation sentences to improve cross-linguistic abilities. |
| Outcome: | The proposed framework improves cross-lingual abilities and mitigates performance gap. |