| Challenge: | Using adversarial training, we can “soft-code” shared and private spaces to avoid sparse sharing. |
| Approach: | They propose to use adversarial training to “soft-code” shared and private spaces to avoid the shared space gets too sparse. |
| Outcome: | The proposed architecture avoids sparse sharing of shared and private spaces, and also deals with low-quality input. |
Similar Papers
Bag-of-Words Transfer: Non-Contextual Techniques for Multi-Task Learning (D19-61)
Copied to clipboard
| Challenge: | Existing approaches to multi-task learning take advantage of transfer among tasks . generative reconstruction of the observations is not included in the standard framework . |
| Approach: | They propose to use a syntactically-oblivious pooling encoder and pre-trained word embeddings to improve sentence-level representations. |
| Outcome: | The proposed techniques yield similar performance on a universe of task combinations while reducing training time and model size. |
Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation (P18-2)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) requires large amounts of bilingual data to learn a translation model with reasonable quality. |
| Approach: | They propose to extend recurrent units with multiple "blocks" along with a trainable "routing network" this allows for adaptive collaboration by dynamic sharing of blocks conditioned on the task at hand, input, and model state. |
| Outcome: | Empirical evaluations of two low-resource translation tasks show +1 BLEU score improvements compared to strong baselines. |
On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment (2020.emnlp-main)
Copied to clipboard
| Challenge: | Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer) however, recent work has shown that this approach can degrade performance on high-resource languages, a phenomenon known as negative interference. |
| Approach: | They propose a meta-learning algorithm that adds language-specific parameters as meta-parameters and trains them in a manner that explicitly improves shared layers’ generalization on all languages. |
| Outcome: | The proposed model improves cross-lingual transferability and generalization on all languages, and improves on the language-specific parameters. |
Cross-lingual Multi-Level Adversarial Transfer to Enhance Low-Resource Name Tagging (N19-1)
Copied to clipboard
| Challenge: | Low-resource language name tagging is an important but challenging task. |
| Approach: | They propose a neural architecture that leverages multi-level adversarial transfer to improve name tagging for low-resource languages. |
| Outcome: | The proposed approach outperforms previous approaches on CoNLL data sets. |
Meta Distant Transfer Learning for Pre-trained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Notable PLMs are available for text classification tasks, but performance of PLM on downstream tasks may be limited by the availability of training set. |
| Approach: | They propose a meta-learning framework to learn the transferable knowledge across tasks using PLMs. |
| Outcome: | The proposed framework outperforms baselines on seven datasets and is task-agnostic and unbiased. |
Unsupervised Cross-Lingual Representation Learning (P19-4)
Copied to clipboard
| Challenge: | a comprehensive survey of cutting-edge weakly-supervised and unsupervised cross-lingual word representations is presented . |
| Approach: | This tutorial provides a comprehensive survey of recent work on weakly-supervised and unsupervised cross-lingual word representations. |
| Outcome: | This tutorial provides a comprehensive survey of cutting-edge weakly-supervised and unsupervised word representations. |
Improving Cross-lingual Transfer with Contrastive Negative Learning and Self-training (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies improve cross-lingual transfer learning by better aligning the internal representations within the multilingual model or exploring the information of the target language using self-training. |
| Approach: | They propose to use negative pairs to align the multilingual model and self-train the model to converge on the obtained clean pseudo-labels. |
| Outcome: | The proposed method improves upon the baseline models and can serve as a beneficial complement to the alignment-based methods. |
Tricks for Training Sparse Translation Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Multitask learning with an unbalanced data distribution skews model learning towards high resource tasks. |
| Approach: | They propose to use a temperature heating mechanism and dense pre-training to mitigate this by training models with a fixed model capacity. |
| Outcome: | The proposed techniques improve performance on two multilingual translation benchmarks compared to BASELayers and Dense scaling baselines and in combination, more than 2x model convergence speed. |
Multi-Source Cross-Lingual Model Transfer: Learning What to Share (P19-1)
Copied to clipboard
| Challenge: | Cross-lingual transfer learning (CLTL) is a viable method for building NLP models for a low-resource target language . however, many languages lack the labeled training data necessary for training deep neural nets for varying NLP tasks. |
| Approach: | They propose a cross-lingual transfer learning method that leverages annotated data from other languages to build NLP models for a target language. |
| Outcome: | The proposed model achieves significant performance gains over prior art over multiple text classification and sequence tagging tasks including a large-scale industry dataset. |
Cross-Lingual Text Classification with Minimal Resources by Transferring a Sparse Teacher (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches for transferring supervision across languages require expensive cross-lingual resources. |
| Approach: | They propose a cross-lingual teacher-student method that generates "weak" supervision in a target language using minimal cross-linguistic resources. |
| Outcome: | The proposed method outperforms state-of-the-art methods with a student classifier in 18 languages . it extracts and transfers only the most important task-specific seed words across languages based on translated seed words . |