Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual models cannot fully leverage training data when it is available in different task-language combinations. |
| Approach: | They propose a single hypernetwork that unifies multi-task and multilingual learning with efficient adaptation. |
| Outcome: | The proposed model achieves the best or competitive gain when a mixture of multiple resources is available while being significantly more efficient than existing models. |
Similar Papers
Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models (2022.acl-long)
Copied to clipboard
| Challenge: | Massively Multilingual Transformer based Language Models have been shown to be effective on zero-shot transfer across languages, though performance varies from language to language depending on pivot language(s) used for fine-tuning. |
| Approach: | They propose to combine multi-task learning problems with multi-lingual Transformers to model zero-shot transfer across languages. |
| Outcome: | The proposed model can predict zero-shot transfer across languages with a multi-task learning problem with pretraining data in very few languages. |
MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current deep pretrained models lack capacity to represent all languages . limited capacity is an issue even for high-resource languages where models are not included in training data at all. |
| Approach: | They propose an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations. |
| Outcome: | The proposed framework outperforms state-of-the-art models on cross-lingual transfer across languages and typologically diverse models. |
Unifying Cross-Lingual Transfer across Scenarios of Resource Scarcity (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to deal with resource scarcity have not been developed to deal effectively with the problem. |
| Approach: | They propose to use a set of tools to harness data from one or more high-resource "source" languages to compensate for a shortage of data in low-resourced "target" languages. |
| Outcome: | The proposed technique can be easily adapted to unseen languages, extending the range of the proposed technique and translation-based transfer more broadly. |
A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning (2022.coling-1)
Copied to clipboard
Kunbo Ding, Weijie Liu, Yuejian Fang, Weiquan Mao, Zhe Zhao, Tao Zhu, Haoyan Liu, Rong Tian, Yiren Chen
| Challenge: | Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries . however, its effect is limited by the gap between embedding clusters of different languages . |
| Approach: | They propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embedders without semantic loss. |
| Outcome: | Experimental results show that the proposed method outperforms existing methods on cross-lingual tasks and can achieve a better multilingual alignment. |
CrossAligner & Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understanding (2022.findings-acl)
Copied to clipboard
| Challenge: | Task-oriented personal assistants enable people to interact with devices and services using natural language. |
| Approach: | They propose a method to acquire task knowledge in a high-resource language and then transfer it to the low-resourced language(s) they use unlabelled parallel data to perform a quantitative analysis of the methods. |
| Outcome: | The proposed methods exceed state-of-the-art (SOTA) scores across nine languages, fifteen test sets and three benchmark multilingual datasets. |
Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in training multilingual models on large datasets have shown promising results in knowledge transfer across languages. |
| Approach: | They challenge the assumption that high zero-shot performance reflects high cross-lingual ability by introducing more challenging setups involving instances with multiple languages. |
| Outcome: | The proposed model can achieve high performance on multilingual benchmarks and on low-resource languages. |
HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretraining and fine-tuning are the dominant paradigms in natural language processing. |
| Approach: | They propose a parameter-efficient multitask learning framework that takes trainable hyper-embeddings and visual modality as input and outputs weights for different modules in a pretrained language model. |
| Outcome: | The proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. |
Towards Zero-Shot Multilingual Transfer for Code-Switched Responses (2023.acl-long)
Copied to clipboard
| Challenge: | Recent task-oriented dialog systems have had great success building English-based personal assistants, but extending these systems to a global audience may take tremendous efforts. |
| Approach: | They propose a framework that allows for efficient transfer by learning task-specific representations and encapsulating source and target language representations. |
| Outcome: | The proposed framework is able to successfully transfer language knowledge even when the target language corpus is limited. |
One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging for Cross-Lingual Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for zero-shot cross-lingual transfer are unreliable due to the lack of pretraining data. |
| Approach: | They propose to accumulatively average model snapshots from different runs into a single model. |
| Outcome: | The proposed protocol decouples performance maximization from hyperparameter tuning. |
Zero-shot Cross-lingual Transfer With Learned Projections Using Unlabeled Target-Language Data (2023.acl-short)
Copied to clipboard
| Challenge: | Zero-shot cross-lingual transfer is enabled by pairing the language adapter in the target language with an appropriate task adapter within a source language. |
| Approach: | They propose to use unlabeled text to enhance zero-shot transfer by pairing language adapters with task adapters in a target language. |
| Outcome: | The proposed framework improves on three cross-lingual tasks with up to 11% relative improvement in Named Entity Recognition (NER), Question Answering (QA) and Natural Language Inference (NLI). |