Hyperdecoders: Instance-specific decoders for multi-task NLP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in NLP has examined the performance of large pretrained transformer-based models in multi-task settings, where a single model is evaluated on multiple tasks simultaneously. |
| Approach: | They propose a method for multi-tasking using a hypernetwork conditioned on the output of an encoder to generate a unique decoder adaptation for every input instance. |
| Outcome: | The proposed method outperforms previous methods for efficient multi-task fine-tuning and maps from encoder representations to output labels. |
Similar Papers
Example-based Hypernetworks for Multi-source Adaptation to Unseen Domains (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In order to achieve unprecedented performance, many out-of-distribution generalization approaches use unlabeled data from the target distribution. |
| Approach: | They propose a framework that leverages labeled data from multiple source domains to generalize to unknown target domains at training. |
| Outcome: | The proposed framework outperforms existing models in two tasks, and it is compared to few-shot GPT-3. |
Learning to Generate Task-Specific Adapters from Task Description (2021.acl-short)
Copied to clipboard
| Challenge: | Pre-trained text-to-text transformers have achieved impressive performance across a range of NLP tasks, such as question answering and commonsense reasoning. |
| Approach: | They propose a framework that improves text-to-text transformer’s generalization ability to unseen tasks by training a hypernetwork to generate task-specific adapters from task descriptions. |
| Outcome: | Experiments on ZEST and a synthetic SQuAD dataset show that Hypter improves upon fine-tuning baselines. |
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. |
| Approach: | They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks. |
| Outcome: | The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task. |
Tied Multitask Learning for Neural Speech Translation (N18-1)
Copied to clipboard
| Challenge: | Recent efforts in endangered language documentation focus on collecting spoken language resources . BULB project uses mobile app to collect spoken resources accompanied by spoken translations . |
| Approach: | They propose a model where the second task decoder receives information from the first task . they apply regularization that encourages transitivity and invertibility to the model . |
| Outcome: | The proposed model improves performance on low-resource speech transcription and translation tasks. |
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022.emnlp-main)
Copied to clipboard
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, Xudong Shen
| Challenge: | a benchmark of 1,616 diverse NLP tasks and their expert-written instructions is used to test generalization of models to unseen tasks . a recent study shows that instruction-following models outperform instruction-based models by over 9% . |
| Approach: | They build a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. |
| Outcome: | The proposed model outperforms existing instruction-following models by over 9% on the benchmark despite being smaller. |
HyperLoRA: Efficient Cross-task Generalization via Constrained Low-Rank Adapters Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to adapt pre-trained language models (PLMs) to emerging tasks are costly and inefficient. |
| Approach: | They propose a meta-network that generates task-specific weights without any optimization. |
| Outcome: | The proposed approach has flexible generalization ability and superior performance over hypenetworks. |
HYPERTTS: Parameter Efficient Adaptation in Text to Speech Using Hypernetworks (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural text-to-speech (TTS) systems limited to predefined speaker styles or specific sets of speaker IDs. |
| Approach: | They propose a network that can adapt adapter parameters to new speakers . they compare two domain adaptation settings and find it to be very efficient . |
| Outcome: | The proposed Adapters improve speech synthesis performance on two domains and compare them with baselines. |
Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual models cannot fully leverage training data when it is available in different task-language combinations. |
| Approach: | They propose a single hypernetwork that unifies multi-task and multilingual learning with efficient adaptation. |
| Outcome: | The proposed model achieves the best or competitive gain when a mixture of multiple resources is available while being significantly more efficient than existing models. |
Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing grammar-constrained decoding methods are limited to specific tasks . a grammar constraint is used to control the generation of LMs, but it is limited to a few tasks a task is not performed. |
| Approach: | They propose grammar-constrained decoding to control the generation of large language models . they demonstrate that grammars can describe the output space for a wider range of tasks . |
| Outcome: | The proposed grammars outperform unconstrained models on information extraction, entity disambiguation, and constituency parsing. |
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models are currently used for many NLP tasks . however, inference for these models requires significant computational resources . |
| Approach: | They propose to use a shared text encoder to amortize the computational cost of inference over multiple tasks. |
| Outcome: | The proposed method reduces the size of the extracted representations by a factor of 16 to store them for later use. |