Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large pre-trained language model can cause computational burdens in inference time due to multiple forward passes. |
| Approach: | They propose a method to learn fixed text representations with source tasks . they learn a task-specific prefix for each source task independently and combine them . |
| Outcome: | The proposed method improves generalizability of representations with source tasks. |
Similar Papers
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Fine-tuning is the prevalent paradigm for using large pretrained language models for downstream tasks, but it requires updating and storing all the parameters of the LM. |
| Approach: | They propose a lightweight alternative to fine-tuning for natural language generation tasks that optimizes a sequence of continuous vectors, which they call the prefix. |
| Outcome: | The proposed approach outperforms fine-tuning in the full data setting and extrapolates better to examples with topics that are unseen during training. |
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning only optimizes a few task-specific parameters with frozen pre-trained model. |
| Approach: | They propose to optimize a prefix vector inserted into Transformer layers to optimize the prefix . they propose to use a gate mechanism to adjust the prefixed to each layer . |
| Outcome: | The proposed approach improves on the SuperGLUE and NER datasets. |
Selective Prefix Tuning for Pre-trained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models are time-consuming and memory-inefficient. |
| Approach: | They propose a method that inserts learnable vectors into each Transformer layer . they propose SL to encourage diversity in prefix tokens . |
| Outcome: | Extensive experiments validate the effectiveness of Prefix Tuning in sentence and token classification tasks. |
Prefix Propagation: Parameter-Efficient Tuning for Long Sequences (2023.acl-short)
Copied to clipboard
| Challenge: | Prefix-tuning prepends trainable tokens to sequences while freezing the rest of the model’s parameters. |
| Approach: | They propose a method that prefixes on previous hidden states to improve model performance. |
| Outcome: | The proposed architecture outperforms prefix-tuning on long-document tasks while using 50% fewer parameters. |
Learning to Generate Task-Specific Adapters from Task Description (2021.acl-short)
Copied to clipboard
| Challenge: | Pre-trained text-to-text transformers have achieved impressive performance across a range of NLP tasks, such as question answering and commonsense reasoning. |
| Approach: | They propose a framework that improves text-to-text transformer’s generalization ability to unseen tasks by training a hypernetwork to generate task-specific adapters from task descriptions. |
| Outcome: | Experiments on ZEST and a synthetic SQuAD dataset show that Hypter improves upon fine-tuning baselines. |
Text Classification with Few Examples using Controlled Generalization (N19-1)
Copied to clipboard
| Challenge: | Current training data for text classification is limited, resulting in limited generalization capacity. |
| Approach: | They propose a feed-forward network that can generalize from unlabeled parsed corpora to produce task-specific semantic vectors. |
| Outcome: | The proposed approach is especially effective in low-data scenarios compared to state-of-the-art methods. |
Prefix-Tuning Based Unsupervised Text Style Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised text style transfer is an important task in computer vision and natural language processing. |
| Approach: | They propose a method that uses pre-trained large language models to train a generative model that can alter the style of the input sentence without using any parallel data. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on well-known datasets. |
TransCoder: Towards Unified Transferable Code Representation Learning Inspired by Human Skills (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune code intelligence models to individual tasks are costly and require large data sets. |
| Approach: | They propose a Transferable fine-tuning strategy for Code representation learning that uses a tunable prefix encoder to capture cross-task and cross-language transferable knowledge and apply it to downstream adaptation. |
| Outcome: | The proposed method can lead to superior performance on code-related tasks and encourage mutual reinforcement. |
Task-adaptive Pre-training of Language Models with Word Embedding Regularization (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models acquire domain-independent knowledge through pre-training with massive textual resources. |
| Approach: | They propose a task-adaptive pre-training process that makes static embeddings close to the word embedds obtained in the target domain. |
| Outcome: | The proposed process improves on BioASQ and SQuAD when the pre-training corpora were not dominated by indomain data. |
Task Compass: Scaling Multi-task Pre-training with Task Prefix (2022.findings-emnlp)
Copied to clipboard
Zhuosheng Zhang, Shuohang Wang, Yichong Xu, Yuwei Fang, Wenhao Yu, Yang Liu, Hai Zhao, Chenguang Zhu, Michael Zeng
| Challenge: | Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. |
| Approach: | They propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. |
| Outcome: | The proposed model can be used as a foundation backbone for a wide range of tasks and as augmentation tool for data augmentation with complementary tasks. |