On the Relationship between Skill Neurons and Robustness in Prompt Tuning (2024.lrec-main)
Copied to clipboard
| Challenge: | Prompt Tuning is a parameter-efficient finetuning method for pre-trained large language models (PLMs). |
| Approach: | They propose to use RoBERTa to fine tune pre-trained large language models by finetuning only a small set of parameters to adjust for downstream tasks. |
| Outcome: | The proposed method activates specific neurons in the transformer’s feed-forward networks that are highly predictive and selective for the given task. |
Similar Papers
Model-tuning Via Prompts Makes NLP Models Adversarially Robust (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models are typically adapted to downstream tasks by appending a randomly initialized multilayer perceptron to their topmost representation layer and fine-tuning the entire model on a downstream task. |
| Approach: | They propose to append a multilayer perceptron to a CLS token and fine-tune the entire model on a downstream task. |
| Outcome: | The proposed model-tuning via prompts outperforms adversarial training-based state-of-art defenses by 3.5% and improves against adversarials by 8% over standard methods. |
On Transferability of Prompt Tuning for Natural Language Processing (2022.naacl-main)
Copied to clipboard
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, Jie Zhou
| Challenge: | Pre-trained language models (PLMs) can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but require much more training time than fine-timing. |
| Approach: | They empirically investigate the transferability of soft prompts across different downstream tasks and PLMs to determine what decides prompt transferability. |
| Outcome: | The proposed method can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but requires much more training time than fine-timing. |
PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing prompt tuning methods have training instability issues due to large variance of scores . existing prompt tuning algorithms have training stability issues due a slight change of input data . |
| Approach: | They propose an algorithm that smooths the loss landscape of vanilla prompt tuning by perturbation-based regularizers. |
| Outcome: | The proposed method improves the state-of-the-art prompt tuning methods by 1.94% and 2.34% on SuperGLUE and FewGLUE benchmarks. |
The Power of Prompt Tuning for Low-Resource Semantic Parsing (2022.acl-short)
Copied to clipboard
| Challenge: | Prompt tuning is an effective method for adapting pre-trained language models to downstream tasks. |
| Approach: | They propose to use prompt tuning for semantic parsing to map natural language utterances onto formal meaning representations. |
| Outcome: | The proposed method outperforms the fine-tuned model on low-resource splits of Overnight and TOPv2 on language representations with increasing model scale and target representations. |
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks (2022.acl-short)
Copied to clipboard
| Challenge: | Existing methods of prompt tuning cannot handle hard sequence labeling tasks. |
| Approach: | They propose to optimize prompt tuning to tune continuous prompts with a frozen language model. |
| Outcome: | The proposed method matches finetuning with prompt tuning while having only 0.1%-3% tuned parameters. |
Finding Skill Neurons in Pre-trained Transformer-based Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have demonstrated superior performance on various natural language processing tasks. |
| Approach: | They find that after prompt tuning, some neurons encode task-specific skills . they also show that skill neurons are most likely generated in pre-training . |
| Outcome: | The neurons are highly predictive of task labels after prompt tuning for specific tasks. |
The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. |
| Approach: | They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. |
| Outcome: | The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters. |
Prompt Tuning for Discriminative Pre-trained Language Models (2022.findings-acl)
Copied to clipboard
Yuan Yao, Bowen Dong, Ao Zhang, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Leyu Lin, Maosong Sun, Jianyong Wang
| Challenge: | Recent studies have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing tasks. |
| Approach: | They propose a prompt tuning framework that reformulates NLP tasks into a discriminative language modeling problem. |
| Outcome: | The proposed framework improves on text classification and question answering tasks and prevents unstable tuning problems in low-resource settings. |
Residual Prompt Tuning: improving prompt tuning with residual reparameterization (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt tuning is one of the most parameter-efficient approaches for parameter-effective tuning of pre-trained language models. |
| Approach: | They propose to reparameterize soft prompt embeddings using a shallow network with a residual connection and use it to tune prompt embeds P. |
| Outcome: | The proposed method outperforms prompt tuning on SuperGLUE, T5-Base and BERT-Bass models and can reduce the prompt length by 10 times without hurting performance. |
XPrompt: Exploring the Extreme of Prompt Tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prompt tuning learns soft prompts to condition pre-trained Language Models for performing downstream tasks in a parameter-efficient manner. |
| Approach: | They propose a Prompt tuning model with an eXtremely small scale that learns soft prompts to condition the frozen Pre-trained Language Models for performing downstream tasks in a parameter-efficient manner. |
| Outcome: | The proposed model outperforms the vanilla Prompt-Tuning and can significantly improve across tasks and model scales. |