StablePT : Towards Stable Prompting for Few-shot Learning via Input Separation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on prompt tuning have shown that language models can be effective few-shot learners with prompting. |
| Approach: | They propose to treat the hard prompt and soft prompt as separate inputs to mitigate noise brought by prompt initialization. |
| Outcome: | Experimental results show that the proposed method outperforms state-of-the-art methods by 6.97% in accuracy and reduces the standard deviation by 1.92 on average. |
Similar Papers
Hard Sample Aware Prompt-Tuning (2023.acl-long)
Copied to clipboard
| Challenge: | Prompt-tuning based few-shot learning has garnered increasing attention in recent years due to its efficiency and promising capability. |
| Approach: | They propose a framework to distinguish informative hard samples from misleading ones in model training. |
| Outcome: | The proposed framework achieves new SOTA results on a series of NLP tasks pushing the SST-5 accuracy to 49.5% (1.1% point absolute improvement), QNLI accuracy to 74.6% (1.9% absolute improvement) |
MEAL: Stable and Active Learning for Few-Shot Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for few-shot classification have high variance across different sets of few shots and finetuning runs. |
| Approach: | They propose novel ensembling methods that significantly reduce run variability and introduce a new active learning criterion for *data selection*. |
| Outcome: | The proposed method significantly reduces run variability and improves performance on five tasks. |
PPT: Pre-trained Prompt Tuning for Few-shot Learning (2022.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning for pre-trained language models has shown remarkable performance . however, prompt tuning is still not fully explored . |
| Approach: | They propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization. |
| Outcome: | The proposed framework outperforms full-model tuning under full-data and few-shot learning settings. |
PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization (2022.coling-1)
Copied to clipboard
| Challenge: | Experimental results show that our method outperforms full-model tuning in few-shot abstractive summarization tasks. |
| Approach: | They propose a soft prompts architecture with prompt pre-training and prompt fine-tuning paradigm to support few-shot abstractive summarization. |
| Outcome: | The proposed model outperforms Prompt Tuning and Profix-Tuning on CNN/DailyMail and XSum datasets and outperfies Profix Tuning by a large margin. |
Making Pre-trained Language Models Better Few-shot Learners (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that the GPT-3 model can perform few-shots on language understanding tasks with a natural-language prompt and a few task demonstrations. |
| Approach: | They propose a technique for fine-tuning language models using a few examples . they propose LM-BFF, which uses prompt-based fine-uning and a pipeline for automating prompt generation . |
| Outcome: | The proposed approach outperforms standard fine-tuning procedures on a range of NLP tasks. |
Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent few-shot learning methods focus on improving downstream task performance, but there is limited understanding of the adversarial robustness of such methods. |
| Approach: | They evaluate prompt-based FSL methods against fully fine-tuned models to better understand the impact of various factors towards robustness. |
| Outcome: | The proposed methods show that they are less robust in the face of adversarial perturbations than fully fine-tuned models. |
MetaPrompting: Learning to Learn Better Prompts (2022.coling-1)
Copied to clipboard
| Challenge: | Recent research on prompting moves from discrete tokens based "hard prompts" to continuous "soft prompts", which employ learnable vectors as pseudo prompt tokens and achieve better performance. |
| Approach: | They propose a generalized soft prompting method that uses model-agnostic meta-learning to find better initialization for soft prompts. |
| Outcome: | The proposed method improves on three datasets and brings new state-of-the-art performance. |
The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. |
| Approach: | They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. |
| Outcome: | The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters. |
True Few-Shot Learning with Prompts—A Real-World Perspective (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work has cast doubt on the effectiveness of prompt-based approaches at few-shot learning in a “true” few- shot setting. |
| Approach: | They propose a method that combines textual instructions with example-based finetuning to give prompt-based learning a powerful method for few-shot text classification. |
| Outcome: | The proposed method performs well in a few-shot setting without a dev set and is able to handle multiple prompts. |
PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing prompt tuning methods have training instability issues due to large variance of scores . existing prompt tuning algorithms have training stability issues due a slight change of input data . |
| Approach: | They propose an algorithm that smooths the loss landscape of vanilla prompt tuning by perturbation-based regularizers. |
| Outcome: | The proposed method improves the state-of-the-art prompt tuning methods by 1.94% and 2.34% on SuperGLUE and FewGLUE benchmarks. |