Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for prompt tuning can overfit to few-shot training samples, causing overfitting . authors propose a new framework for prompt learning with supervised meta-learning . |
| Approach: | They propose a self-supervised meta-prompt learning framework with MEta-gradient Regularization for few-shot generalization that leverages self-recognized meta-learning with a diverse set of meta-tasks to learn a universal prompt initialization using only unlabeled data. |
| Outcome: | The proposed framework learns a universal prompt initialization for efficient adaptation using only unlabeled data. |
Similar Papers
Prompt-Based Meta-Learning For Few-shot Text Classification (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn text labels require large amounts of data to build many few-shot tasks. |
| Approach: | They propose a Prompt-Based Meta-Learning model that adds the prompting mechanism to the meta-learning method. |
| Outcome: | The proposed method improves on four text classification datasets with high accuracy and robustness. |
PPT: Pre-trained Prompt Tuning for Few-shot Learning (2022.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning for pre-trained language models has shown remarkable performance . however, prompt tuning is still not fully explored . |
| Approach: | They propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization. |
| Outcome: | The proposed framework outperforms full-model tuning under full-data and few-shot learning settings. |
MetaPrompting: Learning to Learn Better Prompts (2022.coling-1)
Copied to clipboard
| Challenge: | Recent research on prompting moves from discrete tokens based "hard prompts" to continuous "soft prompts", which employ learnable vectors as pseudo prompt tokens and achieve better performance. |
| Approach: | They propose a generalized soft prompting method that uses model-agnostic meta-learning to find better initialization for soft prompts. |
| Outcome: | The proposed method improves on three datasets and brings new state-of-the-art performance. |
Towards Unified Prompt Tuning for Few-shot Text Classification (2022.findings-emnlp)
Copied to clipboard
Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang, Ming Gao
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained Language Models (PLMs) on few-shot text classification, but PLMs are unfamiliar with prompt-style expressions during pre-training, which limits the few- shot learning performance on downstream tasks. |
| Approach: | They propose a framework for prompt-based fine-tuning that captures prompting semantics from non-target NLP datasets and propose 'Prompt-Options-Verbalizer' for joint prompt learning across different NLP tasks. |
| Outcome: | Experiments show that the proposed framework outperforms state-of-the-art prompt-based fine-tuning frameworks on few-shot text classification tasks. |
Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning? (2023.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning (PT) based on frozen pre-trained language models has shown remarkable performance in few-shot learning . however, it relies heavily on good initialization of the prompt embeddings. |
| Approach: | They propose to use meta prompt tuning to improve cross-task generalization by learning to initialize prompt embeddings from other relevant tasks. |
| Outcome: | The proposed method outperforms PT on classification tasks, but not multi-task learning. |
Adaptive Meta-learner via Gradient Similarity for Few-shot Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for few-shot text classification suffer from overfitting due to the lack of matching between the few amount of samples and complicated models. |
| Approach: | They propose a method to improve model generalization ability to a new task by leveraging a meta-learner via gradient similarity method. |
| Outcome: | The proposed method improves few-shot text classification performance on several benchmarks. |
The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. |
| Approach: | They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. |
| Outcome: | The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters. |
Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning is a parameter-efficient approach to adapting pre-trained language models to downstream tasks. |
| Approach: | They propose to combine pre-trained modules with pre-trains to boost prompt tuning for few-shot learning. |
| Outcome: | The proposed model outperforms prompt tuning, full model tuning, and prior prompt pre-training methods in few-shot learning settings. |
True Few-Shot Learning with Prompts—A Real-World Perspective (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work has cast doubt on the effectiveness of prompt-based approaches at few-shot learning in a “true” few- shot setting. |
| Approach: | They propose a method that combines textual instructions with example-based finetuning to give prompt-based learning a powerful method for few-shot text classification. |
| Outcome: | The proposed method performs well in a few-shot setting without a dev set and is able to handle multiple prompts. |
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners (2022.findings-naacl)
Copied to clipboard
| Challenge: | LiST is an efficient method for fine-tuning large pre-trained language models in few-shot learning settings. |
| Approach: | They propose a method for efficient fine-tuning of large pre-trained language models in few-shot settings using self-training and meta-learning. |
| Outcome: | The proposed method outperforms GPT-3 in-context learning by 33% on few-shot tasks. |