MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehension (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing soft prompt methods focus on designing the input-independent prompts that steer the model to fit the domain of the new dataset. |
| Approach: | They propose a multi-level prompt tuning method that utilizes prompts at task-specific, domain-specific and context-specific levels to enhance the comprehension of input semantics. |
| Outcome: | The proposed method improves on 12 benchmarks on various QA formats and achieves an average improvement of 1.94% over the state-of-the-art methods. |
Similar Papers
APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, Dongfang Liu
| Challenge: | Existing prompt tuning methods only introduce prompts at the input layer, limiting performance and leaving large room for improvement. |
| Approach: | They propose a method that involves tuning a small set of soft prompts for pre-trained language models. |
| Outcome: | The proposed method outperforms state-of-the-art methods with pre-trained models on the SuperGLUE benchmark. |
IAPT: Instance-Aware Prompt Tuning for Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for prompt tuning require many soft tokens to guarantee performance . large language models still require a large amount of GPU memory and computations to fine-tune . |
| Approach: | They propose to use a parameter-efficient soft prompt generator to generate idiosyncratic soft prompts for each input instruction. |
| Outcome: | The proposed method outperforms the baselines with comparable tunable parameters and is more efficient than LoRA under the single-backbone multi-tenant setting. |
XPrompt: Exploring the Extreme of Prompt Tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prompt tuning learns soft prompts to condition pre-trained Language Models for performing downstream tasks in a parameter-efficient manner. |
| Approach: | They propose a Prompt tuning model with an eXtremely small scale that learns soft prompts to condition the frozen Pre-trained Language Models for performing downstream tasks in a parameter-efficient manner. |
| Outcome: | The proposed model outperforms the vanilla Prompt-Tuning and can significantly improve across tasks and model scales. |
The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. |
| Approach: | They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. |
| Outcome: | The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters. |
Position Really Matters: Towards a Holistic Approach for Prompt Tuning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Prompt tuning is effective in extracting knowledge from foundation models, but its effectiveness is uncertain. |
| Approach: | They propose a parametric prompt tuning strategy that dynamically determines different factors of prompts based on specific tasks or instances. |
| Outcome: | The proposed approach improves performance across a wide range of tasks including NLP, vision recognition, and vision-language tasks. |
Prompt Compression for Large Language Models: A Survey (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for improving LLM efficiency focus on optimizing the model itself, while prompt-centric methods focus on lowering the complexity of input. |
| Approach: | They propose to use prompt compression to optimize the compression encoder and combine hard and soft prompt methods to improve the efficiency of LLMs. |
| Outcome: | The proposed methods are categorized into hard prompt methods and soft prompt methods. |
The Power of Prompt Tuning for Low-Resource Semantic Parsing (2022.acl-short)
Copied to clipboard
| Challenge: | Prompt tuning is an effective method for adapting pre-trained language models to downstream tasks. |
| Approach: | They propose to use prompt tuning for semantic parsing to map natural language utterances onto formal meaning representations. |
| Outcome: | The proposed method outperforms the fine-tuned model on low-resource splits of Overnight and TOPv2 on language representations with increasing model scale and target representations. |
SMoP: Towards Efficient and Effective Prompt Tuning with Sparse Mixture-of-Prompts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prompt tuning has emerged as a successful parameter-efficient alternative to the full fine-tuning of language models. |
| Approach: | They propose a prompt tuning method that utilizes short soft prompts for efficient training and inference while maintaining performance gains typically induced by longer soft prompt. |
| Outcome: | The proposed method outperforms baseline methods while preserving memory usage. |
StablePT : Towards Stable Prompting for Few-shot Learning via Input Separation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on prompt tuning have shown that language models can be effective few-shot learners with prompting. |
| Approach: | They propose to treat the hard prompt and soft prompt as separate inputs to mitigate noise brought by prompt initialization. |
| Outcome: | Experimental results show that the proposed method outperforms state-of-the-art methods by 6.97% in accuracy and reduces the standard deviation by 1.92 on average. |
Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RL (2024.acl-long)
Copied to clipboard
Yunseon Choi, Sangmin Bae, Seonghyun Ban, Minchan Jeong, Chuheng Zhang, Lei Song, Li Zhao, Jiang Bian, Kee-Eung Kim
| Challenge: | Prompt tuning is an important technique for directing model behaviors and eliciting desired responses. |
| Approach: | They propose to find optimal prompt tokens using soft Q-learning to optimize models for prompt tuning. |
| Outcome: | The proposed method improves on baseline prompt tuning, and the results are more natural and interpretable. |