Few-shot Unified Question Answering: Tuning Models or Prompts? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Question-answering (QA) tasks investigate specific question types, knowledge domains, or reasoning skills, leading to specialized models catering to specific categories of QA tasks. |
| Approach: | They propose to use model and prompt tuning for unified QA in a low-resource setting to overcome drawbacks of unified models. |
| Outcome: | The proposed model and prompt tuning paradigms outperform model tuning in a few-shot setting with a good initialization and achieve a significant performance boost from pre-training in 'low-resource' setting. |
Similar Papers
Towards Unified Prompt Tuning for Few-shot Text Classification (2022.findings-emnlp)
Copied to clipboard
Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang, Ming Gao
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained Language Models (PLMs) on few-shot text classification, but PLMs are unfamiliar with prompt-style expressions during pre-training, which limits the few- shot learning performance on downstream tasks. |
| Approach: | They propose a framework for prompt-based fine-tuning that captures prompting semantics from non-target NLP datasets and propose 'Prompt-Options-Verbalizer' for joint prompt learning across different NLP tasks. |
| Outcome: | Experiments show that the proposed framework outperforms state-of-the-art prompt-based fine-tuning frameworks on few-shot text classification tasks. |
FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained models need fine-tuning on tens of thousands of examples to achieve good results. |
| Approach: | They propose a framework that leverages pre-trained text-to-text models and aligns them with their pre-training framework. |
| Outcome: | The proposed framework outperforms the XLM-Roberta-large on multiple QA benchmarks and is applicable to multilingual situations. |
PPT: Pre-trained Prompt Tuning for Few-shot Learning (2022.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning for pre-trained language models has shown remarkable performance . however, prompt tuning is still not fully explored . |
| Approach: | They propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization. |
| Outcome: | The proposed framework outperforms full-model tuning under full-data and few-shot learning settings. |
Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning is a parameter-efficient approach to adapting pre-trained language models to downstream tasks. |
| Approach: | They propose to combine pre-trained modules with pre-trains to boost prompt tuning for few-shot learning. |
| Outcome: | The proposed model outperforms prompt tuning, full model tuning, and prior prompt pre-training methods in few-shot learning settings. |
ProQA: Structural Prompt-based Pre-training for Unified Question Answering (2022.naacl-main)
Copied to clipboard
Wanjun Zhong, Yifan Gao, Ning Ding, Yujia Qin, Zhiyuan Liu, Ming Zhou, Jiahai Wang, Jian Yin, Nan Duan
| Challenge: | Existing QA research on question answering is focused on specific question types, knowledge domains, or reasoning skills. |
| Approach: | They propose a unified QA paradigm that solves various tasks through a single model. |
| Outcome: | The proposed model improves QA-centric ability on 11 QA benchmarks. |
Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. |
| Approach: | They experiment with 30 prompts manually written for natural language inference (NLI) they find that models can learn just as fast with many irrelevant or pathologically misleading prompts . |
| Outcome: | The proposed model can learn as fast with irrelevant or pathologically misleading prompts as with instructively “good” prompts. |
PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization (2022.coling-1)
Copied to clipboard
| Challenge: | Experimental results show that our method outperforms full-model tuning in few-shot abstractive summarization tasks. |
| Approach: | They propose a soft prompts architecture with prompt pre-training and prompt fine-tuning paradigm to support few-shot abstractive summarization. |
| Outcome: | The proposed model outperforms Prompt Tuning and Profix-Tuning on CNN/DailyMail and XSum datasets and outperfies Profix Tuning by a large margin. |
Prompt Tuning for Unified Multimodal Pretrained Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt tuning has demonstrated success in natural language pretraining and even vision pretraining. |
| Approach: | They propose to apply prompt tuning to a unified sequence-to-sequence pretrained model by adding a sequence of learnable embeddings to each layer and finetuning the pretrained models on downstream tasks. |
| Outcome: | The proposed method outperforms other parameter-efficient tuning methods on multimodal models and is robust against adversarial attacks. |
Is Prompt Transfer Always Effective? An Empirical Study of Prompt Transfer for Question Answering (2024.naacl-short)
Copied to clipboard
| Challenge: | Prompt tuning is an efficient method for initializing pre-trained models . but initialization of prompts is sensitive when the model size is small . |
| Approach: | They propose a method to measure catastrophic forgetting by analyzing prompts for the first time . they characterize a question answering task based on answer format and prompt initialization . |
| Outcome: | The proposed approach can help deepen understanding of prompt tuning. |
Prompt Tuning for Few-shot Relation Extraction via Modeling Global and Local Graphs (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies show that prompt-tuning is effective for few-shot relation extraction tasks. |
| Approach: | They propose to incorporate the knowledge in relation labels into prompt-tuning by inserting prompt templates into the input. |
| Outcome: | The proposed method improves on four datasets under low-resource conditions. |