Xiaodi Sun, Sunny Rajagopalan, Priyanka Nigam, Weiyi Lu, Yi Xu, Iman Keivanloo, Belinda Zeng, Trishul Chilimbi
| Challenge: | Recent research shows that large language models pretrained using unsupervised approaches can achieve significant performance improvement on many downstream tasks. |
| Approach: | They propose an unsupervised approach to fine-tuning large language models using unsupervised approaches to many downstream tasks. |
| Outcome: | The proposed approach improves on four e-commerce applications and can achieve an average improvement of 10% in few-shot settings and 3.7% in data-rich settings over the standard approach. |
Similar Papers
PPT: Pre-trained Prompt Tuning for Few-shot Learning (2022.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning for pre-trained language models has shown remarkable performance . however, prompt tuning is still not fully explored . |
| Approach: | They propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization. |
| Outcome: | The proposed framework outperforms full-model tuning under full-data and few-shot learning settings. |
The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. |
| Approach: | They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. |
| Outcome: | The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters. |
Towards Unified Prompt Tuning for Few-shot Text Classification (2022.findings-emnlp)
Copied to clipboard
Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang, Ming Gao
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained Language Models (PLMs) on few-shot text classification, but PLMs are unfamiliar with prompt-style expressions during pre-training, which limits the few- shot learning performance on downstream tasks. |
| Approach: | They propose a framework for prompt-based fine-tuning that captures prompting semantics from non-target NLP datasets and propose 'Prompt-Options-Verbalizer' for joint prompt learning across different NLP tasks. |
| Outcome: | Experiments show that the proposed framework outperforms state-of-the-art prompt-based fine-tuning frameworks on few-shot text classification tasks. |
Visual Prompt Tuning for Few-Shot Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work on pretraining models for text classification uses image encoders instead of visual prompts. |
| Approach: | They propose a method to deploy large-scale pre-trained models in the prompt-tuning paradigm in few-shot learning. |
| Outcome: | The proposed method outperforms the most recent prompt-tuning methods on five public text classification datasets. |
A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent few-shot learning models such as GPT3 are expensive and slow to deploy for real-world applications. |
| Approach: | They propose a prompt-based low-resource learning method for VL tasks with a few examples . they pre-train a sequence-to-sequence transformer model with prefix and masked language modeling . |
| Outcome: | The proposed method outperforms Frozen on vision-language tasks with prompt-based learning by 18.2% point. |
SMASH: Improving SMAll Language Models’ Few-SHot Ability with Prompt-Based Distillation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale language models with prompts have shown remarkable performance on few-shot learning. |
| Approach: | They propose an approach to improve SMAll language models’ few-SHot ability by training on intermediate tasks before prompt-based fine-tuning on downstream tasks. |
| Outcome: | The proposed model improves on sentence-pair and sentiment classification tasks by training on intermediate tasks before fine-tuning on downstream tasks. |
Prompt Tuning for Discriminative Pre-trained Language Models (2022.findings-acl)
Copied to clipboard
Yuan Yao, Bowen Dong, Ao Zhang, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Leyu Lin, Maosong Sun, Jianyong Wang
| Challenge: | Recent studies have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing tasks. |
| Approach: | They propose a prompt tuning framework that reformulates NLP tasks into a discriminative language modeling problem. |
| Outcome: | The proposed framework improves on text classification and question answering tasks and prevents unstable tuning problems in low-resource settings. |
Prompt Tuning for Few-shot Relation Extraction via Modeling Global and Local Graphs (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies show that prompt-tuning is effective for few-shot relation extraction tasks. |
| Approach: | They propose to incorporate the knowledge in relation labels into prompt-tuning by inserting prompt templates into the input. |
| Outcome: | The proposed method improves on four datasets under low-resource conditions. |
PrAd: Prompt Adaptive Tuning for Decoder-only Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt-based methods suffer from increased input lengths and sensitivity to weight initialization . adapter-based approaches can substantially increase inference time . |
| Approach: | a new paradigm for prompt-based tuning addresses the problem of fine tuning pretrained models . prompt--based methods suffer from increased input lengths and sensitivity to weight initialization . a prompt-oriented approach employs adapters for flexible input transformation . |
| Outcome: | a proposed framework can achieve comparable or better performance and higher inference efficiency even in multi-task scenarios. |
Prompt-free and Efficient Few-shot Learning with Language Models (2022.acl-long)
Copied to clipboard
Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, Majid Yazdani
| Challenge: | Existing methods for few-shot fine-tuning of pretrained language models require carefully engineered prompts and verbalizers to convert inputs into a cloze-format that the PLM can score. |
| Approach: | They propose a method for few-shot fine-tuning of pretrained language models that uses task-specific adapters instead of manually engineered prompts and verbalizers. |
| Outcome: | The proposed method outperforms existing state-of-the-art methods on a wide range of few shot NLP tasks. |