What Makes Pre-trained Language Models Better Zero-shot Learners? (2023.acl-long)
Copied to clipboard
| Challenge: | Current methods for prompt learning in zero-shot scenarios rely on a development set with sufficient human-annotated data to select the best-performing prompt template. |
| Approach: | They propose a method for screening reasonable prompt templates in zero-shot text classification using language discrepancy. |
| Outcome: | The proposed method improves prediction performance in a realistic zero-shot setting, eliminating the need for labelled examples. |
Similar Papers
Demystifying Prompts in Language Models via Perplexity Estimation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models can be prompted to perform a wide variety of tasks with zero- and few-shot learning. |
| Approach: | They propose a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. |
| Outcome: | The proposed method extends a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. |
Pre-trained Language Models Can be Fully Zero-Shot Learners (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to pre-trained language models require fine-tuning on labeled datasets or manually constructing proper prompts. |
| Approach: | They propose a nonparametric prompting PLM for fully zero-shot language understanding . they compare it to previous methods for text classification and text entailment . |
| Outcome: | The proposed method outperforms previous methods on diverse tasks. |
Prompt-based Zero-shot Text Classification with Conceptual Knowledge (2023.acl-srw)
Copied to clipboard
| Challenge: | Existing approaches to pre-training language models rely on verbalizers to translate the predicted vocabulary to task-specific labels. |
| Approach: | They propose a framework that incorporates conceptual knowledge for text classification in the extreme zero-shot setting. |
| Outcome: | The proposed framework outperforms prompt-based approaches on four widely-used datasets for sentiment analysis and topic detection on the same experimental settings. |
Zero-shot Approach to Overcome Perturbation Sensitivity of Prompts (2023.acl-long)
Copied to clipboard
| Challenge: | Recent studies have demonstrated that natural-language prompts can help to leverage the knowledge learned by pre-trained language models for the binary sentence-level sentiment classification task. |
| Approach: | They propose to use few-shot learning settings to fine-tune the sentiment classification model using manual or automatically generated prompts. |
| Outcome: | The proposed method outperforms the base prompt and the prompts generated using few-shot learning for the binary sentence-level sentiment classification task. |
Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Prediction (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for zero-shot text classification lack prompt engineering due to prompt brittleness . however, these methods are not effective for zero shot text classifications . |
| Approach: | They propose a method that predicts token probabilities across multiple positions and simulates comprehensive sampling of generation paths in a single run of a language model. |
| Outcome: | The proposed approach improves accuracy and reduces standard deviation by 98% . it maintains comparable performance even without a prompt, reducing the need for prompt engineering . |
Making Pre-trained Language Models Better Few-shot Learners (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that the GPT-3 model can perform few-shots on language understanding tasks with a natural-language prompt and a few task demonstrations. |
| Approach: | They propose a technique for fine-tuning language models using a few examples . they propose LM-BFF, which uses prompt-based fine-uning and a pipeline for automating prompt generation . |
| Outcome: | The proposed approach outperforms standard fine-tuning procedures on a range of NLP tasks. |
Don’t Prompt, Search! Mining-based Zero-Shot Learning with Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has obtained strong zero-shot results by prompting language models. |
| Approach: | They propose a mining-based approach that uses regular expressions to mine labeled examples from unlabeled corpora and fine tune a pretrained model. |
| Outcome: | The proposed method outperforms prompting on a wide range of tasks when using comparable templates. |
Are Prompt-based Models Clueless? (2022.acl-long)
Copied to clipboard
| Challenge: | Prompting has reduced the data requirement by reusing the language model head and formatting the task input to match the pre-training objective. |
| Approach: | They propose to examine whether few-shot prompt-based models exploit superficial cues by reusing the model head and formatting the input to match the pre-training objective. |
| Outcome: | The proposed models perform well on instances with superficial cues, but often outperform random accuracy on instances without superficial cuing. |
Correcting Language Model Bias for Text Classification in True Zero-Shot Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Experimental results show that pre-trained language models outperform standard prompt learning in zero-shot settings. |
| Approach: | They propose a pipeline for annotating and filtering examples from unlabeled examples . they propose 'model bias validation' method that utilizes unlabed examples as validation set . |
| Outcome: | The proposed approach outperforms standard prompt learning on six text classification tasks. |
Beyond prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for zero-shot text classification involve heavy human engineering or complicated self-training pipelines. |
| Approach: | They propose to fit unlabeled text with a Bayesian Gaussian Mixture Model and use class names to cluster them. |
| Outcome: | The proposed approach outperforms prompt-based methods on topic and sentiment datasets and outperformed previous studies significantly on unbalanced datasets. |