| Challenge: | Recent studies have found that entailment pretraining benefits weakly supervised fine-tuning. |
| Approach: | They propose a prompting strategy that formulates different NLU tasks as contextual entailment and propose an algorithm for better pseudo-labeling quality in self-training. |
| Outcome: | The proposed approach improves the zero-shot adaptation performance on downstream tasks. |
Similar Papers
Self-training with Few-shot Rationalization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work focused on training largescale and complex neural network models, but they are opaque in terms of their decision-making process. |
| Approach: | They propose a multi-task teacher-student framework for self-training pre-trained language models with limited task-specific labels and annotated rationales. |
| Outcome: | The proposed model improves performance in low-resource settings by making it aware of its rationalized predictions. |
Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing curriculum learning approaches rely on manually defined difficulty metrics which may not accurately reflect the model’s own perspective. |
| Approach: | They propose a self-adaptive curriculum learning paradigm that prioritizes fine-tuning examples based on difficulty scores predicted by pre-trained language models (PLMs) they evaluate four datasets covering binary and multi-class classification tasks. |
| Outcome: | The proposed model leads to faster convergence and improved performance compared to standard random sampling. |
Issues with Entailment-based Zero-shot Text Classification (2021.acl-short)
Copied to clipboard
| Challenge: | Pre-trained BERT models with no fine-tuning can yield competitive performance against BERT fine- tuned for NLI. |
| Approach: | They propose to use any target label into a sentence of hypothesis and verify whether it could be entailed by the input. |
| Outcome: | The proposed models perform better than models fine-tuned for BERT, but the results are in general negative. |
Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start (2020.emnlp-main)
Copied to clipboard
| Challenge: | a current approach to solving NLP problems is to build a problem-specific dataset . current approaches do not allow for transforming tasks into textual entailment . |
| Approach: | They propose a pretrained textual entailment system that can generalize across domains . they argue that when is it worth transforming an NLP task into textual detailment? |
| Outcome: | The proposed model can generalize across domains with few examples, the authors argue . they show that it can be used for several downstream NLP tasks with limited annotations . |
Pre-trained Language Models Can be Fully Zero-Shot Learners (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to pre-trained language models require fine-tuning on labeled datasets or manually constructing proper prompts. |
| Approach: | They propose a nonparametric prompting PLM for fully zero-shot language understanding . they compare it to previous methods for text classification and text entailment . |
| Outcome: | The proposed method outperforms previous methods on diverse tasks. |
A Robust Self-Learning Framework for Cross-Lingual Text Classification (D19-1)
Copied to clipboard
| Challenge: | Recent advances in pretrained contextual representation models have made significant progress on a number of different English NLP tasks. |
| Approach: | They propose a robust framework to include unlabeled non-English samples in the fine-tuning process of pretrained multilingual representation models. |
| Outcome: | The proposed framework includes unlabeled non-English samples in the fine-tuning process of pretrained multilingual representation models. |
EICO: Improving Few-Shot Text Classification via Explicit and Implicit Consistency Regularization (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for few-shot text classification are limited by labeled data. |
| Approach: | They propose to use consistency regularization to improve few-shot text classification by generating pseudo-labels from weakly-augmented and strongly-augmented views. |
| Outcome: | The proposed method achieves competitive performance with 16 labeled examples with prompt and verbalizer. |
Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent prompt-based approaches allow pretrained language models to achieve strong performances on few-shot finetuning by reformulating downstream task instances as a language modeling problem. |
| Approach: | They propose to reformulate downstream tasks as a language modeling problem and add a regularization that preserves pretraining weights to the model to mitigate the destructive tendency of few-shot finetuning. |
| Outcome: | The proposed model performs better on low data regimes than the standard model on few-shot finetuning. |
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners (2022.findings-naacl)
Copied to clipboard
| Challenge: | LiST is an efficient method for fine-tuning large pre-trained language models in few-shot learning settings. |
| Approach: | They propose a method for efficient fine-tuning of large pre-trained language models in few-shot settings using self-training and meta-learning. |
| Outcome: | The proposed method outperforms GPT-3 in-context learning by 33% on few-shot tasks. |
Prompt-based Text Entailment for Low-Resource Named Entity Recognition (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-trained Language Models (PLMs) have been applied in NLP tasks but require labeled data for downstream tasks. |
| Approach: | They propose a method for low-resource named entity recognition that uses prompts to get entailment scores for each candidate and inject tagging labels into prompts. |
| Outcome: | The proposed method achieves competitive performance on the CoNLL03 dataset, and better than fine-tuned counterparts on the MIT Movie and Few-NERD datasets in low-resource settings. |