ZARA: Improving Few-Shot Self-Rationalization for Small Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies demonstrate great performance gain for self-rationalization by few-shot prompting LMs with rationale-augmented exemplars. |
| Approach: | They propose to leverage explanations for small LMs to improve few-shot self-rationalization by reducing the problem of plausibility judgement to natural language inference. |
| Outcome: | The proposed approach achieves SOTA performance on the FEB benchmark, for both the task accuracy and the explanation metric. |
Similar Papers
Self-training with Few-shot Rationalization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work focused on training largescale and complex neural network models, but they are opaque in terms of their decision-making process. |
| Approach: | They propose a multi-task teacher-student framework for self-training pre-trained language models with limited task-specific labels and annotated rationales. |
| Outcome: | The proposed model improves performance in low-resource settings by making it aware of its rationalized predictions. |
Few-Shot Self-Rationalization with Natural Language Prompts (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing models that generate free-text explanations for tasks are limited by human-written explanations. |
| Approach: | They propose to use a standardized collection of natural language prompts to create a model that generates free-text explanations for tasks. |
| Outcome: | The proposed model can predict task labels and generate free-text explanations for predictions . plausibility of human explanations is 76%, while human explanation is 51% . |
Few Shot Rationale Generation using Self-Training with Dual Teachers (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing models that generate free-text explanations for annotated labels are expensive and require a large annotation dataset. |
| Approach: | They propose a self-training approach leveraging both labeled and unlabeled data to further improve few-shot models by combining teacher models and a multi-tasking student model. |
| Outcome: | The proposed model improves on three public datasets and can generate a free-text explanation for predicted labels. |
Making Pre-trained Language Models Better Few-shot Learners (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that the GPT-3 model can perform few-shots on language understanding tasks with a natural-language prompt and a few task demonstrations. |
| Approach: | They propose a technique for fine-tuning language models using a few examples . they propose LM-BFF, which uses prompt-based fine-uning and a pipeline for automating prompt generation . |
| Outcome: | The proposed approach outperforms standard fine-tuning procedures on a range of NLP tasks. |
Zero-Shot Rationalization by Multi-Task Transfer Learning from Question Answering (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract rationales from input text are difficult and impractical. |
| Approach: | They propose a method that leverages multi-task learning and transfer learning to generate rationales through question answering in a zero-shot fashion. |
| Outcome: | The proposed method achieves comparable or even better performance without supervised signal for two benchmark rationalization datasets. |
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretraining ever-larger language models on massive corpora requires enormous amounts of compute. |
| Approach: | They propose to convert textual inputs into cloze questions that contain a task description . they also exploit unlabeled data to improve their performance . |
| Outcome: | The proposed model outperforms GPT-3 with PET/iPET with cloze questions and unlabeled data. |
Characterizing Large Language Models as Rationalizers of Knowledge-intensive Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models generate fluent text with minimal task-specific supervision, but their ability to generate rationales for knowledge-intensive tasks (KITs) remains under-explored. |
| Approach: | They propose to generate retrieval-augmented rationalization of KIT model predictions via external knowledge guidance within a few-shot setting. |
| Outcome: | The proposed rationales were compared with crowd-sourced rationale models on factuality, sufficiency, and convincingness. |
Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent prompt-based approaches allow pretrained language models to achieve strong performances on few-shot finetuning by reformulating downstream task instances as a language modeling problem. |
| Approach: | They propose to reformulate downstream tasks as a language modeling problem and add a regularization that preserves pretraining weights to the model to mitigate the destructive tendency of few-shot finetuning. |
| Outcome: | The proposed model performs better on low data regimes than the standard model on few-shot finetuning. |
Self-AMPLIFY: Improving Small Language Models with Self Post Hoc Explanations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive Large Language Models (LLMs) have demonstrated "emergent abilities" such as in-context learning, instruction following and reasoning. |
| Approach: | They propose a method that generates rationales from post hoc explanation methods applied to small language models to improve their own performance. |
| Outcome: | The proposed method improves on four SLMs and five datasets with strong reasoning abilities. |
Revisiting Self-training for Few-shot Learning of Language Model (2021.emnlp-main)
Copied to clipboard
| Challenge: | Unlabeled data are useful for few-shot learning of language models. |
| Approach: | They propose a prompt-based few-shot learner that uses unlabeled data to fine-tune language models. |
| Outcome: | The proposed approach outperforms state-of-the-art models on six sentence classification and six sentence-pair classification benchmarking tasks. |