Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models have shown promising results for few-shot learning in task-oriented dialog (ToD) systems. |
| Approach: | They propose a self-training approach that iteratively labels the most confident unlabeled data to train a stronger Student model. |
| Outcome: | The proposed approach improves state-of-the-art pre-trained models in few-shot learning scenarios for task-oriented dialog (ToD) systems when only a small number of labeled data are available. |
Similar Papers
Revisiting Self-training for Few-shot Learning of Language Model (2021.emnlp-main)
Copied to clipboard
| Challenge: | Unlabeled data are useful for few-shot learning of language models. |
| Approach: | They propose a prompt-based few-shot learner that uses unlabeled data to fine-tune language models. |
| Outcome: | The proposed approach outperforms state-of-the-art models on six sentence classification and six sentence-pair classification benchmarking tasks. |
TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models with self-attention encoder architectures are less useful in practice. |
| Approach: | They propose to use user and system tokens to model dialogue behavior during pre-training . they propose a contrastive objective function to simulate the response selection task . |
| Outcome: | The proposed model outperforms baseline models on four downstream tasks . it also has a few-shot ability that can mitigate the data scarcity problem . |
Self-training Improves Pre-training for Natural Language Understanding (2021.naacl-main)
Copied to clipboard
Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, Alexis Conneau
| Challenge: | Unsupervised pretraining has led to improvements in natural language understanding . a data augmentation method can be used to generate labels for unlabeled examples . |
| Approach: | They propose a semi-supervised method which uses unlabeled data to retrieve sentences from a database of billions of unlabed sentences crawled from the web. |
| Outcome: | The proposed method improves on standard text classification benchmarks by 2.6% and knowledge distillation by few shots. |
Few Shot Rationale Generation using Self-Training with Dual Teachers (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing models that generate free-text explanations for annotated labels are expensive and require a large annotation dataset. |
| Approach: | They propose a self-training approach leveraging both labeled and unlabeled data to further improve few-shot models by combining teacher models and a multi-tasking student model. |
| Outcome: | The proposed model improves on three public datasets and can generate a free-text explanation for predicted labels. |
Self-training with Few-shot Rationalization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work focused on training largescale and complex neural network models, but they are opaque in terms of their decision-making process. |
| Approach: | They propose a multi-task teacher-student framework for self-training pre-trained language models with limited task-specific labels and annotated rationales. |
| Outcome: | The proposed model improves performance in low-resource settings by making it aware of its rationalized predictions. |
DS-TOD: Efficient Domain Specialization for Task-Oriented Dialog (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that self-supervised dialog-specific pretraining on large conversational datasets yields substantial gains over traditional language modeling (LM) pretraining. |
| Approach: | They propose a resource-efficient and modular domain specialization by means of domain adapters in which domain knowledge is encoded. |
| Outcome: | The proposed framework extracts domain-specific terms and then uses them to build DomainCC and DomainReddit resources based on masked language modeling and response selection objectives. |
Effectiveness of Pre-training for Few-shot Intent Classification (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing paradigms further pre-train language models such as BERT on vast amount of unlabeled corpus, but we find it highly effective and efficient to simply fine-tune BERT with roughly 1,000 labeled utterances from public datasets. |
| Approach: | They propose to fine-tune BERT with a small set of labeled utterances from public datasets to achieve a pre-trained model based on a set of 1,000 labeles. |
| Outcome: | The proposed model can outperform existing models on domains with very different semantics on novel domains. |
Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing approaches to fine-tune pre-trained language models for downstream tasks require labeled data. |
| Approach: | They propose to self-train pre-trained language models to improve performance on data-scarce varieties by as large as 10% F1 and 2% accuracy. |
| Outcome: | The proposed model improves zero-shot MSA-to-DA transfer by as large as 10% F1 (NER) and 2% accuracy (POS tagging). |
A Unified Dialogue User Simulator for Few-shot Data Augmentation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to augment large-scale task-oriented dialogues rely on annotated data. |
| Approach: | They propose to build a unified dialogue user simulation model by pre-training on publicly available datasets. |
| Outcome: | The proposed model can be tuned on a target domain with few-shot data. |
An Improved, Strong Baseline for Pre-Trained Large Language Models as Task-Oriented Dialogue Systems (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that Large Language Models perform insufficiently as TOD systems. |
| Approach: | They propose a self-checking mechanism to improve LLM performance as TOD systems. |
| Outcome: | The proposed model outperforms existing models and improves their performance. |