| Challenge: | Existing approaches to generate training data with pre-trained language models have been found effective in various scenarios. |
| Approach: | They propose an unsupervised zero-shot learning method that generates a dataset from scratch and trains a tiny task model under supervision of the synthesized dataset. |
| Outcome: | The proposed method is annotated-free and efficient, but can provide useful insights from the perspective of data-free model-agnostic knowledge distillation and unreferenced text generation evaluation. |
Similar Papers
ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on dataset-generation-based zero-shot learning has shown promising results by training a task-specific model with a dataset synthesized from large pre-trained language models (PLMs). |
| Approach: | They propose a progressive zero-shot dataset generation framework which leverages feedback from the task-specific model to guide the generation of new training data via in-context examples. |
| Outcome: | The proposed framework achieves on-par or superior performance with only 1% synthetic dataset size, when compared to baseline methods without in-context feedback. |
FuseGen: PLM Fusion for Data-generation based Zero-shot Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train Small Task-specific Models (STMs) using synthetic datasets are limited by the low quality of such datasets. |
| Approach: | They propose a data-generation based zero-shot learning framework that uses multiple PLMs to train small task-specific models. |
| Outcome: | The proposed framework outperforms existing methods in boosting performance across tasks. |
Zero- and Few-Shot NLP with Pretrained Language Models (2022.acl-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to introduce NLP researchers to the latest techniques for learning from little-to-no data . aims at bringing interested researchers up to speed about the latest and ongoing techniques . |
| Approach: | They aim to introduce techniques for learning from little-to-no data using pretrained language models. |
| Outcome: | This tutorial aims to bring interested NLP researchers up to speed about recent techniques . it will cover methods from manual engineering, better inference algorithms to better tuning methods . |
Few-Shot Data-to-Text Generation via Unified Representation and Multi-Source Learning (2023.acl-long)
Copied to clipboard
Alexander Hanbo Li, Mingyue Shang, Evangelia Spiliopoulou, Jie Ma, Patrick Ng, Zhiguo Wang, Bonan Min, William Yang Wang, Kathleen McKeown, Vittorio Castelli, Dan Roth, Bing Xiang
| Challenge: | Existing methods for data-to-text generation focus on specific types of structured data. |
| Approach: | They propose a method that provides a unified representation that can handle various forms of structured data such as tables, knowledge graph triples, and meaning representations. |
| Outcome: | The proposed method improves zero-shot and few-shot scenarios and can adapt to new structured data. |
Exploring efficient zero-shot synthetic dataset generation for Information Retrieval (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in large language models offer a new avenue of generating synthetic training data to train neural retrieval models for unlabelled data collections. |
| Approach: | They propose a method to generate high-quality synthetic datasets using a small language model and a filtering mechanism to ensure the quality of generated questions. |
| Outcome: | The proposed method outperforms unsupervised retrieval methods such as BM25 and pretrained monoT5. |
ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that large pretrained language models can generate training data with no task-specific or cross-task data. |
| Approach: | They propose a retrieval-enhanced framework to create training data from a general-domain unlabeled corpus. |
| Outcome: | The proposed framework achieves 4.3% gain over baselines and saves 70% of time compared with baselines using large language models. |
Zero-Shot Text Classification with Self-Training (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large pretrained language models have increased attention to zero-shot text classification. |
| Approach: | They propose a plug-and-play method to bridge this gap by requiring only class names along with an unlabeled dataset. |
| Outcome: | The proposed model can be trained on a natural language inference dataset and performs on dozens of unseen tasks without the need for domain expertise or trial and error. |
Pre-trained Language Models Can be Fully Zero-Shot Learners (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to pre-trained language models require fine-tuning on labeled datasets or manually constructing proper prompts. |
| Approach: | They propose a nonparametric prompting PLM for fully zero-shot language understanding . they compare it to previous methods for text classification and text entailment . |
| Outcome: | The proposed method outperforms previous methods on diverse tasks. |
On the Zero-Shot Generalization of Machine-Generated Text Detectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | rampant proliferation of large language models generates text indistinguishable from human-written language. |
| Approach: | They train neural detectors on outputs of a new generator and test their performance on held-out generators. |
| Outcome: | The proposed detectors can be built on training data from medium-sized models. |
Neural Pipeline for Zero-Shot Data-to-Text Generation (2022.acl-long)
Copied to clipboard
| Challenge: | In data-to-text generation, training on in-domain data leads to overfitting and repeating training data noise. |
| Approach: | They propose to train pretrained language models on general-domain text-based operations by transforming single-item descriptions with modules trained on ordering, aggregation, and paragraph compression. |
| Outcome: | The proposed approach enables D2T generation from RDF triples in zero-shot settings. |