Dialogue Distillation: Open-Domain Dialogue Augmentation Using Unpaired Data (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing research has focused on training open-domain dialogue models using unpaired data. |
| Approach: | They propose a data-level distillation method for training open-domain dialogue models by utilizing unpaired data. |
| Outcome: | The proposed method produces high-quality dialogue pairs with diverse contents, and can improve competitive baselines. |
Similar Papers
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2024.naacl-long)
Copied to clipboard
Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, Yejin Choi
| Challenge: | Impossible Distillation is a framework for paraphrasing and sentence summarization that can be trained from a low-quality teacher model. |
| Approach: | They propose a framework that distills a high-quality dataset from a low-quality teacher . they hypothesize and verify the paraphrastic proximity intrinsic to pre-trained LMs . |
| Outcome: | The proposed framework outperforms baseline models on unconstrained paraphrase generation and sentence summarization benchmarks. |
Diversifying Neural Dialogue Generation via Negative Distillation (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to generate generic responses are ignoring low-frequency but generic responses and bringing low- frequency but meaningless responses. |
| Approach: | They propose a negative training paradigm that reminds dialogue models not to generate high-frequency responses during training. |
| Outcome: | The proposed method outperforms previous methods in the generic response problem while minimizing low-frequency but meaningless responses. |
Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | generative models are less practical for building real-time conversation systems due to high latency and large memory footprint. |
| Approach: | They propose a method that preserves the efficiency of a retrieval model while leveraging the conversational ability of generative models. |
| Outcome: | The proposed method preserves the efficiency of a retrieval model while leveraging the conversational ability of generative models. |
A Model-agnostic Data Manipulation Method for Persona-based Dialogue Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models for introducing explicit personas are expensive due to their expensive collection costs. |
| Approach: | They propose a data manipulation method which is model-agnostic to be packed with any persona-based dialogue generation model to improve their performance. |
| Outcome: | The proposed method is model-agnostic to be packed with any persona-based dialogue generation model to improve their performance. |
Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweight (2020.acl-main)
Copied to clipboard
| Challenge: | Current state-of-the-art neural dialogue models learn from human conversations . however, due to the open-ended nature of human conversations, the quality of training data varies . |
| Approach: | They propose a data manipulation framework to augment and highlight effective training samples . they also propose to increase its manipulation skills through gradient descent with validation samples a reshaping framework to proactively restructure the data distribution towards reliable samples is also proposed . |
| Outcome: | The proposed framework improves the performance of open-domain neural dialogue models with respect to evaluation metrics and human judgments. |
Textual Dataset Distillation via Language Model Embedding (2024.findings-emnlp)
Copied to clipboard
| Challenge: | prevailing methods for dataset distillation generate distilled data as embedding vectors, which are not human-readable. |
| Approach: | They propose a model-agnostic, data-efficient method that leverages Language Model embeddings . their method offers enhanced flexibility and improved transferability . |
| Outcome: | The proposed method achieves comparable performance with faster processing times compared to other methods . it offers enhanced flexibility and improved transferability, expanding the range of potential applications . |
Unveiling the Magic: Investigating Attention Distillation in Retrieval-Augmented Generation (2024.naacl-short)
Copied to clipboard
| Challenge: | Retrieval-augmented generation framework addresses the limitations of large language models by enabling real-time knowledge updates for more accurate answers. |
| Approach: | They propose to use attention distillation to improve retrieval-augmented language models' learning performance by identifying key factors influencing their workflow and proposing indicators for optimizing models’ training methods and avoiding ineffective training. |
| Outcome: | The proposed framework improves the learning performance of large language models in the training phase but also reduces the impact of ineffective training. |
Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some performance benefits. |
| Approach: | They propose to augment the distillation with generated unlabelled examples that match the target distribution and upsamples data points among the training set that are similar to the target. |
| Outcome: | The proposed method outperforms previous robustness solutions on the task of natural language inference (NLI) it also improves performance on OOD domains even beyond the target domain. |
Data and Model Distillation as a Solution for Domain-transferable Fact Verification (2021.naacl-main)
Copied to clipboard
| Challenge: | Neural networks depend heavily on lexicalized information, which transfers poorly between domains. |
| Approach: | They propose a method to delexicize lexicalized data and a model distillation technique to prevent aggressive data distillation. |
| Outcome: | The proposed method improves performance on lexicalized data and out of domain models. |
DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to extract word embeddings from training datasets are not efficient for training other models. |
| Approach: | They propose a method to distill a training dataset into a textual model by combining a small number of informative synthetic samples. |
| Outcome: | The proposed method outperforms existing methods on training datasets and language models. |