| Challenge: | Existing methods for pretraining dialog context encoders are still in their infancy. |
| Approach: | They propose to use unsupervised pretraining objectives for dialog context representations to fine-tune and evaluate them on a set of downstream dialog tasks. |
| Outcome: | The proposed methods improve performance on a set of dialog tasks and are less data hungry. |
Similar Papers
Domain-Adaptive Pretraining Methods for Dialogue Understanding (2021.acl-short)
Copied to clipboard
| Challenge: | Recent advances in pretraining methods have achieved promising results on NLP tasks . however, it is unclear which pretraining objective is the most effective for each downstream task . |
| Approach: | They evaluate the effectiveness of domain-adaptive pretraining objectives on downstream tasks . they use open-domain data to pretrain language models like BERT and SpanBERT . |
| Outcome: | The proposed model improves on two dialogue understanding tasks with domain-adaptive pretraining objectives. |
Hierarchical Pre-training for Sequence Labelling in Spoken Dialog (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Sequence labelling tasks like Dialog Act and Emotion/Sentiment identification are a key component of spoken dialog systems. |
| Approach: | They propose a new approach to learn generic representations adapted to spoken dialog using a hierarchical encoder based on transformer architectures. |
| Outcome: | The proposed model is model-agnostic and contains 10 datasets of various sizes. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
Pretrain-Finetune Based Training of Task-Oriented Dialogue Systems in a Real-World Setting (2021.naacl-industry)
Copied to clipboard
| Challenge: | a challenge in building task-oriented dialogue systems is the limited amount of supervised training data available. |
| Approach: | They propose a method for training retrieval-based dialogue systems using annotated data and a larger, unlabeled dataset. |
| Outcome: | The proposed method improves model performance offline and online compared with no pretraining . the model is deployed in an agent-support application and evaluated on live customer service contacts . |
Which *BERT? A Survey Organizing Contextualized Encoders (2020.emnlp-main)
Copied to clipboard
| Challenge: | a survey on language representation learning aims to highlight common themes . we focus on the areas of progress, compared to other fields, and discuss how each area is evaluated. |
| Approach: | They present a survey on language representation learning to highlight common themes . they compare contributions in contextualized text encoders to ideas from other fields . |
| Outcome: | The proposed survey aims to highlight common themes in the field of language representation learning. |
DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation (2022.acl-long)
Copied to clipboard
Wei Chen, Yeyun Gong, Song Wang, Bolun Yao, Weizhen Qi, Zhongyu Wei, Xiaowu Hu, Bartuer Zhou, Yi Mao, Weizhu Chen, Biao Cheng, Nan Duan
| Challenge: | Existing pre-trained dialog models shed light on various downstream tasks in natural language processing (NLP). |
| Approach: | They propose a dialog pre-training framework that introduces latent variables into the enhanced encoder-decoder pre-train framework to increase relevance and diversity of responses. |
| Outcome: | The proposed model achieves state-of-the-art on personaChat, DailyDialog, and DSTC7-AVSD datasets. |
Pretrained Language Models for Sequential Sentence Classification (D19-1)
Copied to clipboard
| Challenge: | Recent successful models for document-level understanding have used hierarchical encoding and CRFs to capture dependencies between subsequent labels. |
| Approach: | They propose a pretrained language model that captures contextual dependencies without hierarchical encoding nor a CRF. |
| Outcome: | The proposed model captures contextual dependencies without hierarchical encoding nor a CRF on four datasets, including a new dataset of structured scientific abstracts. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
Representation Learning for Conversational Data using Discourse Mutual Information Maximization (2022.naacl-main)
Copied to clipboard
Bishal Santra, Sumegh Roychowdhury, Aishik Mandal, Vasu Gurram, Atharva Naik, Manish Gupta, Pawan Goyal
| Challenge: | Existing language modeling pretraining objectives do not take structural information of conversational text into account. |
| Approach: | They propose a structure-aware Mutual Information based loss-function DMI for training dialog-representation models that captures the inherent uncertainty in response prediction. |
| Outcome: | The proposed model outperforms strong baseline models on nine diverse tasks. |
Role of Context in Unsupervised Sentence Representation Learning: the Case of Dialog Act Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised learning of word representations involves capturing the contextual information surrounding word occurrences. |
| Approach: | They propose to use text-based dialog act tags to compare content- and context-oriented sentence representations inferred on telephone conversations to examine whether a contextual signal is of any significant benefit to general-purpose sentence representation. |
| Outcome: | The proposed model outperforms content-based and context-oriented representations on telephone conversations and shows that it increases the dimensionality of the vectors. |