| Challenge: | Recent advances in large-scale pre-training language models (PrLMs) have achieved remarkable successes in a variety of natural language processing tasks. |
| Approach: | They propose to use SPIDER to capture dialogue exclusive features from dialogue texts. |
| Outcome: | The proposed model performs well on widely used dialogue benchmarks. |
Similar Papers
Dialogue-oriented Pre-training (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PrLMs) have shown impressive improvements for various downstream tasks including various dialogue related ones. |
| Approach: | They propose to use pre-trained language models to simulate dialogue features on general plain text with common language model training objectives to improve performance. |
| Outcome: | The proposed method is fine-tuned on three public multi-turn dialogue datasets and achieves significant and consistent improvement over the plain PrLMs. |
Domain-Adaptive Pretraining Methods for Dialogue Understanding (2021.acl-short)
Copied to clipboard
| Challenge: | Recent advances in pretraining methods have achieved promising results on NLP tasks . however, it is unclear which pretraining objective is the most effective for each downstream task . |
| Approach: | They evaluate the effectiveness of domain-adaptive pretraining objectives on downstream tasks . they use open-domain data to pretrain language models like BERT and SpanBERT . |
| Outcome: | The proposed model improves on two dialogue understanding tasks with domain-adaptive pretraining objectives. |
StructuralLM: Structural Pre-training for Form Understanding (2021.acl-long)
Copied to clipboard
| Challenge: | Existing pre-trained language models focus on text-only representation, neglecting cell-level layout information. |
| Approach: | They propose a pre-training approach to leverage cell and layout information from scanned documents. |
| Outcome: | The proposed model achieves state-of-the-art in various downstream tasks . it uses 2Dposition embeddings to model word-level layout information . |
Learning-by-Narrating: Narrative Pre-Training for Zero-Shot Dialogue Comprehension (2022.acl-short)
Copied to clipboard
| Challenge: | Existing models for dialogue comprehension are not available for the pre-training of such a model. |
| Approach: | They propose a narrative-guided pre-training strategy that learns by narrating key information from a dialogue input. |
| Outcome: | The proposed model performs better on four dialogue-based tasks and is comparable to existing models. |
TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models with self-attention encoder architectures are less useful in practice. |
| Approach: | They propose to use user and system tokens to model dialogue behavior during pre-training . they propose a contrastive objective function to simulate the response selection task . |
| Outcome: | The proposed model outperforms baseline models on four downstream tasks . it also has a few-shot ability that can mitigate the data scarcity problem . |
Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment (2023.acl-long)
Copied to clipboard
Tianshu Yu, Haoyu Gao, Ting-En Lin, Min Yang, Yuchuan Wu, Wentao Ma, Chao Wang, Fei Huang, Yongbin Li
| Challenge: | Existing speech-text pre-training methods are limited to one or two specific tasks, despite their success in speech-language processing tasks. |
| Approach: | They propose a temporal position prediction task to capture the speech-text alignment . they use a textual dialog pre-training task to generalize a response selection task . |
| Outcome: | The proposed model is superior in learning speech-text alignment and multi-turn dialog context. |
Post-Training Dialogue Summarization using Pseudo-Paraphrasing (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to dialogue summarization use dialogue-specific features that require additional knowledge to recognize or make the models harder to tune. |
| Approach: | They propose to post-train pretrained language models to rephrase from dialogue to narratives and fine-tune them as usual. |
| Outcome: | The proposed approach outperforms existing models by summary quality and implementation costs. |
Semantic-based Pre-training for Dialogue Understanding (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-trained language models are weak in understanding the main semantic meaning of a dialogue context. |
| Approach: | They propose a semantic-based framework that leverages explicit semantic knowledge to capture the core semantic information in dialogues during pre-training. |
| Outcome: | The proposed model is superior to existing models on chit-chats and task-oriented dialogues. |
Enhancing Pre-trained Models with Text Structure Knowledge for Question Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing question generation models treat input passage as a sequence-to-sequence generative task, but they are not aware of text structure. |
| Approach: | They propose to model text structure as answer position and syntactic dependency and propose a mask attention mechanism to make syntaktic structure of input passage accessible. |
| Outcome: | The proposed model outperforms the strong pre-trained model ProphetNet on a SQuAD dataset and achieves competitive results with the state-of-the-art model. |
STRUDEL: Structured Dialogue Summarization for Dialogue Comprehension (2022.emnlp-main)
Copied to clipboard
Borui Wang, Chengcheng Feng, Arjun Nair, Madelyn Mao, Jai Desai, Asli Celikyilmaz, Haoran Li, Yashar Mehdad, Dragomir Radev
| Challenge: | Abstractive dialogue summarization is an important standalone task in natural language processing, but no previous work has explored whether it can be used to boost an NLP system's performance on other important dialogue comprehension tasks. |
| Approach: | They propose a novel type of dialogue summarization task that decomposes and imitates the hierarchical, systematic and structured mental process that human beings usually go through when understanding and analyzing dialogues. |
| Outcome: | The proposed model improves the performance of transformer encoder language models on two important dialogue comprehension tasks. |