Dialogue Coherence Assessment Without Explicit Dialogue Act Labels (2020.acl-main)
Copied to clipboard
| Challenge: | Recent dialogue coherence models use coherency features designed for monologue texts to represent utterances and then explicitly augment them with dialogue-relevant features, e.g., dialogue act labels. |
| Approach: | They propose a multi-task learning approach that uses dialogue act prediction to obtain informative utterance representations for coherence assessment. |
| Outcome: | The proposed model outperforms its strong competitors on the DailyDialogue corpus and performs on par with them on the SwitchBoard corpus for ranking dialogues concerning their coherence. |
Similar Papers
How coherent are neural models of coherence? (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to model coherence are limited to small newswire corpora . evaluators need to be trained on lexical and document levels to perform evaluations . |
| Approach: | They propose four generic evaluation tasks that capture coherence-specific properties . they aim at capturing correct use of discourse connectives and lexical cohesion . |
| Outcome: | The proposed tasks capture coherence-specific properties, including correct use of discourse connectives, lexical cohesion, temporal consistency among events and participants in a story. |
Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | a common approach to coherence evaluation is shuffling the sentence order of a text, creating incoherent text samples that need to be discriminated from the original. |
| Approach: | They propose an extendable set of test suites addressing different aspects of discourse and dialogue coherence. |
| Outcome: | The proposed evaluation paradigm is suited to evaluate linguistic qualities that contribute to the notion of coherence. |
Joint Modeling of Entities and Discourse Relations for Coherence Assessment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on coherence modeling focuses on entity features or discourse relation features, with little attention given to combining the two. |
| Approach: | They propose two methods for jointly modeling entities and discourse relations for coherence assessment. |
| Outcome: | The proposed methods significantly improve the performance of coherence models on three benchmark datasets. |
GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics only consider surface features or utterance-level semantics, without explicitly considering the fine-grained topic transition dynamics of dialogue flows. |
| Approach: | They propose a graph-enhanced evaluation metric GRADE to evaluate dialogue coherence . GRADE incorporates utterance-level contextualized representations and fine-grained topic-level graph representations to improve communication logic. |
| Outcome: | The proposed evaluation metric outperforms state-of-the-art metrics on measuring diverse dialogue models in terms of Pearson and Spearman correlations with human judgments. |
A Novel Computational Modeling Foundation for Automatic Coherence Assessment (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models for text coherence assessment rely on a proxy task . however, this approach does not capture the full range of factors contributing to coherency. |
| Approach: | They propose a formal linguistic definition of what makes a discourse coherent and formalize these conditions as respective computational tasks that are jointly trained. |
| Outcome: | The proposed model improves on two human-rated coherence benchmarks. |
A Unified Neural Coherence Model (D19-1)
Copied to clipboard
| Challenge: | Existing models for coherence modeling fail on harder tasks with more realistic application scenarios. |
| Approach: | They propose a unified coherence model that incorporates sentence grammar, inter-sentence coherent relations, and global coherency patterns into a common neural framework. |
| Outcome: | The proposed model outperforms existing models on local and global discrimination tasks and outperformed existing models by a good margin. |
How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to evaluate summary coherence are often evaluated using disparate datasets and metrics. |
| Approach: | They propose to use automatic evaluation to evaluate coherence of summaries by selecting high-scoring candidates. |
| Outcome: | The proposed methods show that they can perform better on an even playing field. |
Evaluating Coherence in Dialogue Systems using Entailment (N19-1)
Copied to clipboard
| Challenge: | Evaluating open-domain dialogue systems is difficult due to the diversity of possible correct answers. |
| Approach: | They propose a set of metrics for evaluating topic coherence using distributed sentence representations and calculable approximations of human judgment using conversational coherency. |
| Outcome: | The proposed metrics can be used as a surrogate for human judgment based on conversational coherence on large-scale datasets and provide an unbiased estimate for the quality of the responses. |
What Did You Refer to? Evaluating Co-References in Dialogue (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing neural end-to-end dialogue models have limitations on exactly interpreting the linguistic structures in dialogue history context. |
| Approach: | They propose to directly measure the capability of neural end-to-end dialogue models on understanding the entity-oriented structures via question answering. |
| Outcome: | The proposed model can understand large-scale English and Chinese human human dialogues using a large-format dataset. |
Non-Topical Coherence in Social Talk: A Call for Dialogue Model Enrichment (2020.acl-srw)
Copied to clipboard
| Challenge: | Current models of dialogue focus on utterances within a topically coherent discourse segment, not on social conversations . a pilot annotation study of NTUs is a first step towards a model capable of rationalizing conversational coherence in social talk. |
| Approach: | They conduct a pilot annotation study of social dialogues as a first step towards a Bayesian game-theoretic model . they first annotate content-based coherence relations that are not available in Disco-SPICE . |
| Outcome: | The proposed model can rationalize conversational coherence in social talk, the authors say . the study focuses on the natural occurring social dialogues in the Disco-SPICE corpus . |