HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work on detecting user disengagement requires hand-labeling many dialog samples . Currently, self-reported user ratings are constrained to a static corpus . |
| Approach: | They propose an efficient annotation framework that denoises dialog samples instead of manually labeling them. |
| Outcome: | The proposed framework improves annotation efficiency significantly and detects user disengagement in two dialog corpora. |
Similar Papers
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing metrics for dialog evaluation are trained on human annotations, which is cumbersome to collect. |
| Approach: | They propose to use user sentiment and other information as proxy to measure the quality of previous dialogs. |
| Outcome: | The proposed model is comparable to models trained on human annotated data. |
Multi-Domain Goal-Oriented Dialogues (MultiDoGO): Strategies toward Curating and Annotating Large Scale Dialogue Data (D19-1)
Copied to clipboard
| Challenge: | a large number of goal-oriented dialogue datasets are limited in their size, linguistic diversity, domain coverage, or annotation granularity. |
| Approach: | They propose a multi-domain goal-oriented dialogue dataset that uses a crowd-sourced worker and a trained annotator to curate and annotate large scale data. |
| Outcome: | The proposed dataset is 8 times the size of the largest comparable dialogue dataset available to the public. |
LIDA: Lightweight Interactive Dialogue Annotator (D19-3)
Copied to clipboard
| Challenge: | Dialogue systems are dependent on the quality of the data used to train them. |
| Approach: | They propose to develop an annotation tool specifically for conversation data that handles the entire dialogue annotation pipeline from raw text to structured conversation data. |
| Outcome: | The proposed tool handles the entire dialogue annotation pipeline from raw text to structured conversation data and has a dedicated interface to resolve inter-annotator disagreements. |
RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue (2023.acl-long)
Copied to clipboard
| Challenge: | Evaluating open-domain dialogue systems is challenging because of the one-to-many problem. |
| Approach: | They propose a reference-based dialogue evaluation approach that leverages the pre-created utterance as reference other than the gold response to relieve the one-to-many problem. |
| Outcome: | The proposed method outperforms state-of-the-art evaluation methods on three datasets and two existing benchmarks. |
Learning an Unreferenced Metric for Online Dialogue Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing tools for dialogue evaluation do not generalize to unseen datasets and/or need a human-generated reference response during inference. |
| Approach: | They propose an unreferenced automated dialogue evaluation metric that uses large pre-trained language models to extract latent representations of utterances and leverages the temporal transitions that exist between them. |
| Outcome: | The proposed model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference. |
Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022.findings-emnlp)
Copied to clipboard
Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, Helen Meng
| Challenge: | a number of safety concerns hinder the deployment of open-domain dialog systems, such as offensive languages and toxic behaviors, such social bias is difficult to detect. |
| Approach: | They propose a Dial-Bias Framework for analyzing social bias in conversations . they introduce a Chinese social bias dialog dataset and conduct in-depth ablation studies . |
| Outcome: | The proposed framework is the first annotated Chinese social bias dialog dataset . the proposed framework also provides a fine-grained dialog bias measurement benchmark . |
A Large-Scale Corpus for Conversation Disentanglement (P19-1)
Copied to clipboard
Jonathan K. Kummerfeld, Sai R. Gouravajhala, Joseph J. Peper, Vignesh Athreya, Chulaka Gunasekara, Jatin Ganhotra, Siva Sankalp Patel, Lazaros C Polymenakos, Walter Lasecki
| Challenge: | a dataset of 77,563 messages manually annotated with reply-structure graphs disentangles conversations and defines internal conversation structure. |
| Approach: | They use a dataset of 77,563 messages manually annotated with reply-structure graphs to disentangle conversations and define internal conversation structure. |
| Outcome: | The new dataset is 16 times larger than all previous datasets combined and includes adjudication of annotation disagreements and context. |
DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deference (2026.acl-long)
Copied to clipboard
Parisa Rabbani, Priyam Sahoo, Ruben Mathew, Aishee Mondal, Harshita Ketharaman, Nimet Beyza Bozdag, Dilek Hakkani-Tür
| Challenge: | a single model can shift toward disagreement (skepticism) on graduate-level science and toward agreement (deference) on social judgment. |
| Approach: | They propose a framework to detect and mitigat framing-induced judgment shifts . they propose 'DialDefer' framework to help model disagreements and disagreements based on attribution . |
| Outcome: | The proposed framework detects and mitigates dialogic deference shifts in LLMs . human-vs-LLM attribution drives the largest shifts (17.7 pp swing) |
ISO-Standard Domain-Independent Dialogue Act Tagging for Conversational Agents (C18-1)
Copied to clipboard
| Challenge: | Existing methods for DA annotation are incompatible with each other and do not cover all aspects necessary for open-domain human-machine interaction. |
| Approach: | They propose to map publicly available corpora to a subset of the ISO standard and create a task-independent training corpus for DA classification. |
| Outcome: | The proposed method can train a domain-independent DA tagger on out-of-domain conversational data and achieve robustness across different DA categories. |
DiQAD: A Benchmark Dataset for Open-domain Dialogue Quality Assessment (2023.findings-emnlp)
Copied to clipboard
Yukun Zhao, Lingyong Yan, Weiwei Sun, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, Dawei Yin
| Challenge: | Existing studies on dialogue quality assessment are uncapable of providing an end-to-end and human-epistemic assessment dataset . open-domain dialogue assessment is complicated and costly, but it can be done by recruiting human evaluators. |
| Approach: | They propose a large-scale dialogue quality assessment dataset for automatically assessing open-domain dialogue quality. |
| Outcome: | The proposed dataset is openly accessible at https://github.com/yukunZhao/Dialogue_quality_evaluation. |