CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems (2024.findings-acl)
Copied to clipboard
| Challenge: | a number of studies have evaluated user satisfaction estimation in TOD systems . current benchmarks for user satisfaction estimates are highly skewed towards dialogues for which the user is satisfied. |
| Approach: | They leverage large language models to generate satisfaction-aware counterfactual dialogues to augment original dialogues of a test collection. |
| Outcome: | The proposed models show higher robustness to increase in dissatisfaction labels than fine-tuned models. |
Similar Papers
“Stupid robot, I want to speak to a human!” User Frustration Detection in Task-Oriented Dialog Systems (2025.coling-industry)
Copied to clipboard
Mireia Hernandez Caralt, Ivan Sekulic, Filip Carevic, Nghia Khau, Diana Nicoleta Popa, Bruna Guedes, Victor Guimaraes, Zeyu Yang, Andre Manso, Meghana Reddy, Paolo Rosso, Roland Mathis
| Challenge: | Detecting user frustration in task-oriented dialog systems is imperative for maintaining overall user satisfaction, engagement and retention. |
| Approach: | They compare out-of-the-box methods for user frustration detection with open-source methods . they find an LLM-based approach is promising, as it captures both emotion and dialog breakdowns . |
| Outcome: | The proposed method outperforms open-source methods in detecting user frustration in a TOD system. |
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies on USM neglect explicitly modeling the user’s task goals fulfillment using the task schema. |
| Approach: | They propose a schema-guided user satisfaction modeling framework that explicitly models the degree to which the user’s preferences regarding task attributes are fulfilled by the system. |
| Outcome: | The proposed framework outperforms existing methods on benchmark datasets and shows that it can interpret and scale well with unseen tasks and can work in low-resource settings. |
Don’t be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System (2021.emnlp-main)
Copied to clipboard
| Challenge: | Consistency Identification has been used for preventing inconsistent response generation, but few efforts have been made to task-oriented dialogue. |
| Approach: | They propose a dataset for Consistency Identification in task-oriented dialog system. |
| Outcome: | The proposed dataset is based on a single label and provides fine-grained labels to encourage model to know what inconsistent sources lead to it. |
AutoEval-ToD: Automated Evaluation of Task-oriented Dialog Systems (2025.naacl-long)
Copied to clipboard
| Challenge: | Current evaluation methodologies heavily depend on human annotators, which can be inefficient, subjective, and expensive to scale. |
| Approach: | They propose an automated end-to-end evaluation framework that interacts with the ToD system and then assesses its performance across key dimensions. |
| Outcome: | The proposed framework first interacts with the ToD system and assesses its performance across key dimensions by analyzing both its responses and internal states. |
TOD-Flow: Modeling the Structure of Task-Oriented Dialogues (2023.emnlp-main)
Copied to clipboard
Sungryull Sohn, Yiwei Lyu, Anthony Liu, Lajanugen Logeswaran, Dong-Ki Kim, Dongsub Shim, Honglak Lee
| Challenge: | Recent advances in task-oriented dialogue systems have limitations regarding transparency and controllability. |
| Approach: | They propose to infer the TOD-flow graph from dialog data annotated with dialog acts and integrate it with any dialogue model to improve its prediction performance, transparency, and controllability. |
| Outcome: | The proposed approach improves dialog act classification and response generation performance in the MultiWOZ and SGD benchmarks. |
An Improved, Strong Baseline for Pre-Trained Large Language Models as Task-Oriented Dialogue Systems (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that Large Language Models perform insufficiently as TOD systems. |
| Approach: | They propose a self-checking mechanism to improve LLM performance as TOD systems. |
| Outcome: | The proposed model outperforms existing models and improves their performance. |
Counterfactual Dialog Mixing as Data Augmentation for Task-Oriented Dialog Systems (2024.lrec-main)
Copied to clipboard
| Challenge: | High-quality training data for Task-Oriented Dialog systems is costly to come by if no corpora are available. |
| Approach: | They propose a data augmentation technique that generates realistic synthetic dialogs via counterfactuals to increase the amount of training data. |
| Outcome: | The proposed technique achieves state-of-the-art on a benchmark dataset and lower resource setting. |
Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models (2024.acl-long)
Copied to clipboard
Ying-Chun Lin, Jennifer Neville, Jack Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, Jaime Teevan
| Challenge: | Existing approaches to user satisfaction estimation are hard to interpret and lack generalizable patterns. |
| Approach: | They propose to use supervised prompting to extract interpretable user satisfaction signals from natural language utterances to tailor an LLM to USE using labeled examples. |
| Outcome: | The proposed method extracts interpretable signals of user satisfaction from natural language utterances more effectively than embedding-based approaches. |
LLM-guided Plan and Retrieval: A Strategic Alignment for Interpretable User Satisfaction Estimation in Dialogue (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for estimating user satisfaction with dialogue systems face challenges due to limited understanding of underlying reasons for user dissatisfaction and high costs of annotating user intentions. |
| Approach: | They propose an interpretable framework for effective user satisfaction prediction . they propose to align utterances with strategies and large language models to retrieve relevant features from utterations. |
| Outcome: | The proposed framework achieves state-of-the-art performance on three benchmarks for the USE task. |
Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation (2023.tacl-1)
Copied to clipboard
| Challenge: | Multilingual task-oriented dialogue (ToD) datasets suffer from severe limitations, such as being small in scale and lacking naturalness and cultural specificity in the target language. |
| Approach: | They propose a novel outline-based annotation process where domain-specific abstract schemata of dialogue are mapped into natural language outlines. |
| Outcome: | The proposed approach improves understanding, dialogue state tracking, and end-to-end dialogue evaluation in Arabic, Indonesian, Russian, and Kiswahili. |