A Similarity Measure for Comparing Conversational Dynamics (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Qualities of a conversation are dependent on how interactions combine to form a “shape” of the conversation. |
| Approach: | They propose a similarity measure to capture differences in conversation dynamics and assess its sensitivity to the topic of the conversation. |
| Outcome: | The proposed measure captures differences in conversation dynamics and assesses its sensitivity to the topic of the conversation. |
Similar Papers
We’ve had this conversation before: A Novel Approach to Measuring Dialog Similarity (2021.emnlp-main)
Copied to clipboard
| Challenge: | Dialogs are a building block of human natural language interactions. |
| Approach: | They propose a new edit distance metric for dialog similarity analysis using conversation semantics, conversation flow, and the participants. |
| Outcome: | The proposed method outperforms existing methods on two publicly available datasets and is better aligned with human perception of conversation similarity. |
Probing the Robustness of Trained Metrics for Conversational Dialogue Systems (2022.acl-short)
Copied to clipboard
| Challenge: | Existing methods for evaluating conversational dialogue systems have been shown to be inefficient and instabile. |
| Approach: | They propose an adversarial method to stress-test trained metrics for evaluation of conversational dialogue systems using Reinforcement Learning. |
| Outcome: | The proposed method outperforms existing methods and can be applied to stress-test trained metrics for conversational dialogue systems. |
TaskDiff: A Similarity Metric for Task-Oriented Conversations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Popularity of chatGPT and Llama 2 has led to a race to build custom task-oriented conversational assistants in enterprise domains like finance and retail. |
| Approach: | They propose a conversational similarity metric that uses different dialogue components to compute similarity. |
| Outcome: | Experiments on a benchmark dataset show that the proposed metric outperforms existing approaches and is more robust than previous approaches. |
Computational Analysis of Conversation Dynamics through Participant Responsivity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Growing literature explores toxicity and polarization in discourse, with comparatively little work on characterizing what makes dialogue prosocial and constructive. |
| Approach: | They develop and evaluate methods for quantifying responsivity through semantic similarity of speaker turns and large language models to identify the relation between two speaker turns. |
| Outcome: | The proposed method is based on semantic similarity of speaker turns and large language models to identify the relation between two speaker turns. |
Evaluation and Facilitation of Online Discussions in the LLM Era: A Survey (2025.emnlp-main)
Copied to clipboard
Katerina Korre, Dimitris Tsirmpas, Nikos Gkoumas, Emma Cabalé, Danai Myrtzani, Theodoros Evgeniou, Ion Androutsopoulos, John Pavlopoulos
| Challenge: | Recent advances in LLMs enable artificial facilitation agents to not only moderate content, but also actively improve the quality of interactions. |
| Approach: | They propose a taxonomy on discussion quality evaluation and a new taxonomies for intervention and facilitation strategies. |
| Outcome: | The proposed methods synthesize ideas from Natural Language Processing (NLP) and Social Sciences to provide a taxonomy on discussion quality evaluation, and a roadmap of good practices and future research directions. |
Can You Follow Me? Testing Situational Understanding for ChatGPT (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have identified certain SU limitations in non-chatbot Large Language models, but the extent and causes of these limitations are not well understood. |
| Approach: | They propose a synthetic environment for SU testing in chat-oriented models . they test models' ability to track and enumerate environment states . |
| Outcome: | The proposed environment allows for controlled and systematic testing of SU in chat-oriented models, and to better understand underlying causes for performance patterns. |
A Collection of Pragmatic-Similarity Judgments over Spoken Dialog Utterances (2024.lrec-main)
Copied to clipboard
| Challenge: | a new study uses pragmatic similarity measures to train speech synthesizers . the average inter-judge correlation between utterance pairs was 0.45. |
| Approach: | they use a re-enactment of a recorded dialog to create pragmatic similarity . they use 9 judges to listen to 220 utterance pairs and rate them on a continuous scale . |
| Outcome: | The results show that human judges listened to 220 utterance pairs and rated them on a continuous scale. |
Characterizing Similarities and Divergences in Conversational Tones in Humans and LLMs by Sampling with People (2024.acl-long)
Copied to clipboard
| Challenge: | Existing taxonomies or text corpora suffer from experimenter bias and are not representative of real-world distributions. |
| Approach: | They propose an iterative method for simultaneously eliciting conversational tones and sentences . they run 50 iterations with human participants and GPT-4 and obtain a dataset of sentences and frequent conversational tone. |
| Outcome: | The proposed method can be used to characterize the differences between humans and LLMs. |
DialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies focus on the self and inter-personal dependencies in multi-modal conversations, but they ignore the temporal and spatial dependencies. |
| Approach: | They propose a Dialogue Transformer for simultaneously modeling the intra-modal and inter-modal emotion dynamics. |
| Outcome: | The proposed models outperform the state-of-the-art on three benchmark datasets. |
Assessing Dialogue Systems with Distribution Distances (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics focus on turnlevel quality, which is not well suited for open-end dialogue tasks. |
| Approach: | They propose to measure the performance of a dialogue system by computing the distributionwise distance between its generated conversations and real-world conversations. |
| Outcome: | The proposed metrics correlate better with human judgments than existing metrics on dialogue systems. |