Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Conventional evaluation methods often overlook variances in model behavior across different levels of structural complexity on interaction graphs. |
| Approach: | They propose a methodological pipeline to investigate model performance across structural attributes of conversations. |
| Outcome: | The proposed method analyzes the performance of an LLM to classify multi-party conversations . it shows that response selection relies more on the textual content of conversations compared to addressee recognition . |
Similar Papers
Is ChatGPT a Good Multi-Party Conversation Solver? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are powerful tools for multi-party conversations, but their capacity to handle multi-parties remains unexplored. |
| Approach: | They propose to evaluate ChatGPT and GPT-4's zero-shot learning capabilities within the context of multi-party conversations (MPCs) they also propose to incorporate MPC structures, encompassing both speaker and addressee architecture. |
| Outcome: | The proposed models perform poorly on a number of MPC tasks while GPT-4 performs well on speaker and addressee architecture. |
MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for multi-party conversation represent interlocutors and utterances individually . existing methods ignore complicated structure of MPC which may provide crucial interlocutor and tertiary semantics. |
| Approach: | They propose a pre-trained model for multi-party conversation that considers learning who says what to whom in a unified model with elaborated self-supervised tasks. |
| Outcome: | The proposed model outperforms existing models on three downstream tasks at two benchmarks. |
Simple Agents, Biased Judges: Efficient Multi-Party Dialogue Generation & The Evaluation Gap (2026.acl-long)
Copied to clipboard
| Challenge: | Multiparty social dialogue is difficult to formalize and expensive to evaluate, especially at scale. |
| Approach: | They propose a lightweight and controllable multi-party dialoguegeneration framework as an experimental instrument for studying generation and evaluation in social interaction. |
| Outcome: | The proposed framework shows that human judgments against state-of-the-art LLM judges are consistent with human preferences for naturalness, engagingness, and overall quality in multi-party social dialogue. |
Do LLMs Understand Dialogues? A Case Study on Dialogue Acts (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable performance on many unseen tasks in a zero-shot setting. |
| Approach: | They propose to identify three key pre-tasks essential for accurate DA prediction: Turn Management, Communicative Function Identification, and Dialogue Structure Prediction. |
| Outcome: | The proposed model fails to outperform basic rule-based tasks on three key pre-tasks, and the results suggest that the model is flawed. |
Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are now commonplace in conversation applications, but their misuse for generating harmful responses has raised serious societal concerns. |
| Approach: | They provide a comprehensive overview of recent studies covering attacks, defenses, and evaluations of Large Language Models (LLMs) . |
| Outcome: | The proposed review summarizes three aspects of LLM conversation safety: attacks, defenses, and evaluations. |
Response Selection for Multi-Party Conversations with Dynamic Topic Tracking (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing response selection methods focus on a two-party single-conversation scenario. |
| Approach: | They propose a multi-task learning framework that frames response selection as a dynamic topic tracking task to match the topic between the response and relevant conversation context. |
| Outcome: | The proposed framework outperforms existing methods on an Ubuntu IRC dataset in response selection and topic disentanglement tasks. |
In Search of the Lost Arch in Dialogue: A Dependency Dialogue Acts Corpus for Multi-Party Dialogues (2025.findings-acl)
Copied to clipboard
Jon Cai, Brendan King, Peyton Cameron, Susan Windisch Brown, Miriam Eckert, Dananjay Srinivas, George Arthur Baker, V Kate Everson, Martha Palmer, James Martin, Jeffrey Flanigan
| Challenge: | Understanding speaker intentions remains a challenge in NLP . a number of corpora annotated using theoretical frameworks of dialogue focus on utterance-level labeling of speaker intent, missing wider context, or the rhetorical structure of a dialogue. |
| Approach: | They propose to annotate a corpus of 33 dialogues and over 9,000 utterance units using the Dependency Dialogue Acts framework. |
| Outcome: | The proposed corpus spans four genres of multi-party conversations from different modalities. |
Are Large Language Models (LLMs) Good Social Predictors? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies suggest that Large Language Models can generate human-like responses, but it is unclear how well they work and where the plausible predictions derive from. |
| Approach: | They propose to use LLMs to generate human-like responses by mutability and accessibility of social inputs to perform a social prediction task. |
| Outcome: | The proposed model performs well in three realistic settings and a novel social prediction task. |
Addressee and Response Selection for Multilingual Conversation (C18-1)
Copied to clipboard
| Challenge: | Developing conversational systems that can converse in many languages is an interesting challenge for natural language processing. |
| Approach: | They propose multilingual addressee and response selection task for conversational systems . they use a multilingual conversation dataset to evaluate their methods . |
| Outcome: | The proposed methods can predict addressee and response in multiple languages . they show that the methods work in a multilingual conversation dataset . |
From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions (2025.acl-long)
Copied to clipboard
Nathanaël Carraz Rakotonirina, Mohammed Hamdy, Jon Ander Campos, Lucas Weber, Alberto Testoni, Marzieh Fadaee, Sandro Pezzelle, Marco Del Tredici
| Challenge: | Large Language Models excel at solving individual problems in isolation, but are they able to effectively collaborate over long-term interactions? |
| Approach: | They propose to use a multi-session dataset to test LLMs' ability to track and execute simple coding instructions amid irrelevant information, simulating a realistic setting. |
| Outcome: | The proposed model performs poorly when instructions are spread across sessions, suggesting that they are not able to integrate information over long interactions. |