Tianle Liang, Yifu Chen, Shengpeng Ji, Yijun Chen, Zhiyang Jia, Jingyu Lu, Fan Zhuo, Xueyi Pu, Yangzhuo Li, Zhou Zhao
| Challenge: | Existing research on end-to-end spoken dialogue models has focused on core perception and generation, with limited exploration of tool-augmented extensions. |
| Approach: | They propose a framework to equip end-to-end spoken dialogue models with comprehensive agentic abilities by leveraging a 470-hour AgentChat dataset. |
| Outcome: | The proposed framework outperforms Gemini-2.5-Pro on spoken agent tasks while maintaining general conversational quality. |
Similar Papers
Spoken Conversational Agents with Large Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial focuses on the evolution of voice-native LLMs . it reviews the adaptation of text LLM to audio, cross-modal alignment, and joint speech–text training . |
| Approach: | This tutorial examines the evolution of voice-native LLMs in conversational agents . it compares cascaded and voice-based LLM systems to end-to-end retrieval-and vision-grounded systems . |
| Outcome: | This tutorial examines the evolution of voice-native LLMs . it compares the performance of voice assistants to current open-domain agents . |
Hello Again! LLM-powered Personalized Agent for Long-term Dialogue (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing dialogue systems focus on brief single-session interactions, neglecting real-world needs for long-term companionship and personalized interactions. |
| Approach: | They propose a model-agnostic framework for long-term dialogue agents . they use event summary and persona management to enable reasoning . |
| Outcome: | The proposed framework incorporates three independently tunable modules dedicated to event perception, persona extraction, and response generation. |
VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing question-answering benchmarks fail to evaluate SLMs’ knowledge understanding due to their inability to support end-to-end speech evaluation and account for varied input audio conditions. |
| Approach: | They propose a new question-answering benchmark that assesses SLMs’ knowledge understanding through pure speech interactions. |
| Outcome: | The proposed benchmark maintains speech format for both inputs and outputs, evaluates model robustness across diverse input audio conditions, and pioneers the assessment of complex tasks like mathematical reasoning in spoken format. |
ActorMind: Emulating Human Actor Reasoning for Speech Role-Playing (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing work on role-playing focuses on textual modalities, neglecting speech . et al., 2025) show that speech role-players can generate spontaneous responses with personalized traits based on the context. |
| Approach: | They propose a framework that allows models to deliver spontaneous responses with personalized verbal traits based on their role, scene, and spoken dialogue. |
| Outcome: | The proposed framework enhances speech role-playing by generating spontaneous responses with personalized traits based on their role, scene, and spoken dialogue. |
SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation (2026.eacl-demo)
Copied to clipboard
Sergio Burdisso, Séverin Baroudi, Yanis Labrak, David Grünert, Pawel Cyrta, Yiyang Chen, Srikanth Madikeri, Esaú Villatoro-tello, Ricard Marxer, Petr Motlicek
| Challenge: | SDialog is an open-source Python toolkit for end-to-end development, simulation, evaluation and analysis of LLM-based conversational agents. |
| Approach: | They present an open-source Python toolkit for end-to-end development, simulation, evaluation and analysis of LLM-based conversational agents. |
| Outcome: | SDialog enables more controlled, transparent, and systematic research on conversational systems. |
ADVISER: A Toolkit for Developing Multi-modal, Multi-domain and Socially-engaged Conversational Agents (2020.acl-demos)
Copied to clipboard
Chia-Yu Li, Daniel Ortega, Dirk Väth, Florian Lux, Lindsey Vanderlyn, Maximilian Schmidt, Michael Neumann, Moritz Völkel, Pavel Denisov, Sabrina Jenne, Zorica Kacarevic, Ngoc Thang Vu
| Challenge: | Existing toolkits for developing dialog systems are limited to core components and do not support multi-modal processing and social signals. |
| Approach: | They propose to use ADVISER to develop multi-modal dialog agents using multi-text and social signals. |
| Outcome: | The proposed toolkit is flexible, easy to use, and easy to extend for linguists and cognitive scientists, thereby providing a flexible platform for collaborative research. |
Towards a Progression-Aware Autonomous Dialogue Agent (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent advances in large-scale language modeling and generation have enabled the creation of dialogue agents that exhibit human-like responses in a wide range of conversational scenarios. |
| Approach: | They propose a framework in which dialogue agents can evaluate the progression of a conversation toward or away from desired outcomes and use this signal to inform planning for subsequent responses. |
| Outcome: | The proposed framework evaluates the progression of a conversation toward or away from desired outcomes and uses this signal to inform planning for subsequent responses. |
Context-Agent: Dynamic Discourse Trees for Non-Linear Dialogue (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to managing non-linear dialogue flow are misaligned with the intrinsically hierarchical and branching structure of natural discourse. |
| Approach: | They propose a framework that models multi-turn dialogue history as a dynamic tree structure. |
| Outcome: | The proposed framework enhances task completion rates and improves token efficiency across various LLMs. |
Beyond Candidates : Adaptive Dialogue Agent Utilizing Persona and Knowledge (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a previous study suggested that human dialogue systems ground persona and knowledge but they require incomplete candidate sets. |
| Approach: | They propose an adaptive dialogue agent that uses persona and knowledge without candidate sets . their model generates consistent and relevant persona descriptions and identifies relevant knowledge . |
| Outcome: | The proposed model outperforms baselines that ground persona and knowledge candidates even with fragmentary information. |
Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are generalist agents capable of operating within complex environments. |
| Approach: | They propose a class of tools that can serve as a middleware layer shielding LLMs from environmental complexity. |
| Outcome: | The proposed tool can shield the LLM from environmental complexity in two representative complex environments. |