SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness (2026.acl-long)
Copied to clipboard
Jingyu Lu, Yuhan Wang, Fan Zhuo, Xize Cheng, Changhao Pan, Xueyi Pu, Yifu Chen, Chenyuhao Wen, Tianle Liang, Zhou Zhao
| Challenge: | SDiaReward is an end-to-end spoken dialogue system that integrates paralinguistic nuances and spontaneous nature of human conversation. |
| Approach: | They propose an end-to-end multi-turn reward model trained on SDiaReward-Dataset . it is a collection of episode-level preference pairs targeting modality and colloquiality gaps . |
| Outcome: | The proposed model outperforms general-purpose audio LLMs in episode-level evaluation. |
Similar Papers
Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models (2026.acl-long)
Copied to clipboard
Yifu Chen, Shengpeng Ji, Zhengqing Liu, Qian Chen, Wen Wang, Ziqing Wang, Yangzhuo Li, Tianle Liang, Zhou Zhao
| Challenge: | Reinforcement learning (RL) has improved text- and vision-language models, but its application in SDMs is hindered. |
| Approach: | They propose a dual-axis Generative Reward Model that provides semantic quality and interaction timing for SDMs. |
| Outcome: | The proposed model achieves state-of-the-art performance on interaction-quality assessment across a wide spectrum of datasets. |
URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios is a major challenge for end-to end spoken dialogue models. |
| Approach: | They propose to provide an extensive evaluation framework for end-to-end spoken dialogue models (SDMs) that includes both cognitive dimensions and paralinguistic cues . |
| Outcome: | The proposed benchmark is divided into two difficulty levels: basic track and pro track, each comprising 20 test sets, evaluating the spoken dialogue model’s abilities in U**nderstanding, **R**easoning, and **O**ral conversation. |
C3: A Bilingual Benchmark for Spoken Dialogue Models Exploring Challenges in Complex Conversations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent developments in spoken dialogue models have created a gap in understanding their effectiveness in comprehending and emulating human conversations. |
| Approach: | They present a benchmark dataset which comprises 1,079 instances in English and Chinese to examine their effectiveness in emulating human conversations. |
| Outcome: | The proposed model outperforms existing models in English and Chinese by using an LLM-based evaluation method that closely aligns with human judgment. |
Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback (2026.findings-acl)
Copied to clipboard
Siddhant Arora, Jinchuan Tian, Jiatong Shi, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Shinji Watanabe
| Challenge: | Existing studies on reinforcement learning from human or AI feedback have focused on semantic rewards at the utterance level. |
| Approach: | They propose a multi-reward RLAIF framework for speech-in/speech-out dialogue systems . they combine semantic, audio-quality, and emotion-consistency rewards . |
| Outcome: | The proposed framework improves speech-in/speech-out dialogue system quality . it combines semantic, audio-quality, and emotion-consistency rewards . the proposed framework is available to download from the cdc. |
Behavior-SD: Behaviorally Aware Spoken Dialogue Generation with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Spoken dialogues lack explicit modeling of behavior traits that are often overlooked in language models . et al.: our work opens new possibilities for developing behaviorally-aware dialogue systems . |
| Approach: | They propose a large-scale dataset with over 100K spoken dialogues (2,164 hours) they propose BeDLM, the first dialogue model capable of generating natural conversations . |
| Outcome: | The proposed model outperforms baseline models in generating natural dialogues . the proposed model can generate natural conversations conditioned on behavioral and narrative contexts - a key feature of spoken language models . |
WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Training (2026.findings-acl)
Copied to clipboard
Yifu Chen, Shengpeng Ji, Qian Chen, Tianle Liang, Yangzhuo Li, Ziqing Wang, Wen Wang, Jingyu Lu, Haoxiao Wang, Xueyi Pu, Fan Zhuo, Zhou Zhao
| Challenge: | End-to-end spoken dialogue models have higher potential ceiling in expressiveness and perceptual ability than cascaded systems. |
| Approach: | They propose a modality-aware adaptive post-training recipe that constrains preference updates to the semantic channel and improves acoustic behavior via explicit anchoring. |
| Outcome: | The proposed model improves speech quality and expressiveness across spoken dialogue benchmarks and architectures. |
SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation (2026.eacl-demo)
Copied to clipboard
Sergio Burdisso, Séverin Baroudi, Yanis Labrak, David Grünert, Pawel Cyrta, Yiyang Chen, Srikanth Madikeri, Esaú Villatoro-tello, Ricard Marxer, Petr Motlicek
| Challenge: | SDialog is an open-source Python toolkit for end-to-end development, simulation, evaluation and analysis of LLM-based conversational agents. |
| Approach: | They present an open-source Python toolkit for end-to-end development, simulation, evaluation and analysis of LLM-based conversational agents. |
| Outcome: | SDialog enables more controlled, transparent, and systematic research on conversational systems. |
Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction (2026.acl-long)
Copied to clipboard
Advait Gosai, Tyler Vuong, Utkarsh Tyagi, Steven Li, Wenjia You, Miheer Bavare, Arda Uçar, Zhongwang Fang, Brian Jang, Bing Liu, Yunzhong He
| Challenge: | End-to-end (E2E) spoken dialogue systems are replacing cascaded pipelines for voice-based human-AI interaction. Existing benchmarks evaluate these systems on synthetic speech and single-turn tasks, leaving multi-turn conversational ability underexplored. |
| Approach: | They propose an open-source benchmark to evaluate spoken dialogue systems under natural multi-turn interaction patterns. |
| Outcome: | The proposed model fails on the highest-performing model with 54.65% pass rate. |
InteractSpeech: A Speech Dialogue Interaction Corpus for Spoken Dialogue Model (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Spoken Dialogue models face challenges in handling nuanced interactional phenomena, such as interruptions and backchannels. |
| Approach: | They propose to use a 150-hour English speech interaction dialogue dataset to empower spoken dialogue models with nuanced real-time interaction capabilities. |
| Outcome: | The proposed dataset trains and evaluates a speech understanding model that classifies key interactional events directly from audio. |
MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues (2024.acl-long)
Copied to clipboard
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, Wanli Ouyang
| Challenge: | Large Language Models (LLMs) have greatly enhanced dialogue systems, but evaluation of their capabilities remains a challenge. |
| Approach: | They propose a model to evaluate the fine-grained abilities of Large Language Models in multi-turn dialogues. |
| Outcome: | The proposed model evaluates 21 popular chatbots based on MT-Bench-101 . it includes 3 overarching abilities and 13 distinct tasks within multi-turn dialogue scenarios. |