| Challenge: | BLSP-Emo model understands both semantics and emotions in speech and generates empathetic responses. |
| Approach: | They propose a language-speech pretraining with emotion support that utilizes existing speech and emotion recognition datasets to create an end-to-end speech-language model. |
| Outcome: | The proposed model can understand both semantics and emotions in speech and generate empathetic responses. |
Similar Papers
OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model (2025.emnlp-demos)
Copied to clipboard
Chen Wang, Tianyu Peng, Wen Yang, YiNan Bai, Guangfu Wang, Jun Lin, Lanpeng Jia, Lingxiang Wu, Jinqiao Wang, Chengqing Zong, Jiajun Zhang
| Challenge: | Empathetic speech models are increasingly closed off, leaving details about the architecture, data and development opaque to researchers. |
| Approach: | They propose an open-source empathetic speech-to-text model with a streaming interleaved decoding architecture and a data pipeline to enable end-to end training. |
| Outcome: | The proposed model is open-source and transparent, with no data or data required to build it. |
ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing speech-to-speech large language models rely on ASR transcription or use encoders to extract latent representations, weakening affective information and contextual coherence in multi-turn dialogues. |
| Approach: | They propose a framework for speech-based empathetic response generation that captures turn-level affective states and dialogue-level emotional dynamics. |
| Outcome: | The proposed framework outperforms baselines in automatic and human evaluations and remains robust across different Large Language Model (LLM) backbones. |
Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvements (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Empathetic dialogue is an essential part of building harmonious social relationships and contributes to the development of a helpful AI. |
| Approach: | They propose three methods to improve the performance of large language models (LLMs) they propose semantically similar in-context learning, two-stage interactive generation and combination with the knowledge base. |
| Outcome: | The proposed methods achieve state-of-the-art in automatic and human evaluations and the possibility of GPT-4 simulating human evaluators. |
Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | AVLM integrates full-face visual cues into a pre-trained expressive speech model. |
| Approach: | They propose an Audio-Visual Language Model (AVLM) for expressive speech generation by integrating full-face visual cues into a pre-trained expressive speech model. |
| Outcome: | The proposed model incorporates full-face visual cues into a pre-trained expressive speech model. |
EMO-RL: Emotion-Rule-Based Reinforcement Learning Enhanced Audio-Language Model for Generalized Speech Emotion Recognition (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in reinforcement learning (RL) have shown promise in improving LALMs’ reasoning abilities, but their performance in affective computing tasks remains suboptimal. |
| Approach: | They propose a framework incorporating reinforcement learning with two key innovations: Emotion Similarity-Weighted Reward (ESWR) and Explicit Structured Reasoning (ESR). |
| Outcome: | The proposed framework improves LALMs' reasoning abilities on MELD and IEMOCAP datasets and shows strong generalization. |
A Large-Scale Dataset for Empathetic Response Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing empathetic datasets are limited in size and cost due to the cost of manual labor. |
| Approach: | They propose to annotate 1M dialogues with 32 fine-grained emotions and eight empathetic response intents and the Neutral category using a silver dataset. |
| Outcome: | The proposed pipeline compares the quality of the proposed dataset with a state-of-the-art gold dataset using offline experiments and visual validation methods. |
EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks fail to achieve ecological validity, signal clarity, and reliable fine-grained labeling in multimodal Emotion Recognition (MER) Existing datasets lack spontaneity of real-life interactions, resulting in poor quality and inconsistent data quality. |
| Approach: | They propose a bilingual benchmark to resolve limitations of ecological validity and noise in existing datasets by combining strictly filtered static slices with a dynamic Streaming Monologue subset. |
| Outcome: | EmoS provides trusted ground truth that captures continuous emotional evolution. |
Constructing Emotional Consensus and Utilizing Unpaired Data for Empathetic Dialogue Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for dialogue empathy focus on the emotion flow in one direction, from context to response. |
| Approach: | They propose a dual-generative model to construct emotional consensus and use unpaired data to produce pseudo paired empathetic samples. |
| Outcome: | The proposed model outperforms baseline models in producing coherent and empathetic responses. |
When Words Smile: Generating Diverse Emotional Facial Expressions from Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems that generate only coarse facial expressions ignore the rich and dynamic nature of face-to-face communication. |
| Approach: | They propose an end-to-end text-to expression model that explicitly focuses on emotional dynamics. |
| Outcome: | The proposed model outperforms baselines on 15,000 text–3D expression pairs on a large-scale dataset. |
The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Modern artificial intelligence is characterized by large pretrained language models with strong language capabilities to be adapted to various downstream tasks. |
| Approach: | They propose to use the task of speech translation (ST) to pretrain speech models for end-to-end SLU on intra- and cross-lingual scenarios. |
| Outcome: | The proposed model achieves higher performance over baselines on monolingual and multilingual intent classification as well as spoken question answering using SLURP, MINDS-14, and NMSQA benchmarks. |