Yuan Ge, Haishu Zhao, AoKai Hao, Junxiang Zhang, Bei Li, Xiaoqian Liu, Chenglong Wang, Jianjin Wang, Bingsen Zhou, Bingyu Liu, JingBo Zhu, Zhengtao Yu, Tong Xiao
| Challenge: | Existing models for emotion understanding do not capture fundamental features of synthesized speech. |
| Approach: | They evaluate emotion recognition models on synthesized speech using SER models and generative models. |
| Outcome: | The proposed model can't generalize to synthesized speech because of speech token prediction . generative models tend to infer emotion from textual semantics while ignoring paralinguistic cues. |
Similar Papers
Sentiment Analysis for Emotional Speech Synthesis in a News Dialogue System (2020.coling-main)
Copied to clipboard
| Challenge: | In smart speakers and conversational robots, the demand for expressive speech synthesis has increased. |
| Approach: | They propose to annotate a news dataset with emotion labels for each sentence and to evaluate its effectiveness using the constructed dataset. |
| Outcome: | The proposed method improves the performance of the proposed model by preferentially annotating news articles with low confidence in the human-in-the-loop machine learning framework. |
MELD-ST: An Emotion-aware Speech Translation Dataset (2024.findings-acl)
Copied to clipboard
Sirou Chen, Sakiko Yahata, Shuichiro Shimizu, Zhengdong Yang, Yihang Li, Chenhui Chu, Sadao Kurohashi
| Challenge: | Emotion plays a crucial role in human conversation. |
| Approach: | They present a MELD-ST dataset for the emotion-aware speech translation task . they show that fine-tuning with emotion labels can enhance translation performance . |
| Outcome: | The proposed dataset shows that fine tuning with emotion labels can improve translation performance in some settings. |
Generative Error Correction for Emotion-aware Speech-to-text Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | Despite recent advances in speech-to-text translation, the impact of the emotion content has been overlooked. |
| Approach: | They propose to use generative error correction (GER) to generate the translation based on the decoded N-best hypotheses and combine emotion and sentiment labels into the LLM finetuning process to enable the model to consider the emotion content. |
| Outcome: | The proposed model can translate speech in English-Chinese using GER and emotion and sentiment labels. |
Automatic Dialogue Generation with Expressed Emotions (N18-2)
Copied to clipboard
| Challenge: | a growing interest in neural dialogue generation systems is focusing on generating human-like responses based on past utterances . despite efforts, few consider putting restrictions on the response itself . authors present three models that concatenate the desired emotion with the source input . |
| Approach: | They propose three models that concatenate the desired emotion with the source input or push the emotion in the decoder. |
| Outcome: | The proposed model is more efficient than the previous models, but it lacks the emotion vector. |
A Unified View on Emotion Representation in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent studies show the presence of emotion concepts in the hidden state representations, but it’s unclear if the model has a robust representation consistent across different datasets. |
| Approach: | They propose a unified view to understand emotion representation in Large Language Models by experimenting with diverse datasets and prompts. |
| Outcome: | The proposed model can be interchanged between datasets with minimal impact on performance. |
The MERSA Dataset and a Transformer-Based Approach for Speech Emotion Recognition (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models for speech emotion recognition lack a comprehensive dataset to design accurate models. |
| Approach: | They propose to use a multimodal dataset to build a model that integrates pre-trained wav2vec 2.0 and BERT to learn hidden representations from fused representations of speech and text. |
| Outcome: | The proposed model predicts emotions on dimensions of arousal, valence, and dominance . it achieved competitive results on the MSP-PODCAST dataset . |
Prompt-Guided Selective Masking Loss for Context-Aware Emotive Text-to-Speech (2025.findings-naacl)
Copied to clipboard
| Challenge: | Emotional dialogue speech synthesis (EDSS) aims to generate expressive speech by leveraging the dialogue context between interlocutors. |
| Approach: | They propose a large language model to generate holistic emotion tags based on prior dialogue context and pinpoint key words in the target utterance that align with the predicted emotional state. |
| Outcome: | The proposed method improves emotional expressiveness and facilitates automatic emotion speech generation during inference. |
Textless Speech Emotion Conversion using Discrete & Decomposed Representations (2022.emnlp-main)
Copied to clipboard
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu Anh Nguyen, Morgan Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, Yossi Adi
| Challenge: | Existing methods for modifying emotion of speech are difficult because emotion affects all levels simultaneously. |
| Approach: | They propose a method to convert a spoken language speech into a model of emotion . they use phonetic-content units, prosodic features, speaker, and emotion to modify the emotion a speech utterance has. |
| Outcome: | The proposed method beats text-based systems in terms of perceived emotion and audio quality. |
Modelling Context Emotions using Multi-task Learning for Emotion Controlled Dialog Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent research has tackled this task using neural generative methods by augmenting emotion classes with the input sequences. |
| Approach: | They propose to use a self-attention based encoder and a decoder with dot product attention mechanism to generate a viable response with a specified emotion. |
| Outcome: | The proposed model outperforms baselines on automatic evaluation measures such as F1 and BLEU scores, thus resulting in more fluent and adequate responses. |
Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have demonstrated that Large Language Models possess a form of emotional intelligence, capable of interpreting emotional stimuli in text. |
| Approach: | They propose a method that translates speech characteristics into natural language descriptions and integrates them into LLMs to perform multimodal emotion analysis via text prompts. |
| Outcome: | The proposed method outperforms baseline models that require structural modifications on two datasets showing significant improvements in emotion recognition accuracy. |