Latent Distribution Decouple for Uncertain-Aware Multimodal Multi-label Emotion Recognition

Challenge:	Existing studies focus on improving fusion strategies and modeling modality-to-label dependencies, but they overlook the impact of aleatoric uncertainty, which is inherent noise in multimodal data.
Approach:	They propose a latent emotional distribution decomposition with uncertainty perception framework to model aleatoric uncertainty in multimodal data.
Outcome:	The proposed framework achieves state-of-the-art performance on the CMU-MOSEI and M3ED datasets, highlighting the importance of uncertainty modeling in MMER.

Similar Papers

Overview

Challenge

Approach

Outcome

UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition (2022.emnlp-main)

Copied to clipboard

Guimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, Yongbin Li

Sentiment Analysis

Challenge:	Existing studies study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two.
Approach:	They propose a multimodal sentiment knowledge-sharing framework that unifies MSA and ERC tasks from features, labels, and models.
Outcome:	The proposed framework achieves consistent improvements on four public benchmark datasets on MOSI, MOSEI, MELD, and IEMOCAP.

Handling Ambiguity in Emotion: From Out-of-Domain Detection to Distribution Estimation (2024.acl-long)

Copied to clipboard

Wen Wu, Bo Li, Chao Zhang, Chung-Cheng Chiu, Qiujia Li, Junwen Bai, Tara Sainath, Phil Woodland

Information Extraction & Text Mining

Challenge:	Experimental results show that incorporating utterances without majority-agreed labels into an additional class reduces the classification performance of the other emotion classes.
Approach:	They propose to combine utterances without majority-agreed labels into an additional class . they propose to quantify uncertainty in emotion classification using evidential deep learning .
Outcome:	The proposed method retains classification accuracy while effectively detects ambiguous emotion expressions.

Integrating Representation Subspace Mapping with Unimodal Auxiliary Loss for Attention-based Multimodal Emotion Recognition (2024.lrec-main)

Copied to clipboard

Xulong Du, Xingnan Zhang, Dandan Wang, Yingying Xu, Zhiyuan Wu, Shiqing Zhang, Xiaoming Zhao, Jun Yu, Liangliang Lou

Sentiment Analysis

Challenge:	Existing methods to identify emotions rely on a large modality gap in their representations .
Approach:	They propose a representation subspace mapping module that maps each modality into two distinct subspaces and a cross-modality attention module that leverages auxiliary loss to remove the noise unrelated to emotion classification.
Outcome:	The proposed approach achieves superior performance to state-of-the-art MER methods on the IEMOCAP and MSP-Improv datasets.

Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021.acl-long)

Copied to clipboard

Jinming Zhao, Ruichen Li, Qin Jin

Sentiment Analysis

Challenge:	Existing multimodal fusion models trained on full-modality samples fail when partial modalities are missing.
Approach:	They propose a model to deal with the uncertain missing modality problem by learning robust joint multimodal representations that can predict the representation of any missing modal given available modalities under different missing-modality conditions.
Outcome:	The proposed model significantly improves performance under uncertain missing-modality testing conditions and full-modalities ideal testing conditions.

Dynamic Emotion and Personality Profiling for Multimodal Deception Detection (2026.acl-long)

Copied to clipboard

Li Zheng, Yanyi Luo, Hao Fei, Yuzhe Ding, Yujie Huang, Fei Li, Chong Teng, Donghong Ji

Multimodality

Challenge:	Existing methods for deception detection lack sample-level dynamic annotations for emotions and personality.
Approach:	They propose a multi-model multi-prompt annotation scheme and a strict label quality evaluation standard for deception, emotion, and personality annotations.
Outcome:	The proposed framework outperforms state-of-the-art models on the MDPE and DDEP datasets.

Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data (2025.acl-long)

Copied to clipboard

Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yiming Tang, Ning An

Robustness in NLP

Challenge:	Existing studies focus on optimizing model structures to handle uncertain missingness, but models still face challenges when dealing with uncertain missing data.
Approach:	They propose a data-centric robust multimodal sentiment analysis method, Proxy-Driven Robust Multimodal Fusion, which maps unimodal data to the latent space of Gaussian distributions to capture core features and structure.
Outcome:	The proposed method outperforms existing models in noise resistance and achieves state-of-the-art performance on multiple benchmark datasets.

MPID: A Modality-Preserving and Interaction-Driven Fusion Network for Multimodal Sentiment Analysis (2025.coling-main)

Copied to clipboard

Tianyi Li, Daming Liu

Multimodality

Challenge:	Current methods for multimodal sensing analysis overlook nuanced differences and similarities across modalities, leading to potential biases.
Approach:	They propose a Modal-Preserving and Interaction-Driven Fusion Network to address these challenges by integrating text with audio and a separate Adaptive Graded Fusion Module for text and visual data.
Outcome:	The proposed model achieves state-of-the-art on CMU-MOSI, CMU -MOSEI, and CH-SIMS datasets.

MMDAG: Multimodal Directed Acyclic Graph Network for Emotion Recognition in Conversation (2022.lrec-1)

Copied to clipboard

Shuo Xu, Yuxiang Jia, Changyong Niu, Hongying Zan

Multimodality

Challenge:	Emotion recognition in conversation is important for an empathetic dialogue system to understand the user’s emotion and then generate appropriate emotional responses.
Approach:	They propose to use multimodal directed acyclic graphs to integrate multimodal information and contextual information into a DAG architecture to exploit multimodal contexts.
Outcome:	Comparative studies on IEMOCAP and MELD show that the proposed model outperforms state-of-the-art models.

Mixture of Multimodal Adapters for Sentiment Analysis (2025.naacl-long)

Copied to clipboard

Kezhou Chen, Shuo Wang, Huixia Ben, Shengeng Tang, Yanbin Hao

Language Models

Challenge:	Pre-trained language models (PLMs) have been used for text sentiment analysis but sentiment is hidden in other modalities.
Approach:	They propose to fuse emotions from different data to analyze sentiments . they use compression parameter for each expert to reduce training burden .
Outcome:	The proposed method achieves state-of-the-art with a tiny trainable parameter count compared to current methods . emotions hidden in body movements or vocal timbres eclipse traditional methods compared with text sentiment analysis .

Towards Label-Agnostic Emotion Embeddings (2021.emnlp-main)

Copied to clipboard

Sven Buechel, Luise Modersohn, Udo Hahn

Semantic Text Processing

Challenge:	Existing representation schemes for emotion analysis are based on label formats, natural languages, and even disparate model architectures.
Approach:	They propose a training scheme that learns a shared latent representation of emotion independent from different label formats, natural languages, and even disparate model architectures.
Outcome:	The proposed model performs well on a wide range of datasets without penalizing prediction quality.