Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition

Challenge:	Existing methods to predict emotion label for a given utterance lack modeling of diverse dependency ranges and inconsistent treatment of contribution for various modalities.
Approach:	They propose a multimodal emotion recognition in conversation task that uses context and multiple modalities to predict emotion label for a given utterance.
Outcome:	The proposed method outperforms the state-of-the-art methods on three multimodal datasets.

Similar Papers

Overview

Challenge

Approach

Outcome

Multi-Condition Guided Diffusion Network for Multimodal Emotion Recognition in Conversation (2025.findings-naacl)

Copied to clipboard

Wenjin Tian, Xianying Huang, Shihao Zou

Sentiment Analysis

Challenge:	Current research emphasizes contextual factors, the speaker’s influence, and extracting complementary information across different modalities.
Approach:	They propose a diffusion-based approach to address the challenges posed by redundant information and redundant information at the semantic level while robustly capturing shared semantics.
Outcome:	The proposed model outperforms existing state-of-the-art models on two multimodal datasets and is generalizable and effective.

Self-supervised Cross-modal Pretraining for Speech Emotion Recognition and Sentiment Analysis (2022.findings-emnlp)

Copied to clipboard

Iek-Heng Chu, Ziyi Chen, Xinlu Yu, Mei Han, Jing Xiao, Peng Chang

Multimodality

Challenge:	Existing approaches to multimodal speech emotion recognition and sentiment analysis have not improved results due to their relatively simple fusion mechanisms and lack of proper cross-modal pretraining.
Approach:	They propose a deep-fused audio-text bi-modal transformer with carefully designed cross-modal fusion mechanism and stage-wise cross-mod pretraining scheme to facilitate cross-modulation.
Outcome:	The proposed method exceeds benchmarks on public IEMOCAP emotion and CMU-MOSEI sentiment datasets by a large margin.

A Dual Contrastive Learning Framework for Enhanced Multimodal Conversational Emotion Recognition (2025.coling-main)

Copied to clipboard

Yunhe Xie, Chengjie Sun, Ziyi Cao, Bingquan Liu, Zhenzhou Ji, Yuanchao Liu, Lili Shan

Semantic Text Processing

Challenge:	Existing methods struggle to capture emotion shifts due to label replication and fail to preserve positive independent modality contributions during fusion.
Approach:	They propose a Dual Contrastive Learning Framework that enhances existing MERC models without additional data.
Outcome:	The proposed framework outperforms existing models on two MERC benchmark datasets and shows that it reduces label dependence and enhances emotion-sensitive independent modality features.

MMDAG: Multimodal Directed Acyclic Graph Network for Emotion Recognition in Conversation (2022.lrec-1)

Copied to clipboard

Shuo Xu, Yuxiang Jia, Changyong Niu, Hongying Zan

Multimodality

Challenge:	Emotion recognition in conversation is important for an empathetic dialogue system to understand the user’s emotion and then generate appropriate emotional responses.
Approach:	They propose to use multimodal directed acyclic graphs to integrate multimodal information and contextual information into a DAG architecture to exploit multimodal contexts.
Outcome:	Comparative studies on IEMOCAP and MELD show that the proposed model outperforms state-of-the-art models.

Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects (2025.findings-emnlp)

Copied to clipboard

ChengYan Wu, Yiqiang Cai, Yang Liu, Pengxu Zhu, Yun Xue, Ziwei Gong, Julia Hirschberg, Bolei Ma

Sentiment Analysis

Challenge:	Multimodal Emotion Recognition in Conversations (MERC) is a new way to enhance human-computer interaction.
Approach:	This survey offers a systematic overview of Multimodal Emotion Recognition in Conversations . it examines motivations, core tasks, representative methods, and evaluation strategies .
Outcome:	The survey examines the effectiveness of MERC and its evaluation strategies.

UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition (2022.emnlp-main)

Copied to clipboard

Guimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, Yongbin Li

Sentiment Analysis

Challenge:	Existing studies study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two.
Approach:	They propose a multimodal sentiment knowledge-sharing framework that unifies MSA and ERC tasks from features, labels, and models.
Outcome:	The proposed framework achieves consistent improvements on four public benchmark datasets on MOSI, MOSEI, MELD, and IEMOCAP.

Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis (N19-1)

Copied to clipboard

Md Shad Akhtar, Dushyant Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, Pushpak Bhattacharyya

Language Models

Challenge:	Existing frameworks for sentiment and emotion analysis are not efficient for inter-task learning.
Approach:	They propose a multi-task learning framework that performs sentiment and emotion analysis together.
Outcome:	The proposed framework improves on a CMU-MOSEI dataset for sentiment and emotion analysis.

DialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation (2021.findings-emnlp)

Copied to clipboard

Yuzhao Mao, Guang Liu, Xiaojie Wang, Weiguo Gao, Xuan Li

Dialogue Systems & Conversational Agents

Challenge:	Existing studies focus on the self and inter-personal dependencies in multi-modal conversations, but they ignore the temporal and spatial dependencies.
Approach:	They propose a Dialogue Transformer for simultaneously modeling the intra-modal and inter-modal emotion dynamics.
Outcome:	The proposed models outperform the state-of-the-art on three benchmark datasets.

Context-aware Interactive Attention for Multi-modal Sentiment and Emotion Analysis (D19-1)

Copied to clipboard

Dushyant Singh Chauhan, Md Shad Akhtar, Asif Ekbal, Pushpak Bhattacharyya

Sentiment Analysis

Challenge:	Multi-modal analysis is a field emerging in the fields of natural language processing, computer vision and speech processing . multimodal analysis uses a variety of information from multiple sources to build efficient systems . acoustic and visual information can provide better information for classification decisions .
Approach:	They propose a recurrent neural network based approach for multi-modal sentiment and emotion analysis . they employ a context-aware attention module to exploit the correspondence among neighboring utterances .
Outcome:	The proposed model learns inter-modal interaction among participating modalities through auto-encoder mechanism . it is compared with existing state-of-the-art models on five standard multi-modal affect analysis datasets .

GRPO-Guided Modality Selection Enhanced LoRA-Tuned LLMs for Multimodal Emotion Recognition (2025.findings-emnlp)

Copied to clipboard

Yang Chen, Shuwan Yang, Yan Xiang, Ran Song, Yuxin Huang, Zhengtao Yu

Sentiment Analysis

Challenge:	Multimodal emotion recognition in conversation (MERC) aims to identify speakers’ emotional states by utilizing text, audio, and visual modalities.
Approach:	They propose an adaptive modality selection framework for multimodal emotion recognition in conversation that integrates all available modalities into one .
Outcome:	The proposed framework outperforms existing methods on multimodal dialogue datasets and is available at https://github.com/youflyaway/Modality-Selection-Enhanced-LoRA-Tuned-LLMs.