Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis

Challenge:	Recent studies focus on learning cross-modal dynamics, but neglect to explore optimal solution for unimodal networks.
Approach:	They propose a new MSA framework to identify contribution of modalities and reduce impact of noisy information.
Outcome:	The proposed model outperforms state-of-the-art methods on publicly available datasets.

Similar Papers

Overview

Challenge

Approach

Outcome

Missing Modality meets Meta Sampling (M3S): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modality (2022.aacl-main)

Copied to clipboard

Haozhe Chi, Minghua Yang, Junhao Zhu, Guanhong Wang, Gaoang Wang

Sentiment Analysis

Challenge:	Existing methods to address missing modalities often assume a particular modality is completely missing due to recording or transmission error.
Approach:	They propose a missing modality-based meta-sampling approach for multimodal sentiment analysis with missing modalities . they conduct experiments on IEMOCAP, SIMS and CMU-MOSI datasets .
Outcome:	The proposed method significantly improves on existing models with a mixture of missing modalities.

Analyzing Modality Robustness in Multimodal Sentiment Analysis (2022.naacl-main)

Copied to clipboard

Devamanyu Hazarika, Yingting Li, Bo Cheng, Shuai Zhao, Roger Zimmermann, Soujanya Poria

Robustness in NLP

Challenge:	despite its importance, little attention has been paid to improving the robustness of multimodal models.
Approach:	They propose simple diagnostic checks for modality robustness in a trained multimodal model . they find MSA models highly sensitive to a single modality, which creates issues .
Outcome:	The proposed checks show that models are highly sensitive to a single modality, which creates issues in their robustness.

Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis (2024.findings-emnlp)

Copied to clipboard

Xinyu Feng, Yuming Lin, Lihua He, You Li, Liang Chang, Ya Zhou

Multimodality

Challenge:	Existing methods focus on equally treating the contribution of each modality or statically using text as the dominant modality to conduct interaction, which neglects the situation where each modal may become dominant.
Approach:	They propose a Knowledge-Guided Dynamic Modality Attention Fusion Framework (KuDA) that uses sentiment knowledge to guide the model dynamically selecting the dominant modality and adjusting the contributions of each modality.
Outcome:	The proposed model can be used to highlight the contribution of dominant modality through the correlation evaluation loss.

UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition (2022.emnlp-main)

Copied to clipboard

Guimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, Yongbin Li

Sentiment Analysis

Challenge:	Existing studies study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two.
Approach:	They propose a multimodal sentiment knowledge-sharing framework that unifies MSA and ERC tasks from features, labels, and models.
Outcome:	The proposed framework achieves consistent improvements on four public benchmark datasets on MOSI, MOSEI, MELD, and IEMOCAP.

MPID: A Modality-Preserving and Interaction-Driven Fusion Network for Multimodal Sentiment Analysis (2025.coling-main)

Copied to clipboard

Tianyi Li, Daming Liu

Multimodality

Challenge:	Current methods for multimodal sensing analysis overlook nuanced differences and similarities across modalities, leading to potential biases.
Approach:	They propose a Modal-Preserving and Interaction-Driven Fusion Network to address these challenges by integrating text with audio and a separate Adaptive Graded Fusion Module for text and visual data.
Outcome:	The proposed model achieves state-of-the-art on CMU-MOSI, CMU -MOSEI, and CH-SIMS datasets.

Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis (2022.findings-emnlp)

Copied to clipboard

Ronghao Lin, Haifeng Hu

Semantic Text Processing

Challenge:	Recent work on multimodal representation learning has focused on uni-modality pre-training or cross-modalities integration.
Approach:	They propose a framework for multimodal representation learning that uses uni-modal contrastive coding and an efficient unimodal feature augmentation strategy to capture intermodal dynamics.
Outcome:	The proposed framework surpasses state-of-the-art methods on two public datasets.

Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis (2023.emnlp-main)

Copied to clipboard

Haoyu Zhang, Yu Wang, Guanghao Yin, Kejun Liu, Yuanyuan Liu, Tianshu Yu

Multimodality

Challenge:	Multimodal Sentiment Analysis (MSA) is effective when using rich information from multiple sources, but the potential sentiment-irrelevant information across modalities may hinder the performance from being further improved.
Approach:	They propose an Adaptive Language-guided Multimodal Transformer (ALMT) that learns an irrelevance/conflict-suppressing representation from visual and audio features under guidance of language features at different scales.
Outcome:	The proposed model achieves state-of-the-art on several popular datasets and an abundance of ablation shows the effectiveness of the proposed model.

Modeling Intra- and Inter-Modal Relations: Hierarchical Graph Contrastive Learning for Multimodal Sentiment Analysis (2022.coling-1)

Copied to clipboard

Zijie Lin, Bin Liang, Yunfei Long, Yixue Dang, Min Yang, Min Zhang, Ruifeng Xu

Multimodality

Challenge:	Existing studies in Multimodal Sentiment Analysis lack a mechanism to understand complex relations between different modalities.
Approach:	They propose a hierarchical graph contrastive learning framework for multimodal sentiment analysis that explores the relationships between modality representations.
Outcome:	The proposed framework outperforms the state-of-the-art in multimodal sentiment analysis on two benchmark datasets.

Self-Supervised Unimodal Label Generation Strategy Using Recalibrated Modality Representations for Multimodal Sentiment Analysis (2023.findings-eacl)

Copied to clipboard

Yewon Hwang, Jong-Hwan Kim

Semantic Text Processing

Challenge:	Multimodal sentiment analysis (MSA) has gained much attention over the last few years due to a lack of unimodal annotations in benchmark datasets.
Approach:	They propose a framework which integrates multimodal and unimodal tasks to optimize learning representations from multimodal data.
Outcome:	The proposed model learns to weight features differently based on features of other modalities and auto-generates unimodal annotations via a unimodule.

Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data (2025.acl-long)

Copied to clipboard

Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yiming Tang, Ning An

Robustness in NLP

Challenge:	Existing studies focus on optimizing model structures to handle uncertain missingness, but models still face challenges when dealing with uncertain missing data.
Approach:	They propose a data-centric robust multimodal sentiment analysis method, Proxy-Driven Robust Multimodal Fusion, which maps unimodal data to the latent space of Gaussian distributions to capture core features and structure.
Outcome:	The proposed method outperforms existing models in noise resistance and achieves state-of-the-art performance on multiple benchmark datasets.