Papers with CMU-MOSEI
Improving Multimodal fusion via Mutual Dependency Maximisation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal sentiment analysis is a trending area of research, and multimodal fusion is one of its most active topics. |
| Approach: | They propose to use modality-based penalties to measure dependency between models to improve accuracy. |
| Outcome: | The proposed methods improve accuracy on two well-known sentiment analysis datasets by 4.3 on the proposed models and by-product includes a statistical network which can interpret the high dimensional representations learnt by the model. |
Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis (N19-1)
Copied to clipboard
Md Shad Akhtar, Dushyant Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, Pushpak Bhattacharyya
| Challenge: | Existing frameworks for sentiment and emotion analysis are not efficient for inter-task learning. |
| Approach: | They propose a multi-task learning framework that performs sentiment and emotion analysis together. |
| Outcome: | The proposed framework improves on a CMU-MOSEI dataset for sentiment and emotion analysis. |
SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies focus on learning the joint representation of multiple modalities, ignoring useful knowledge contained in language modal. |
| Approach: | They propose to incorporate sentimental words knowledge into the fusion network to guide the learning of joint representation of multimodal features. |
| Outcome: | The proposed method improves the fusion representation of multimodal features on a YouTube and video dataset. |
t-HNE: A Text-guided Hierarchical Noise Eliminator for Multimodal Sentiment Analysis (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis assume that all modalities contribute equally to model performance. |
| Approach: | They propose a text-guided Hierarchical Noise Eliminator model that extracts modality-consistent information from unimodal data and integrates it into multimodal representations for sentiment classification. |
| Outcome: | The proposed model reduces noise caused by modality inconsistency by maximizing mutual information between textual representations and visual and acoustic representations. |
Multimodal Multi-loss Fusion Network for Sentiment Analysis (2024.naacl-long)
Copied to clipboard
| Challenge: | This paper examines the optimal selection and fusion of feature encoders across multiple modalities and combines them in one neural network to improve sentiment detection. |
| Approach: | They propose to combine feature encoders across multiple modalities into one neural network to improve sentiment detection. |
| Outcome: | The proposed model achieves state-of-the-art performance for three datasets . it also shows that integrating context significantly improves model performance. |
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph (P18-1)
Copied to clipboard
| Challenge: | Analyzing human multimodal language is emerging area of research in NLP. |
| Approach: | They propose a multimodal fusion technique to exploit how modalities interact in multimodal language. |
| Outcome: | The proposed technique exploits how modalities interact with each other in human multimodal language. |
Integrating Multimodal Information in Large Pretrained Transformers (2020.acl-main)
Copied to clipboard
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, Ehsan Hoque
| Challenge: | Recent Transformer-based contextual word representations have shown state-of-the-art performance in multiple disciplines within NLP. |
| Approach: | They propose an attachment to BERT and XLNet that allows them to accept multimodal nonverbal data during fine-tuning. |
| Outcome: | The proposed attachment allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. |
Towards Speech-only Opinion-level Sentiment Analysis (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing systems that estimate user preferences only in static manners or exploit interaction history are inadequate to accurately assess user preferences. |
| Approach: | They propose to integrate rank consistent ordinal regression into a speech-only sentiment prediction task performed by ResNet-like systems and use speaker verification extractors trained on larger datasets as low-level feature extractor. |
| Outcome: | The proposed system beats state-of-the-art unimodal systems on multimodal Opinion Sentiment and Emotion Intensity databases. |
MPID: A Modality-Preserving and Interaction-Driven Fusion Network for Multimodal Sentiment Analysis (2025.coling-main)
Copied to clipboard
| Challenge: | Current methods for multimodal sensing analysis overlook nuanced differences and similarities across modalities, leading to potential biases. |
| Approach: | They propose a Modal-Preserving and Interaction-Driven Fusion Network to address these challenges by integrating text with audio and a separate Adaptive Graded Fusion Module for text and visual data. |
| Outcome: | The proposed model achieves state-of-the-art on CMU-MOSI, CMU -MOSEI, and CH-SIMS datasets. |
Self-supervised Cross-modal Pretraining for Speech Emotion Recognition and Sentiment Analysis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multimodal speech emotion recognition and sentiment analysis have not improved results due to their relatively simple fusion mechanisms and lack of proper cross-modal pretraining. |
| Approach: | They propose a deep-fused audio-text bi-modal transformer with carefully designed cross-modal fusion mechanism and stage-wise cross-mod pretraining scheme to facilitate cross-modulation. |
| Outcome: | The proposed method exceeds benchmarks on public IEMOCAP emotion and CMU-MOSEI sentiment datasets by a large margin. |
Contextual Inter-modal Attention for Multi-modal Sentiment Analysis (D18-1)
Copied to clipboard
Deepanway Ghosal, Md Shad Akhtar, Dushyant Chauhan, Soujanya Poria, Asif Ekbal, Pushpak Bhattacharyya
| Challenge: | Existing methods for multi-modal sentiment analysis are limited due to the use of text, visual and acoustic inputs. |
| Approach: | They propose a recurrent neural network based multi-modal attention framework that leverages contextual information for utterance-level sentiment prediction. |
| Outcome: | The proposed framework performs better on two multi-modal sentiment analysis benchmark datasets with accuracies of 82.31% and 79.80% for the MOSI and MOSEI datasets. |
From Coarse to Fine: A Multi-Granularity Multimodal Framework for Teacher Sentiment Analysis (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to teacher sentiment analysis treat it as a static label . current approaches fail to capture structured heterogeneity of classroom expressions . |
| Approach: | They propose a coarse-to-fine multimodal framework that decomposes teacher sentiment into three granularities and employ CLS-guided cross-modal attention to recover effective signals from regulated displays. |
| Outcome: | The proposed framework outperforms state-of-the-art models on T-MED and CMU-MOSEI. |
T2DR: A Two-Tier Deficiency-Resistant Framework for Incomplete Multimodal Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing incomplete multimodal learning frameworks are inadequate for integrating multimodal data. |
| Approach: | They propose a framework for incomplete multimodal learning that is deficiency-resistant and provides two modules to address fine-grained deficiencies. |
| Outcome: | The proposed framework outperforms the SOTA models on two well-known multimodal benchmarks. |
Dual-Path Dynamic Fusion with Learnable Query for Multimodal Sentiment Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis struggle with global and fine-grained contributions and over-reliance on text. |
| Approach: | They propose a multimodal sentiment analysis architecture that processes inputs through two complementary paths: global and local. |
| Outcome: | The proposed architecture achieves state-of-the-art in fine-grained sentiment prediction on the CMU-MOSI and CMU MOSEI benchmarks. |
CaM-HG: Causal-Enhanced MoE and Hypergraphs Network for Incomplete Multimodal Emotion Recognition in Conversations (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods that focus on statistical reconstruction often fail to bridge these gaps, effectively leaving semantic holes. |
| Approach: | They propose a Causal-Enhanced Mixture-of-Experts and Hypergraph Network to bridge missing features . they use experts to synthesize missing features that are realistic and causally consistent . |
| Outcome: | The proposed model synthesizes missing features that are realistic and causally consistent . it surpasses benchmarks on IEMOCAP, CMU-MOSI, and CMU MOSEI by 1.43% and 1.25% . |
Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Emotion recognition is a crucial task for human conversation understanding . multimodal data, e.g., language, voice, and facial expressions, add complexity to the task. |
| Approach: | They propose a relational temporal Graph Neural Network with Auxiliary Cross-Modality Interaction framework that captures conversation-level cross-modality interactions and utterance-level temporal dependencies with modality-specific manner for conversation understanding. |
| Outcome: | The proposed framework captures conversation-level cross-modality interactions and utterance-level temporal dependencies with the modality-specific manner for conversation understanding. |
Latent Distribution Decouple for Uncertain-Aware Multimodal Multi-label Emotion Recognition (2025.findings-acl)
Copied to clipboard
Jingwang Huang, Jiang Zhong, Qin Lei, Gaojinpeng Gaojinpeng, Ymyang Ymyang, Sirui Wang, PeiguangLi PeiguangLi, Kaiwen Wei
| Challenge: | Existing studies focus on improving fusion strategies and modeling modality-to-label dependencies, but they overlook the impact of aleatoric uncertainty, which is inherent noise in multimodal data. |
| Approach: | They propose a latent emotional distribution decomposition with uncertainty perception framework to model aleatoric uncertainty in multimodal data. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the CMU-MOSEI and M3ED datasets, highlighting the importance of uncertainty modeling in MMER. |