Papers with CMU-MOSI
Improving Multimodal fusion via Mutual Dependency Maximisation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal sentiment analysis is a trending area of research, and multimodal fusion is one of its most active topics. |
| Approach: | They propose to use modality-based penalties to measure dependency between models to improve accuracy. |
| Outcome: | The proposed methods improve accuracy on two well-known sentiment analysis datasets by 4.3 on the proposed models and by-product includes a statistical network which can interpret the high dimensional representations learnt by the model. |
SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies focus on learning the joint representation of multiple modalities, ignoring useful knowledge contained in language modal. |
| Approach: | They propose to incorporate sentimental words knowledge into the fusion network to guide the learning of joint representation of multimodal features. |
| Outcome: | The proposed method improves the fusion representation of multimodal features on a YouTube and video dataset. |
Mitigating Inconsistencies in Multimodal Sentiment Analysis under Uncertain Missing Modalities (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies ignore the inconsistency phenomenon of missing modality in multimodal sentiment analysis . neglect of missing modalities may lead to incorrect semantic results . |
| Approach: | They propose an ensemble-based Missing Modality Reconstruction network to detect and recover missing modality features. |
| Outcome: | The proposed method is superior to existing methods on CMU-MOSI and IEMOCAP datasets. |
t-HNE: A Text-guided Hierarchical Noise Eliminator for Multimodal Sentiment Analysis (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis assume that all modalities contribute equally to model performance. |
| Approach: | They propose a text-guided Hierarchical Noise Eliminator model that extracts modality-consistent information from unimodal data and integrates it into multimodal representations for sentiment classification. |
| Outcome: | The proposed model reduces noise caused by modality inconsistency by maximizing mutual information between textual representations and visual and acoustic representations. |
Multimodal Multi-loss Fusion Network for Sentiment Analysis (2024.naacl-long)
Copied to clipboard
| Challenge: | This paper examines the optimal selection and fusion of feature encoders across multiple modalities and combines them in one neural network to improve sentiment detection. |
| Approach: | They propose to combine feature encoders across multiple modalities into one neural network to improve sentiment detection. |
| Outcome: | The proposed model achieves state-of-the-art performance for three datasets . it also shows that integrating context significantly improves model performance. |
Integrating Multimodal Information in Large Pretrained Transformers (2020.acl-main)
Copied to clipboard
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, Ehsan Hoque
| Challenge: | Recent Transformer-based contextual word representations have shown state-of-the-art performance in multiple disciplines within NLP. |
| Approach: | They propose an attachment to BERT and XLNet that allows them to accept multimodal nonverbal data during fine-tuning. |
| Outcome: | The proposed attachment allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. |
MPID: A Modality-Preserving and Interaction-Driven Fusion Network for Multimodal Sentiment Analysis (2025.coling-main)
Copied to clipboard
| Challenge: | Current methods for multimodal sensing analysis overlook nuanced differences and similarities across modalities, leading to potential biases. |
| Approach: | They propose a Modal-Preserving and Interaction-Driven Fusion Network to address these challenges by integrating text with audio and a separate Adaptive Graded Fusion Module for text and visual data. |
| Outcome: | The proposed model achieves state-of-the-art on CMU-MOSI, CMU -MOSEI, and CH-SIMS datasets. |
Contextual Inter-modal Attention for Multi-modal Sentiment Analysis (D18-1)
Copied to clipboard
Deepanway Ghosal, Md Shad Akhtar, Dushyant Chauhan, Soujanya Poria, Asif Ekbal, Pushpak Bhattacharyya
| Challenge: | Existing methods for multi-modal sentiment analysis are limited due to the use of text, visual and acoustic inputs. |
| Approach: | They propose a recurrent neural network based multi-modal attention framework that leverages contextual information for utterance-level sentiment prediction. |
| Outcome: | The proposed framework performs better on two multi-modal sentiment analysis benchmark datasets with accuracies of 82.31% and 79.80% for the MOSI and MOSEI datasets. |
T2DR: A Two-Tier Deficiency-Resistant Framework for Incomplete Multimodal Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing incomplete multimodal learning frameworks are inadequate for integrating multimodal data. |
| Approach: | They propose a framework for incomplete multimodal learning that is deficiency-resistant and provides two modules to address fine-grained deficiencies. |
| Outcome: | The proposed framework outperforms the SOTA models on two well-known multimodal benchmarks. |
Dual-Path Dynamic Fusion with Learnable Query for Multimodal Sentiment Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis struggle with global and fine-grained contributions and over-reliance on text. |
| Approach: | They propose a multimodal sentiment analysis architecture that processes inputs through two complementary paths: global and local. |
| Outcome: | The proposed architecture achieves state-of-the-art in fine-grained sentiment prediction on the CMU-MOSI and CMU MOSEI benchmarks. |
CaM-HG: Causal-Enhanced MoE and Hypergraphs Network for Incomplete Multimodal Emotion Recognition in Conversations (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods that focus on statistical reconstruction often fail to bridge these gaps, effectively leaving semantic holes. |
| Approach: | They propose a Causal-Enhanced Mixture-of-Experts and Hypergraph Network to bridge missing features . they use experts to synthesize missing features that are realistic and causally consistent . |
| Outcome: | The proposed model synthesizes missing features that are realistic and causally consistent . it surpasses benchmarks on IEMOCAP, CMU-MOSI, and CMU MOSEI by 1.43% and 1.25% . |