Modeling Intra- and Inter-Modal Relations: Hierarchical Graph Contrastive Learning for Multimodal Sentiment Analysis (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies in Multimodal Sentiment Analysis lack a mechanism to understand complex relations between different modalities. |
| Approach: | They propose a hierarchical graph contrastive learning framework for multimodal sentiment analysis that explores the relationships between modality representations. |
| Outcome: | The proposed framework outperforms the state-of-the-art in multimodal sentiment analysis on two benchmark datasets. |
Similar Papers
Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on multimodal representation learning has focused on uni-modality pre-training or cross-modalities integration. |
| Approach: | They propose a framework for multimodal representation learning that uses uni-modal contrastive coding and an efficient unimodal feature augmentation strategy to capture intermodal dynamics. |
| Outcome: | The proposed framework surpasses state-of-the-art methods on two public datasets. |
Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies focus on learning cross-modal dynamics, but neglect to explore optimal solution for unimodal networks. |
| Approach: | They propose a new MSA framework to identify contribution of modalities and reduce impact of noisy information. |
| Outcome: | The proposed model outperforms state-of-the-art methods on publicly available datasets. |
CLGSI: A Multimodal Sentiment Analysis Framework based on Contrastive Learning Guided by Sentiment Intensity (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have focused on contrastive learning, but lack detailed learning of the distribution of sample pairs with different sentiment intensity differences in the contrastive training representation space. |
| Approach: | They propose a framework for multimodal sentiment analysis based on contrastive learning guided by sentiment intensity (CLGSI) it selects positive and negative sample pairs based upon sentiment intensity differences and assigns corresponding weights accordingly. |
| Outcome: | The proposed framework extracts common features between different modalities and then uses them to predict sentiment intensity. |
Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimoda Emotion Recognition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing graph-based methods fail to depict global contextual features and local diverse unimodal features in a dialogue. |
| Approach: | They propose a method for joint modality fusion and graph contrastive learning for multimodal emotion recognition using a multimodal fusion mechanism and a graph contrastative learning framework. |
| Outcome: | The proposed method improves multimodal emotion recognition on unbalanced and small-scale emotional datasets. |
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two. |
| Approach: | They propose a multimodal sentiment knowledge-sharing framework that unifies MSA and ERC tasks from features, labels, and models. |
| Outcome: | The proposed framework achieves consistent improvements on four public benchmark datasets on MOSI, MOSEI, MELD, and IEMOCAP. |
Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to classify human affect and subjective information from multiple data sources are limited by the lack of high-level feature associations. |
| Approach: | They propose a hierarchical multimodal architecture with attention and word-level fusion to classify utterance-level sentiment and emotion from text and audio data. |
| Outcome: | The proposed model outperforms state-of-the-art approaches on published datasets and visualizes and interprets synchronized attention over modalities. |
Multi-modal Semantic Understanding with Contrastive Cross-modal Feature Alignment (2024.lrec-main)
Copied to clipboard
| Challenge: | Current work on multi-modal semantic understanding primarily exploits a dual-encoder structure to separate image and text, but fails to learn cross-modal feature alignment. |
| Approach: | They propose a CLIP-guided contrastive-learning-based architecture to perform multi-modal feature alignment by projecting features from different modalities into a unified deep space. |
| Outcome: | The proposed model outperforms baseline models on sarcasm detection and sentiment analysis tasks and is simple to implement without using task-specific external knowledge. |
Affective Knowledge Enhanced Multiple-Graph Fusion Networks for Aspect-based Sentiment Analysis (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for sentiment analysis ignore the roles of syntax dependency relation labels and affective semantic information in determining the sentiment polarity of social media users. |
| Approach: | They propose a new multi-graph fusion network to leverage the richer syntax dependency relation labels and affective semantic information of words. |
| Outcome: | The proposed model outperforms state-of-the-art methods on three datasets. |
Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Sentiment Analysis (MSA) is effective when using rich information from multiple sources, but the potential sentiment-irrelevant information across modalities may hinder the performance from being further improved. |
| Approach: | They propose an Adaptive Language-guided Multimodal Transformer (ALMT) that learns an irrelevance/conflict-suppressing representation from visual and audio features under guidance of language features at different scales. |
| Outcome: | The proposed model achieves state-of-the-art on several popular datasets and an abundance of ablation shows the effectiveness of the proposed model. |
Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs (2025.acl-long)
Copied to clipboard
| Challenge: | Multimodal Sentiment Analysis (MSA) is a rapidly developing field that integrates multimodal information to recognize sentiments. |
| Approach: | They propose a multimodal fusion model that integrates multimodal information to recognize sentiments using multimodal transformers. |
| Outcome: | The proposed model achieves significantly higher performance than MulTs and the existing model is robust. |