Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Sentiment Analysis (MSA) is effective when using rich information from multiple sources, but the potential sentiment-irrelevant information across modalities may hinder the performance from being further improved. |
| Approach: | They propose an Adaptive Language-guided Multimodal Transformer (ALMT) that learns an irrelevance/conflict-suppressing representation from visual and audio features under guidance of language features at different scales. |
| Outcome: | The proposed model achieves state-of-the-art on several popular datasets and an abundance of ablation shows the effectiveness of the proposed model. |
Similar Papers
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two. |
| Approach: | They propose a multimodal sentiment knowledge-sharing framework that unifies MSA and ERC tasks from features, labels, and models. |
| Outcome: | The proposed framework achieves consistent improvements on four public benchmark datasets on MOSI, MOSEI, MELD, and IEMOCAP. |
Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focus on equally treating the contribution of each modality or statically using text as the dominant modality to conduct interaction, which neglects the situation where each modal may become dominant. |
| Approach: | They propose a Knowledge-Guided Dynamic Modality Attention Fusion Framework (KuDA) that uses sentiment knowledge to guide the model dynamically selecting the dominant modality and adjusting the contributions of each modality. |
| Outcome: | The proposed model can be used to highlight the contribution of dominant modality through the correlation evaluation loss. |
Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies focus on learning cross-modal dynamics, but neglect to explore optimal solution for unimodal networks. |
| Approach: | They propose a new MSA framework to identify contribution of modalities and reduce impact of noisy information. |
| Outcome: | The proposed model outperforms state-of-the-art methods on publicly available datasets. |
CLGSI: A Multimodal Sentiment Analysis Framework based on Contrastive Learning Guided by Sentiment Intensity (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have focused on contrastive learning, but lack detailed learning of the distribution of sample pairs with different sentiment intensity differences in the contrastive training representation space. |
| Approach: | They propose a framework for multimodal sentiment analysis based on contrastive learning guided by sentiment intensity (CLGSI) it selects positive and negative sample pairs based upon sentiment intensity differences and assigns corresponding weights accordingly. |
| Outcome: | The proposed framework extracts common features between different modalities and then uses them to predict sentiment intensity. |
Improving Multimodal Sentiment Analysis: Supervised Angular margin-based Contrastive Learning for Enhanced Fusion Representation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis focus on forming positive and negative pairs, neglecting the variation in sentiment scores within the same class. |
| Approach: | They propose a framework to enhance discrimination and generalizability of the multimodal representation and overcome biases in the fusion vector’s modality. |
| Outcome: | The proposed model improves discrimination and generalizability of the multimodal representation and overcomes biases in the fusion vector’s modality. |
Sequential Fusion of Text-close and Text-far Representations for Multimodal Sentiment Analysis (2025.coling-main)
Copied to clipboard
| Challenge: | Multimodal Sentiment Analysis (MSA) aims to identify human attitudes from diverse modalities such as visual, audio and text. |
| Approach: | They propose a framework to combine text-close and text-far representations to refine multimodal representations from multimodal data. |
| Outcome: | The proposed framework explores similarities and differences between text and audio/visual modalities and fuses extracted representations more effectively. |
Word-Aware Modality Stimulation for Multimodal Fusion (2024.lrec-main)
Copied to clipboard
| Challenge: | Multimodal learning is expected to make more accurate predictions than text-only analysis. |
| Approach: | They propose a method for fusing multimodal inputs with text-based fusion methods . they propose fusion that integrates non-verbal modalities with text . |
| Outcome: | The proposed method improves sentiment prediction by using non-verbal modalities with text . the proposed method is unsuitable for applying attention to text modality in the fusion phase . |
Self-supervised Cross-modal Pretraining for Speech Emotion Recognition and Sentiment Analysis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multimodal speech emotion recognition and sentiment analysis have not improved results due to their relatively simple fusion mechanisms and lack of proper cross-modal pretraining. |
| Approach: | They propose a deep-fused audio-text bi-modal transformer with carefully designed cross-modal fusion mechanism and stage-wise cross-mod pretraining scheme to facilitate cross-modulation. |
| Outcome: | The proposed method exceeds benchmarks on public IEMOCAP emotion and CMU-MOSEI sentiment datasets by a large margin. |
Self-Supervised Unimodal Label Generation Strategy Using Recalibrated Modality Representations for Multimodal Sentiment Analysis (2023.findings-eacl)
Copied to clipboard
| Challenge: | Multimodal sentiment analysis (MSA) has gained much attention over the last few years due to a lack of unimodal annotations in benchmark datasets. |
| Approach: | They propose a framework which integrates multimodal and unimodal tasks to optimize learning representations from multimodal data. |
| Outcome: | The proposed model learns to weight features differently based on features of other modalities and auto-generates unimodal annotations via a unimodule. |
MPID: A Modality-Preserving and Interaction-Driven Fusion Network for Multimodal Sentiment Analysis (2025.coling-main)
Copied to clipboard
| Challenge: | Current methods for multimodal sensing analysis overlook nuanced differences and similarities across modalities, leading to potential biases. |
| Approach: | They propose a Modal-Preserving and Interaction-Driven Fusion Network to address these challenges by integrating text with audio and a separate Adaptive Graded Fusion Module for text and visual data. |
| Outcome: | The proposed model achieves state-of-the-art on CMU-MOSI, CMU -MOSEI, and CH-SIMS datasets. |