Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing models for multimodal sentiment analysis are limited in their capacity to be deployed in the real world. |
| Approach: | They propose a model that can dynamically refine erroneous sentiment words by leveraging multimodal sentiment clues. |
| Outcome: | The proposed model surpasses the state-of-the-art models on three datasets. |
Similar Papers
Improving Multimodal Sentiment Analysis: Supervised Angular margin-based Contrastive Learning for Enhanced Fusion Representation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis focus on forming positive and negative pairs, neglecting the variation in sentiment scores within the same class. |
| Approach: | They propose a framework to enhance discrimination and generalizability of the multimodal representation and overcome biases in the fusion vector’s modality. |
| Outcome: | The proposed model improves discrimination and generalizability of the multimodal representation and overcomes biases in the fusion vector’s modality. |
Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Sentiment Analysis (MSA) is effective when using rich information from multiple sources, but the potential sentiment-irrelevant information across modalities may hinder the performance from being further improved. |
| Approach: | They propose an Adaptive Language-guided Multimodal Transformer (ALMT) that learns an irrelevance/conflict-suppressing representation from visual and audio features under guidance of language features at different scales. |
| Outcome: | The proposed model achieves state-of-the-art on several popular datasets and an abundance of ablation shows the effectiveness of the proposed model. |
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two. |
| Approach: | They propose a multimodal sentiment knowledge-sharing framework that unifies MSA and ERC tasks from features, labels, and models. |
| Outcome: | The proposed framework achieves consistent improvements on four public benchmark datasets on MOSI, MOSEI, MELD, and IEMOCAP. |
Tackling Modality Heterogeneity with Multi-View Calibration Network for Multimodal Sentiment Detection (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on fusing different features but ignore the challenge of modality heterogeneity. |
| Approach: | They propose a text-guided fusion module with novel Sparse-Attention to reduce the negative impacts of redundant visual elements and a sentiment-based congruity constraint task to calibrate the feature shift in the representation space. |
| Outcome: | The proposed model is competitive against existing methods and achieves state-of-the-art results on two public benchmark datasets. |
Word-Aware Modality Stimulation for Multimodal Fusion (2024.lrec-main)
Copied to clipboard
| Challenge: | Multimodal learning is expected to make more accurate predictions than text-only analysis. |
| Approach: | They propose a method for fusing multimodal inputs with text-based fusion methods . they propose fusion that integrates non-verbal modalities with text . |
| Outcome: | The proposed method improves sentiment prediction by using non-verbal modalities with text . the proposed method is unsuitable for applying attention to text modality in the fusion phase . |
Analyzing Modality Robustness in Multimodal Sentiment Analysis (2022.naacl-main)
Copied to clipboard
| Challenge: | despite its importance, little attention has been paid to improving the robustness of multimodal models. |
| Approach: | They propose simple diagnostic checks for modality robustness in a trained multimodal model . they find MSA models highly sensitive to a single modality, which creates issues . |
| Outcome: | The proposed checks show that models are highly sensitive to a single modality, which creates issues in their robustness. |
SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies focus on learning the joint representation of multiple modalities, ignoring useful knowledge contained in language modal. |
| Approach: | They propose to incorporate sentimental words knowledge into the fusion network to guide the learning of joint representation of multimodal features. |
| Outcome: | The proposed method improves the fusion representation of multimodal features on a YouTube and video dataset. |
RethinkingTMSC: An Empirical Study for Target-Oriented Multimodal Sentiment Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that current TMSC systems rely on textual information, and the progress in tackling this task has slowed down. |
| Approach: | They propose to integrate both visual and textual information to improve the performance of TMSC by considering multimodal information. |
| Outcome: | The proposed model integrates both visual and textual information to improve performance. |
Self-Supervised Unimodal Label Generation Strategy Using Recalibrated Modality Representations for Multimodal Sentiment Analysis (2023.findings-eacl)
Copied to clipboard
| Challenge: | Multimodal sentiment analysis (MSA) has gained much attention over the last few years due to a lack of unimodal annotations in benchmark datasets. |
| Approach: | They propose a framework which integrates multimodal and unimodal tasks to optimize learning representations from multimodal data. |
| Outcome: | The proposed model learns to weight features differently based on features of other modalities and auto-generates unimodal annotations via a unimodule. |
Multimodal Multi-loss Fusion Network for Sentiment Analysis (2024.naacl-long)
Copied to clipboard
| Challenge: | This paper examines the optimal selection and fusion of feature encoders across multiple modalities and combines them in one neural network to improve sentiment detection. |
| Approach: | They propose to combine feature encoders across multiple modalities into one neural network to improve sentiment detection. |
| Outcome: | The proposed model achieves state-of-the-art performance for three datasets . it also shows that integrating context significantly improves model performance. |