| Challenge: | Pre-trained language models (PLMs) have been used for text sentiment analysis but sentiment is hidden in other modalities. |
| Approach: | They propose to fuse emotions from different data to analyze sentiments . they use compression parameter for each expert to reduce training burden . |
| Outcome: | The proposed method achieves state-of-the-art with a tiny trainable parameter count compared to current methods . emotions hidden in body movements or vocal timbres eclipse traditional methods compared with text sentiment analysis . |
Similar Papers
CLGSI: A Multimodal Sentiment Analysis Framework based on Contrastive Learning Guided by Sentiment Intensity (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have focused on contrastive learning, but lack detailed learning of the distribution of sample pairs with different sentiment intensity differences in the contrastive training representation space. |
| Approach: | They propose a framework for multimodal sentiment analysis based on contrastive learning guided by sentiment intensity (CLGSI) it selects positive and negative sample pairs based upon sentiment intensity differences and assigns corresponding weights accordingly. |
| Outcome: | The proposed framework extracts common features between different modalities and then uses them to predict sentiment intensity. |
Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Sentiment Analysis (MSA) is effective when using rich information from multiple sources, but the potential sentiment-irrelevant information across modalities may hinder the performance from being further improved. |
| Approach: | They propose an Adaptive Language-guided Multimodal Transformer (ALMT) that learns an irrelevance/conflict-suppressing representation from visual and audio features under guidance of language features at different scales. |
| Outcome: | The proposed model achieves state-of-the-art on several popular datasets and an abundance of ablation shows the effectiveness of the proposed model. |
Self-supervised Cross-modal Pretraining for Speech Emotion Recognition and Sentiment Analysis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multimodal speech emotion recognition and sentiment analysis have not improved results due to their relatively simple fusion mechanisms and lack of proper cross-modal pretraining. |
| Approach: | They propose a deep-fused audio-text bi-modal transformer with carefully designed cross-modal fusion mechanism and stage-wise cross-mod pretraining scheme to facilitate cross-modulation. |
| Outcome: | The proposed method exceeds benchmarks on public IEMOCAP emotion and CMU-MOSEI sentiment datasets by a large margin. |
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two. |
| Approach: | They propose a multimodal sentiment knowledge-sharing framework that unifies MSA and ERC tasks from features, labels, and models. |
| Outcome: | The proposed framework achieves consistent improvements on four public benchmark datasets on MOSI, MOSEI, MELD, and IEMOCAP. |
Towards Exploiting Sticker for Multimodal Sentiment Analysis in Social Media: A New Dataset and Baseline (2022.coling-1)
Copied to clipboard
| Challenge: | Sentiment analysis in social media is challenging because of the lack of context. |
| Approach: | They propose to use stickers to perform a multimodal sentiment analysis task using Chinese stickers. |
| Outcome: | The proposed model performs best compared with other models. |
Sentiment Knowledge Enhanced Self-supervised Learning for Multimodal Sentiment Analysis (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have used general approaches to alleviate the overfitting of supervised models based on video data with sentiment annotations. |
| Approach: | They propose to capture common sentimental patterns in unlabeled videos using sentiment knowledge and non-verbal behavior to embed sentiment information into pre-trained multimodal representations. |
| Outcome: | The proposed model outperforms the baseline and achieves new State-Of-The-Art (SOTA) results. |
MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal models focus on the correspondence between images and text, but this only covers a subset of real-world interactions. |
| Approach: | They propose an approach to enhance multimodal models by training separate expert models for each type of interaction, such as redundancy present in both modalities, uniqueness in one modality, or synergy that emerges when both . modality is used to capture overlaps in semantic content between images and text, making a strong multi-view redundancies assumption. |
| Outcome: | The proposed approach improves on a sarcasm detection and humor detection task. |
Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to multimodal Aspect-Based Sentiment Analysis (MABSA) ignore crossmodalalignment and use pre-trained visual and textual models. |
| Approach: | They propose a multimodal multimodal encoder-decoder framework for MABSA that uses a unified multimodal decoder architecture for all the pretrainingand downstream tasks. |
| Outcome: | The proposed framework outperforms state-of-the-art approaches on three MABSA subtasks. |
M-SENA: An Integrated Platform for Multimodal Sentiment Analysis (2022.acl-demo)
Copied to clipboard
| Challenge: | M-SENA is an open-source platform for multimodal sentiment analysis. |
| Approach: | They propose to use a platform for multimodal sentiment analysis to facilitate advanced research by providing flexible toolkits, reliable benchmarks, and intuitive demonstrations. |
| Outcome: | The proposed framework provides reliable benchmarks and baseline results of different modality features and MSA benchmarks. |
Multimodal Multi-loss Fusion Network for Sentiment Analysis (2024.naacl-long)
Copied to clipboard
| Challenge: | This paper examines the optimal selection and fusion of feature encoders across multiple modalities and combines them in one neural network to improve sentiment detection. |
| Approach: | They propose to combine feature encoders across multiple modalities into one neural network to improve sentiment detection. |
| Outcome: | The proposed model achieves state-of-the-art performance for three datasets . it also shows that integrating context significantly improves model performance. |