Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition (2020.aacl-main)
Copied to clipboard
| Challenge: | despite recent advances in multimodal emotion recognition, two problems still exist: sub-optimal performance and low-resource emotions. |
| Approach: | They propose a modality-transferable model with emotion embeddings to solve these problems . they use pre-trained word embedders to represent emotion categories for textual data . |
| Outcome: | The proposed model outperforms baselines in zero-shot and few-shot scenarios for unseen emotions. |
Similar Papers
Integrating Representation Subspace Mapping with Unimodal Auxiliary Loss for Attention-based Multimodal Emotion Recognition (2024.lrec-main)
Copied to clipboard
Xulong Du, Xingnan Zhang, Dandan Wang, Yingying Xu, Zhiyuan Wu, Shiqing Zhang, Xiaoming Zhao, Jun Yu, Liangliang Lou
| Challenge: | Existing methods to identify emotions rely on a large modality gap in their representations . |
| Approach: | They propose a representation subspace mapping module that maps each modality into two distinct subspaces and a cross-modality attention module that leverages auxiliary loss to remove the noise unrelated to emotion classification. |
| Outcome: | The proposed approach achieves superior performance to state-of-the-art MER methods on the IEMOCAP and MSP-Improv datasets. |
Layer-wise Fusion with Modality Independence Modeling for Multi-modal Emotion Recognition (2023.acl-long)
Copied to clipboard
Jun Sun, Shoukang Han, Yu-Ping Ruan, Xiaoning Zhang, Shu-Kai Zheng, Yulong Liu, Yuxin Huang, Taihao Li
| Challenge: | Existing studies focus on developing models that exploit the unification of multiple modalities. |
| Approach: | They propose to maintain modality independence by using a multi-modal transformer model that fuses all modalities. |
| Outcome: | The proposed model outperforms state-of-the-art models in multi-modal emotion recognition. |
Multimodal and Multi-view Models for Emotion Recognition (P19-1)
Copied to clipboard
| Challenge: | combining lexical and acoustic information results in more robust and accurate models . combining both modalities may be a bottleneck in a deployment pipeline due to computational complexity or privacy constraints . |
| Approach: | They propose to combine acoustic and lexical information to provide a deployable acustic model . they use multimodal models and two attention mechanisms to assess the benefits of lexicals . |
| Outcome: | The proposed model outperforms the state-of-the-art on the USC-IEMOCAP dataset . it significantly surpasses models that have been exclusively trained with acoustic features . |
Self-supervised Cross-modal Pretraining for Speech Emotion Recognition and Sentiment Analysis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multimodal speech emotion recognition and sentiment analysis have not improved results due to their relatively simple fusion mechanisms and lack of proper cross-modal pretraining. |
| Approach: | They propose a deep-fused audio-text bi-modal transformer with carefully designed cross-modal fusion mechanism and stage-wise cross-mod pretraining scheme to facilitate cross-modulation. |
| Outcome: | The proposed method exceeds benchmarks on public IEMOCAP emotion and CMU-MOSEI sentiment datasets by a large margin. |
Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to predict emotion label for a given utterance lack modeling of diverse dependency ranges and inconsistent treatment of contribution for various modalities. |
| Approach: | They propose a multimodal emotion recognition in conversation task that uses context and multiple modalities to predict emotion label for a given utterance. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on three multimodal datasets. |
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors (P18-1)
Copied to clipboard
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, Louis-Philippe Morency
| Challenge: | Multimodal research is a growing field of artificial intelligence, and fusion is one of the main research problems. |
| Approach: | They propose a low-rank multimodal fusion method which integrates multiple unimodal representations into one compact multimodal representation. |
| Outcome: | The proposed method achieves competitive results on multimodal sentiment analysis, speaker trait analysis, and emotion recognition tasks while reducing computational complexity. |
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two. |
| Approach: | They propose a multimodal sentiment knowledge-sharing framework that unifies MSA and ERC tasks from features, labels, and models. |
| Outcome: | The proposed framework achieves consistent improvements on four public benchmark datasets on MOSI, MOSEI, MELD, and IEMOCAP. |
Amanda: Adaptively Modality-Balanced Domain Adaptation for Multimodal Emotion Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Emotion recognition is a multimodal learning method that can be used for data scarcity. |
| Approach: | They propose to use Adaptively modality-balanced domain adaptation to balance the alignment of different modalities for multimodal emotion recognition. |
| Outcome: | The proposed model outperforms competing models on common datasets on multimodal emotion recognition. |
Multimodal End-to-End Sparse Model for Emotion Recognition (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work in emotion recognition uses a two-phase pipeline, but the extracted features are fixed and cannot be fine-tuned on different tasks. |
| Approach: | They propose a two-phase pipeline for emotion recognition and personality recognition . they propose restructured datasets to enable fully end-to-end training . |
| Outcome: | The proposed model outperforms the current state-of-the-art models on emotion recognition and personality recognition tasks with half less computation in the feature extraction part. |
Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on multimodal representation learning has focused on uni-modality pre-training or cross-modalities integration. |
| Approach: | They propose a framework for multimodal representation learning that uses uni-modal contrastive coding and an efficient unimodal feature augmentation strategy to capture intermodal dynamics. |
| Outcome: | The proposed framework surpasses state-of-the-art methods on two public datasets. |