Investigating Audio, Video, and Text Fusion Methods for End-to-End Automatic Personality Prediction (P18-2)
Copied to clipboard
| Challenge: | Using stacked Convolutional Neural Networks, we can predict personality traits from video clips with different channels for audio, text, and video data. |
| Approach: | They propose a tri-modal architecture to predict Big Five personality trait scores from video clips with different channels for audio, text, and video data. |
| Outcome: | The proposed model outperforms the best individual modality with 9.4% accuracy over the best channel. |
Similar Papers
Adaptive Fusion Techniques for Multimodal Data (2021.eacl-main)
Copied to clipboard
| Challenge: | Effective fusion of data from multiple modalities is challenging due to the heterogeneous nature of multimodal data. |
| Approach: | They propose two adaptive fusion techniques that aim to combine multimodal data effectively. |
| Outcome: | The proposed networks can model context from other modalities better than existing methods. |
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors (P18-1)
Copied to clipboard
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, Louis-Philippe Morency
| Challenge: | Multimodal research is a growing field of artificial intelligence, and fusion is one of the main research problems. |
| Approach: | They propose a low-rank multimodal fusion method which integrates multiple unimodal representations into one compact multimodal representation. |
| Outcome: | The proposed method achieves competitive results on multimodal sentiment analysis, speaker trait analysis, and emotion recognition tasks while reducing computational complexity. |
Late Fusion with Triplet Margin Objective for Multimodal Ideology Prediction and Analysis (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on ideology prediction has focused on single modalities, i.e., text or images. |
| Approach: | They propose a task where a model predicts binary or five-point scale ideological leanings given a text-image pair with political content. |
| Outcome: | The proposed model outperforms the state-of-the-art model by almost 4% and a strong multimodal baseline with no pretraining by over 3%. |
Personality Vector: Modulating Personality of Large Language Models by Model Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to induce personality in large language models (LLMs) fail to capture the continuous nature of human traits. |
| Approach: | They propose a method for personality modulation in large language models by model merging by subtracting weights of pre-trained models from those of fine-tuned models. |
| Outcome: | The proposed method allows LLMs to exhibit desired personality traits without additional training. |
Self-supervised Cross-modal Pretraining for Speech Emotion Recognition and Sentiment Analysis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multimodal speech emotion recognition and sentiment analysis have not improved results due to their relatively simple fusion mechanisms and lack of proper cross-modal pretraining. |
| Approach: | They propose a deep-fused audio-text bi-modal transformer with carefully designed cross-modal fusion mechanism and stage-wise cross-mod pretraining scheme to facilitate cross-modulation. |
| Outcome: | The proposed method exceeds benchmarks on public IEMOCAP emotion and CMU-MOSEI sentiment datasets by a large margin. |
Open Your Model’s Eyes: Video and Context-Aware Multimodal Backchannel Prediction (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for predicting backchannels rely on audio and text . existing methods omit visual cues and conversational contexts for accurate prediction . |
| Approach: | They propose a framework that leverages visual cues and conversational contexts to enhance backchannel prediction. |
| Outcome: | The proposed framework outperforms existing methods and simple multimodal baselines in recognizing complex backchannels such as empathy. |
Harmonizing Code-mixed Conversations: Personality-assisted Code-mixed Response Generation in Dialogues (2024.findings-eacl)
Copied to clipboard
| Challenge: | blending multiple languages within a single conversation presents a formidable challenge, given the wide-ranging variations influenced by individual speaking styles and cultural backgrounds. |
| Approach: | They propose a novel approach to harness the Big Five personality traits acquired in an unsupervised manner from code-mixed conversations to bolster the performance of response generation. |
| Outcome: | The proposed approach enhances contextual relevance and performance of the proposed model by combining personality traits with dialogue context. |
BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Data (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for embedding human personality traits into LLMs are limited by realism and validity issues. |
| Approach: | They propose to use a large-scale dataset to embed human personality traits into LLMs . they use supervised fine-tuning and direct preference optimization to train LLM models . |
| Outcome: | The proposed methods outperform prompting on personality assessments and IPIP-NEO, and show higher conscientiousness, agreeableness, lower extraversion, and lower neuroticism on reasoning tasks. |
Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning (D19-1)
Copied to clipboard
| Challenge: | Existing studies on video captioning focus on the association relationships between multiple modalities. |
| Approach: | They propose a video captioning model with high-order cross-modal attention (HOCA) they propose low-rank HOCA which adopts tensor decomposition to reduce the space requirement . |
| Outcome: | The proposed model captures cross-modal interaction of different modalities and reduces space requirement. |
Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning (N18-2)
Copied to clipboard
| Challenge: | Existing multi-modal fusion methods have shown encouraging results in video understanding, but how to selectively fuse the multi-dimensional representations at different levels of details remains unexplored. |
| Approach: | They propose a hierarchically aligned cross-modal attention framework to fuse audio and visual cues at different levels of detail. |
| Outcome: | The proposed framework outperforms the previous best systems on the video captioning task. |