Papers by Sijie Mai
Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to multimodal affective computing learn spurious correlations from training data rather than genuine causal relationships, harming generalization under distribution shifts or noisy modalities. |
| Approach: | They propose a causal modality-invariant representation framework that separates each modality into ‘causal invariant’ and ‘environment-specific spurious representation’ from a modal inference perspective. |
| Outcome: | Experiments on multiple multimodal benchmarks show that the proposed framework achieves state-of-the-art performance. |
Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies focus on learning cross-modal dynamics, but neglect to explore optimal solution for unimodal networks. |
| Approach: | They propose a new MSA framework to identify contribution of modalities and reduce impact of noisy information. |
| Outcome: | The proposed model outperforms state-of-the-art methods on publicly available datasets. |
Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have used the correlation information stored in samples for self-supervised learning, but they feed the training pairs in a random order without consideration of difficulty. |
| Approach: | They propose to inject curriculum learning into weakly supervised multimodal correlation learning by scoring and feeding pairs according to difficulty. |
| Outcome: | The proposed model achieves state-of-the-art on multimodal sentiment analysis without human annotation. |
Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to multimodal fusion are based on fusing features at holistic level instead of focusing on local and global interactions. |
| Approach: | They propose a general strategy called ‘divide, conquer and combine’ for multimodal fusion that combines local and global interactions in a hierarchy. |
| Outcome: | The proposed strategy achieves state-of-the-art performance on multimodal affective computing with higher efficiency. |
Supervised Attention Mechanism for Low-quality Multimodal Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current studies address missing and noisy modalities separately in multimodal data . missing modality is often caused by unavailable data collection equipment or sensor failures . |
| Approach: | They propose a framework for multimodal affective computing that addresses missing and noisy modalities to enhance model robustness in low-quality data scenarios. |
| Outcome: | The proposed model outperforms state-of-the-art baselines on multiple datasets under the settings of complete modalities, missing modalités, and noisy modality. |