Efficient Prompting for Continual Adaptation to Missing Modalities (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods combine various missing cases to train recovery modules or align multimodal features, resulting in suboptimal performance, high computational costs, and catastrophic forgetting. |
| Approach: | They propose a continual multimodal missing modality task that uses prompts to learn modalities . existing methods often aggregate various missing cases to train recovery modules . authors conduct extensive experiments on three public datasets . |
| Outcome: | The proposed method consistently outperforms state-of-the-art methods on three public datasets. |
Similar Papers
Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis often fail due to equipment failure, data corruption, privacy issues and the like. |
| Approach: | They propose a multimodal Transformer framework using prompt learning to address the issue of missing modalities. |
| Outcome: | The proposed framework outperforms existing methods significantly across evaluation metrics. |
ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing MCIT methods do not fully exploit the unique attribute of Large Multimodal Models and often gain performance at the expense of efficiency. |
| Approach: | They propose a multimodal continual instruction learning framework that exploits the ability of LMMs to learn mixed instruction datasets and prompts for each task. |
| Outcome: | The proposed framework achieves +14.26% performance gain on MCIT benchmarks with remarkable x1.42 inference speed free from growing computation. |
Two Challenges, One Solution: Robust Multimodal Learning through Dynamic Modality Recognition and Enhancement (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods require full-modality data during training phase or require explicit annotations to detect missing modalities. |
| Approach: | They propose a Dynamic modality Recognition and Enhancement for Adaptive Multimodal fusion framework that directs selective reconstruction of missing or underperforming modalities. |
| Outcome: | The proposed framework outperforms several baseline and state-of-the-art models on three benchmark datasets. |
Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive neural machine translation models suffer from multi-modality problem . despite their autoregressivity, most NMT models suffer with slow decoding speed . |
| Approach: | They propose a semi-autoregressive model which generates a translation as a sequence of segments while each segment is predicted token-by-token. |
| Outcome: | The proposed model can achieve 4 times speedup while maintaining comparable performance. |
MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to embed multimodal models face limitations such as suboptimal causal attention in VLMs and limited diversity in training objectives and data. |
| Approach: | They propose a framework for transforming pre-trained VLMs into bidirectional multimodal embedding models. |
| Outcome: | The proposed model improves performance across MMEB and ViDoRe-v2 benchmarks and exhibits strong scalability with both model size and training data on MMEF. |
Continual Prompt Tuning for Dialog State Tracking (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to train a model on a sequence of tasks are not efficient enough to mitigate catastrophic forgetting. |
| Approach: | They propose a parameter-efficient framework that prevents forgetting and enables knowledge transfer between tasks by learning and freezing a pre-trained model. |
| Outcome: | The proposed framework avoids forgetting and enables knowledge transfer between tasks. |
LRMM: Learning to Recommend with Missing Modalities (D18-1)
Copied to clipboard
| Challenge: | Existing methods for content-based recommendation with missing or corrupted modalities are lacking in learning multimodal models. |
| Approach: | They propose a multimodal multimodal autoencoder that learns multimodal representations for complementing and imputing missing modalities. |
| Outcome: | The proposed framework achieves state-of-the-art performance on rating prediction tasks and is more robust to previous methods in alleviating data-sparsity and the cold-start problem. |
Causal Representation Learning from Multimodal Clinical Records under Non-Random Modality Missingness (2025.emnlp-main)
Copied to clipboard
| Challenge: | Clinical notes are often missing from clinical notes, resulting in modality missing-not-at-random (MMNAR) . large language models fine-tuned or adapted to clinical tasks have shown promise in medical reasoning, outcome prediction, and decision support. |
| Approach: | They propose a framework that leverages observed data and informative missingness in multimodal clinical records. |
| Outcome: | The proposed framework leverages observed data and informative missingness in multimodal clinical records. |
Modular and Parameter-Efficient Multimodal Fusion with Prompting (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent research has made impressive progress in large-scale multimodal pre-training. |
| Approach: | They propose to use prompt vectors to align multimodal modalities by pretraining text inputs with prompts or embedding vectors. |
| Outcome: | The proposed method achieves comparable performance to several other multimodal fusion methods in low-resource settings. |
MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to multimodal learning assume a complete input modality setting, i.e., each modality is either complete or completely missing in both training and test sets. |
| Approach: | They propose an alignment dynamics learning module based on the theory of optimal transport for missing data imputation and a denoising training algorithm to enhance the quality of iputation and accuracy of model predictions. |
| Outcome: | The proposed method performs faster and more accurate inferences under different missing conditions and alleviates the overfitting issue. |