Dynamic Regularization in UDA for Transformers in Multimodal Classification (2023.acl-long)
Copied to clipboard
| Challenge: | Multimodal machine learning is a cutting-edge field that explores ways to combine information from multiple sources into models. |
| Approach: | They propose a multimodal BERT-ViT model that exploits weaker modality while regularizing the loss function. |
| Outcome: | The proposed model exploits weaker modality while regularizing the loss function. |
Similar Papers
Two Challenges, One Solution: Robust Multimodal Learning through Dynamic Modality Recognition and Enhancement (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods require full-modality data during training phase or require explicit annotations to detect missing modalities. |
| Approach: | They propose a Dynamic modality Recognition and Enhancement for Adaptive Multimodal fusion framework that directs selective reconstruction of missing or underperforming modalities. |
| Outcome: | The proposed framework outperforms several baseline and state-of-the-art models on three benchmark datasets. |
Tutorial on Multimodal Machine Learning (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Multimodal machine learning is a challenging but crucial area with numerous applications in multimedia, affective computing, robotics, finance, HCI, and healthcare. |
| Approach: | This tutorial will describe an updated taxonomy on multimodal machine learning synthesizing its core technical challenges and major directions for future research. |
| Outcome: | The proposed taxonomy synthesizes the core technical challenges and major directions for future research. |
Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Sentiment Analysis (MSA) is effective when using rich information from multiple sources, but the potential sentiment-irrelevant information across modalities may hinder the performance from being further improved. |
| Approach: | They propose an Adaptive Language-guided Multimodal Transformer (ALMT) that learns an irrelevance/conflict-suppressing representation from visual and audio features under guidance of language features at different scales. |
| Outcome: | The proposed model achieves state-of-the-art on several popular datasets and an abundance of ablation shows the effectiveness of the proposed model. |
Adaptive Fusion Techniques for Multimodal Data (2021.eacl-main)
Copied to clipboard
| Challenge: | Effective fusion of data from multiple modalities is challenging due to the heterogeneous nature of multimodal data. |
| Approach: | They propose two adaptive fusion techniques that aim to combine multimodal data effectively. |
| Outcome: | The proposed networks can model context from other modalities better than existing methods. |
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks (2026.findings-acl)
Copied to clipboard
| Challenge: | Survey aims to identify challenges of multimodal unlearning for vision, language, audio and video . retraining after deletion requests or policy updates is often impractical, survey finds . |
| Approach: | They propose to enable selective removal across modalities while retaining overall utility. |
| Outcome: | This study compares models with existing models to identify weaknesses and improves performance. |
UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing multimodal fusion methods ignore inter-modality relationship, treat each modality equally, suffer sensor noise, and thus reduce multimodal learning performance. |
| Approach: | They propose a multimodal contrastive method to explore more reliable multimodal representations under the weak supervision of unimodal predicting. |
| Outcome: | The proposed method outperforms current state-of-the-art multimodal learning methods on image-text classification benchmarks UPMC-Food-101 and N24News. |
Improving Multimodal Sentiment Analysis: Supervised Angular margin-based Contrastive Learning for Enhanced Fusion Representation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis focus on forming positive and negative pairs, neglecting the variation in sentiment scores within the same class. |
| Approach: | They propose a framework to enhance discrimination and generalizability of the multimodal representation and overcome biases in the fusion vector’s modality. |
| Outcome: | The proposed model improves discrimination and generalizability of the multimodal representation and overcomes biases in the fusion vector’s modality. |
Learning Flexible Large Multimodal Models with Arbitrary Modality Combinations (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) have potential for cross-modal understanding . but extending MLLM to handle diverse modalities introduces two challenges . |
| Approach: | They propose a dual-stage compression mechanism to reduce the number of modality tokens per modality and condense it into a single, compact token sequence. |
| Outcome: | Experiments show that Flex-M3 outperforms its counterpart trained on only full-modality data. |
Adaptive Transformers for Learning Multimodal Representations (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing approaches for learning visiolinguistic representations with transformers are over-parametrized and require extensive training. |
| Approach: | They propose to extend attention spans, sparse, and structured dropout methods to learn more about how the network perceives the complexity of input sequences. |
| Outcome: | The proposed approaches improve on language semantics and visiolinguistic representations, but are often over-parametrized and require large amounts of computation. |
On Vision Features in Multimodal Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is given to the quality of vision models. |
| Approach: | They develop a selective attention model to study the patch-level contribution of an image in multimodal machine translation. |
| Outcome: | The proposed model is able to learn translation from the visual modality on probing tasks and is compared with existing models. |