DELAN: Dual-Level Alignment for Vision-and-Language Navigation by Cross-Modal Contrastive Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies focus on cross-modal attention at the fusion stage, but modality features generated by disparate uni-encoders reside in their own spaces, leading to a decline in the quality of cross-modulation and decision-making. |
| Approach: | They propose a framework to align navigation-related modalities before fusion by cross-modal contrastive learning. |
| Outcome: | The proposed framework integrates with the majority of existing models, resulting in improved navigation performance on various VLN benchmarks, including R2R, R4R, and CVDN. |
Similar Papers
Modality Alignment between Deep Representations for Effective Video-and-Language Learning (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing Video-and-Language models do not take into account the different characteristics of video and text representations. |
| Approach: | They propose a method that exploits Centered Kernel Alignment (CKA) to enhance cross-modality attention by combining multiple modalities. |
| Outcome: | The proposed method outperforms conventional multi-modal methods significantly on video QA tasks with +3.57% accuracy increment compared to the baseline in a popular benchmark dataset. |
Multi-modal Semantic Understanding with Contrastive Cross-modal Feature Alignment (2024.lrec-main)
Copied to clipboard
| Challenge: | Current work on multi-modal semantic understanding primarily exploits a dual-encoder structure to separate image and text, but fails to learn cross-modal feature alignment. |
| Approach: | They propose a CLIP-guided contrastive-learning-based architecture to perform multi-modal feature alignment by projecting features from different modalities into a unified deep space. |
| Outcome: | The proposed model outperforms baseline models on sarcasm detection and sentiment analysis tasks and is simple to implement without using task-specific external knowledge. |
Diagnosing Vision-and-Language Navigation: What Really Matters (2022.naacl-main)
Copied to clipboard
Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Wang, Qi Wu, Miguel Eckstein, William Yang Wang
| Challenge: | Existing models claim to be able to align object tokens with specific visual targets, but there are non-negligible gaps between the two. |
| Approach: | They conduct diagnostic experiments to examine how the agents perceive multimodal input by ablation diagnostics input data. |
| Outcome: | The results show that indoor and outdoor navigation agents refer to object and direction tokens when making decisions. |
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing vision language navigation tasks require soft attention over words to locate instructions . a new approach uses syntax information to ground instructions with visual information . |
| Approach: | They propose a vision language navigation agent that utilizes syntax information to enhance alignment between the instruction and the current visual scenes. |
| Outcome: | The proposed agent outperforms the baseline model that does not use syntax information on the Room-to-Room dataset, especially in the unseen environment. |
DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing multimodal sentence representation learning methods focus on aligning images and text at a coarse level, resulting in cross-modal misalignment bias and intra-modal semantic divergence. |
| Approach: | They propose a dual-level alignment learning framework for multimodal sentence representation learning that promotes cross-modal and intra-modal alignment. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on semantic textual similarity and transfer tasks on semantic similarity, ranking distillation and global intra-modal alignment learning. |
RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-training (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing V&L pre-training methods rely on strictly-aligned multilingual image-text pairs generated from English-centric datasets. |
| Approach: | They propose a regularized cross-lingual visual contrastive learning objective that constrains representation proximity of weakly-aligned multilingual image-text pairs. |
| Outcome: | The proposed model outperforms competing models with weak zero-shot capability on 5 multi-modal tasks across 6 languages. |
CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations (2022.findings-naacl)
Copied to clipboard
| Challenge: | Using multilingual instructions to learn a better cross-lingual representation is challenging for multilingual agents. |
| Approach: | They propose to use multilingual instructions to learn a shared cross-lingual language representation for the three languages in a Room-Across-Room dataset. |
| Outcome: | The proposed model improves on the room-Across-room and vision-and-dialogue navigation tasks by maximizing similarity between semantically aligned image pairs from different environments. |
CLASP: Cross-modal Alignment Using Pre-trained Unimodal Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in speech-text pretraining rely on parallel speech- text data . however, data accessibility is a challenge due to the limited data available. |
| Approach: | They propose a framework for jointly performing speech and text processing without parallel corpora during pre-training but only downstream. |
| Outcome: | The proposed framework extracts distinct representations for speech and text, aligning them effectively in a newly defined space using a multi-level contrastive learning mechanism. |
Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing word alignment models capture few interactions between input sentence pairs, which severely degrades the word alignment quality. |
| Approach: | They propose to model deep interactions between input and target sentences using a two-stage training framework to train the model. |
| Outcome: | The proposed model achieves the state-of-the-art (SOTA) performance on four out of five language pairs. |
VLN-Trans: Translator for the Vision and Language Navigation Agent (2023.acl-long)
Copied to clipboard
| Challenge: | We observe two kinds of instructions that make the grounding in the vision-and-language navigation task quite challenging. |
| Approach: | They propose to use a translator module to convert instructions into easy-to-follow sub-instruction representations at each step. |
| Outcome: | The proposed model is based on a Room2Room (R2R), Room4room (R4R), and Room2room Last (R1R-Last) datasets and achieves state-of-the-art results on multiple benchmarks. |