| Challenge: | Feed-forward networks are widely used in cross-modal applications to bridge modalities . success of such systems depends entirely on ability of mapping to make neighborhood structure akin to that of the target vectors. |
| Approach: | They propose to use a similarity measure to measure the neighborhood structure of neural network mappings. |
| Outcome: | The proposed model shows that the predicted neighborhood structure resembles more that of the input vectors than that of target vectors. |
Similar Papers
Low-resource Neural Machine Translation with Cross-modal Alignment (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing neural machine translation techniques rely on large monolingual corpus, which is costly for some low-resource languages. |
| Approach: | They propose a cross-modal contrastive learning method to learn a shared space for all languages by additional visual modality. |
| Outcome: | The proposed method can learn cross-modal and cross-lingual alignment with small amount of image-text pairs and achieves significant improvements over the text-only baseline. |
Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to identify key neurons for interpretability of multi-modal large language models are unclear. |
| Approach: | They propose a method to identify key neurons for interpretability by multi-modal large language models. |
| Outcome: | The proposed method improves conventional works upon efficiency and applied range by removing needs of costly gradient computation. |
Are Any-to-Any Models More Consistent Across Modality Transfers Than Specialists? (2025.acl-long)
Copied to clipboard
| Challenge: | equivariance evaluations uncover weak but observable consistency through structured analyses of the intermediate latent space enabled by multiple editing operations. |
| Approach: | They use a dataset of 1,000 images paired with captions, editing instructions, and Q&A pairs to evaluate cross-modal transfers rigorously. |
| Outcome: | The proposed models do not consistently demonstrate greater cross-modal consistency than specialized models in pointwise evaluations such as cyclic consistency. |
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to model multimodal data do not leverage cross-modal information . augmenting input text using cross-module attribute insertions results in poor performance . |
| Approach: | They propose a multimodal deep learning approach that adds visual attributes to inputs to enhance model robustness. |
| Outcome: | The proposed approach is modular, controllable, and task-agnostic. |
Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows that vision-language models have modality gaps that persist even in well-aligned models. |
| Approach: | They propose a modality-dominance score to measure and leverage modality gaps . they propose automatic interpretability metrics to evaluate these features in a scalable manner . |
| Outcome: | The proposed framework allows for training-free probing and editing methods for understanding model perception across genders and generating adversarial examples. |
How to represent a word and predict it, too: Improving tied architectures for language modelling (D18-1)
Copied to clipboard
| Challenge: | Recent state-of-the-art models use word embeddings as input and output mappings instead of tied models. |
| Approach: | They propose to decouple hidden state from word embedding prediction . they extend their proposed modification to word2vec models . |
| Outcome: | The proposed architectures achieve comparable or better results compared to previous models without tying . the proposed architecture reduces parameters, enabling more compact models and faster learning. |
Analyzing the Limitations of Cross-lingual Word Embedding Mappings (P19-1)
Copied to clipboard
| Challenge: | Existing methods for cross-lingual word embeddings have limited results . existing methods require little or no cross-linguistic signal to work . |
| Approach: | They compare offline mapping methods to an extension of skip-gram that jointly learns both embedding spaces. |
| Outcome: | The proposed method yields more isomorphic embeddings, is less sensitive to hubness, and achieves stronger results in bilingual lexicon induction. |
Cross-Modal Taxonomic Generalization in (Vision-) Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that language models learn from surface form to learn from more grounded evidence. |
| Approach: | They propose to use a vision-language model to learn hypernyms from images . they find that the model can recover this knowledge and generalize even when there is no hypernomia in the image. |
| Outcome: | The proposed model can recover this knowledge and generalize even when the model receives no evidence of hypernyms during training. |
Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that deep vision-only and language-only models project inputs into a partially aligned representational space. |
| Approach: | They investigate whether a model's representational code is semantically shared . they find that alignment peaks in mid-to-late layers of both model types . |
| Outcome: | a forced-choice "Pick-a-Pic" task shows human preferences for image-caption matches are mirrored in embedding spaces across vision-language model pairs. |
Cross-Modal Discrete Representation Learning (2022.acl-long)
Copied to clipboard
| Challenge: | a new framework for learning representations from multimodal data is proposed . the proposed framework uses discretized embedding vectors to capture finer levels of granularity . |
| Approach: | They propose a self-supervised representation learning framework that captures finer levels of granularity across different modalities. |
| Outcome: | The proposed representation can capture finer levels of granularity across different modalities . it can be used on cross-modal retrieval tasks without direct supervision . |