| Challenge: | Existing methods to learn multimodal multilingual embeddings for text and image retrieval tasks are limited to English. |
| Approach: | They propose a new approach to learn multimodal multilingual embeddings for matching images and captions in two languages by combing two existing objective functions and adapting alignment between existing languages. |
| Outcome: | The proposed model achieves state-of-the-art in retrieval and caption-caption tasks while adapting existing language alignments. |
Similar Papers
Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task (D19-64)
Copied to clipboard
| Challenge: | Existing methods to learn multimodal multilingual embeddings for text and image retrieval tasks are limited to English. |
| Approach: | They propose a new approach to learn multimodal multilingual embeddings for matching images and captions in two languages by combing two existing objective functions and adapting alignment between existing languages. |
| Outcome: | The proposed model achieves state-of-the-art in retrieval and caption-caption tasks while adapting existing language alignments. |
Cross-lingual Cross-modal Pretraining for Multimodal Retrieval (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent pretrained vision-language models have achieved impressive performance on cross-modal retrieval tasks in English. |
| Approach: | They propose a new approach to learn cross-lingual cross-modal representations for matching images and captions in multiple languages using an annotated corpus. |
| Outcome: | The proposed model achieves impressive performance on two multimodal multilingual image caption benchmarks: Multi30k with German captions and MSCOCO with Japanese captions. |
A Simple Approach to Learning Unsupervised Multilingual Embeddings (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on unsupervised cross-lingual embeddings in the bilingual setting has given the impetus to learning a shared embeddable space for several languages. |
| Approach: | They propose to solve two sub-problems together to learn a shared embedding space for several languages. |
| Outcome: | The proposed approach outperforms existing methods in bilingual lexicon induction, cross-lingual word similarity, multilingual document classification, and multilingual dependency parsing tasks. |
Cross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning (2025.acl-short)
Copied to clipboard
| Challenge: | Multilingual alignment of sentence representations has mostly required bitexts to bridge the gap between languages. |
| Approach: | They propose to use image captions to implicitly align text representations between languages to make them usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval. |
| Outcome: | The proposed approach is usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval. |
Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that account for perceptual differences in image captions are limited to use in English . culture-based tasks such as recognition, detection, and image retrieval are hindered by relying on English supervision. |
| Approach: | They propose and evaluate caption augmentation strategies to address these gaps . they use captions from german perception and captions that have been machine-translated or human-transcribed from English into german . |
| Outcome: | The proposed models achieve a mean recall improvement of +1.3, but still lack flexibility . cultural differences present in language with respect to object specificity and importance . |
Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing work uses sentences within the same batch as negatives, which suffers from easy negatives. |
| Approach: | They propose to align sentence representations from different languages into a unified embedding space . they adapt MoCo to further improve the quality of alignment . |
| Outcome: | The proposed model achieves state-of-the-art on several tasks. |
Multilingual-To-Multimodal (M2M): Unlocking New Languages with Monolingual Text (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing multimodal models rely on machine translation, but performance drops for other languages due to limited multilingual multimodal resources. |
| Approach: | They propose a lightweight alignment method that learns only a few linear layers using English text alone to map multilingual text embeddings into multimodal space. |
| Outcome: | M2M achieves strong zero-shot transfer on XTD Text-to-Image retrieval in English and spanish . it learns only a few linear layers to map multilingual text embeddings into multimodal space . |
A Multi-task Approach to Learning Multilingual Representations (P18-2)
Copied to clipboard
| Challenge: | Using a multi-task model, we learn word and sentence embeddings in a single task. |
| Approach: | They propose a multi-task modeling approach that trains a skip-gram model and a cross-lingual sentence similarity model to learn word and sentence embeddings together. |
| Outcome: | The proposed model can learn word and sentence embeddings in a multilingual distributed representations of text using a cross-lingual sentence similarity model. |
Cross-lingual Sentence Embedding using Multi-Task Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual sentence embedding models require large parallel corpora to learn efficiently, limiting their scope. |
| Approach: | They propose a sentence embedding framework based on an unsupervised loss function . they capture semantic similarity and relatedness between sentences using a multi-task loss function. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on STS, BUCC and Tatoeba benchmarks and on a monolingual benchmark. |
Improving Multi-lingual Alignment Through Soft Contrastive Learning (2024.naacl-srw)
Copied to clipboard
| Challenge: | Existing methods to train multi-lingual sentence embeddings ruins the mono-lingual space. |
| Approach: | They propose a method to align multi-lingual embeddings based on similarity of sentences measured by a pre-trained mono-lingual teacher model. |
| Outcome: | The proposed method outperforms existing multi-lingual embeddings including LaBSE on five languages and on a translation pair for Tatoeba dataset. |