m3P: Towards Multimodal Multilingual Translation with Multimodal Prompt (2024.lrec-main)
Copied to clipboard
Jian Yang, Hongcheng Guo, Yuwei Yin, Jiaqi Bai, Bing Wang, Jiaheng Liu, Xinnian Liang, LinZheng Chai, Liqun Yang, Zhoujun Li
| Challenge: | Existing multimodal neural machine translation models focus on bilingual translation, but experimental results show that they outperform the text-only baselines and multilingual multimodal methods by a large margin. |
| Approach: | They propose a framework to leverage the multimodal prompt to guide the Multimodal Multilingual Neural Machine Translation (m3P) this framework aligns the representations of different languages with the same meaning and generates the conditional vision-language memory for translation. |
| Outcome: | The proposed framework outperforms previous text-only baselines and multilingual multimodal methods by a large margin. |
Similar Papers
LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances struggle to train a separate model for each language pair, which is costly and unaffordable when the number of languages increases in the real world. |
| Approach: | They propose to train different MMT models to support translations between different languages. |
| Outcome: | The proposed model is able to handle the above issues by providing a shared semantic space for multiple languages. |
Multilingual-To-Multimodal (M2M): Unlocking New Languages with Monolingual Text (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing multimodal models rely on machine translation, but performance drops for other languages due to limited multilingual multimodal resources. |
| Approach: | They propose a lightweight alignment method that learns only a few linear layers using English text alone to map multilingual text embeddings into multimodal space. |
| Outcome: | M2M achieves strong zero-shot transfer on XTD Text-to-Image retrieval in English and spanish . it learns only a few linear layers to map multilingual text embeddings into multimodal space . |
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies report improvements when equipping models with multimodal information, but it remains unclear whether such improvements actually come from the multimodal part. |
| Approach: | They propose to extend conventional text-only translation models with multimodal information by extending them with visual input. |
| Outcome: | The proposed models replicate similar gains as recently developed multimodal-integrated systems achieved, but learn to ignore multimodal information. |
Multilingual and Multimodal Topic Modelling with Pretrained Embeddings (2022.coling-1)
Copied to clipboard
| Challenge: | a novel neural topic model for comparable data maps texts from multiple languages and images into a shared topic space. |
| Approach: | They propose a novel multimodal multilingual neural topic model that maps texts from multiple languages and images into a shared topic space. |
| Outcome: | The proposed model outperforms a zero-shot topic model in predicting topic distributions for comparable multilingual data and performs as well on unaligned embeddings as it does on aligned embeds. |
Probing the Need for Visual Context in Multimodal Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Current work on multimodal machine translation (MMT) suggests that the visual modality is either unnecessary or only marginally beneficial. |
| Approach: | They propose to use the visual modality to combine visual and textual information to generate better translations by partially depriving models from source-side textual context. |
| Outcome: | The proposed model can combine visual and textual information to generate better translations under limited textual context. |
Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Multimodal machine translation (MMT) aims to improve translation quality by incorporating information from other modalities, such as vision. |
| Approach: | They propose a framework for multimodal machine translation that utilizes large-scale non-triple data and a multimodal translation dataset. |
| Outcome: | The proposed method can significantly improve translation performance with more non-triple data. |
Context-Informed Machine Translation of Manga using Multimodal Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Automated manga translation is a promising potential solution, but it is underdeveloped due to the need to incorporate visual elements into the translation process to resolve ambiguities. |
| Approach: | They propose a method that leverages the vision component of multimodal large language models to improve translation quality and evaluate the impact of translation unit size, context length, and propose 'token efficient' approach for manga translation. |
| Outcome: | The proposed method achieves state-of-the-art results for Japanese-English translation and sets a new standard for Japanese and Polish translation. |
Multimodal Neural Machine Translation: A Survey of the State of the Art (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal neural machine translation (MNMT) is a task that aims to translate text into the target language using neural networks. |
| Approach: | They propose to integrate other modalities with textual data to enhance translation performance. |
| Outcome: | The proposed task aims to integrate visual modality with textual data to improve translation quality. |
Multimodal Machine Translation with Text-Image In-depth Questioning (2025.findings-acl)
Copied to clipboard
| Challenge: | Multimodal machine translation (MMT) models focus on intermodal interactions, but focus on simple interactions between nouns and entities in image, overlooking global semantic alignment. |
| Approach: | They propose a Text-Image In-depth Questioning method to deepen interactions and optimize translations by utilizing visual data to capture global semantic alignment. |
| Outcome: | The proposed method achieves state-of-the-art results on five translation directions of Multi30K and AmbigCaps, with +2.35 BLEU on the challenging MSCOCO benchmark. |
Contrastive Learning for Many-to-many Multilingual Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing multilingual machine translation approaches focus on English-centric directions, while non-English directions lag behind. |
| Approach: | They propose a multilingual machine translation system with an emphasis on non-English directions. |
| Outcome: | The proposed model outperforms existing models on English-centric and non-English directions on multilingual translation benchmarks. |