| Challenge: | Existing evidence that visual context helps multimodal machine translation systems is unconvincing due to inconsistencies between text-similarity metrics and human judgements. |
| Approach: | They propose an adversarial evaluation method to examine the utility of image data in multimodal machine translation. |
| Outcome: | The proposed method shows that only one out of three publicly available systems is sensitive to this perturbation of the data. |
Similar Papers
Understanding the Effect of Textual Adversaries in Multimodal Machine Translation (D19-64)
Copied to clipboard
| Challenge: | Existing studies show that multimodal machine translation systems are better than text-only systems at translating phrases that have a direct correspondence in the image. |
| Approach: | They conduct experiments with both visual and textual adversaries to understand the role of textual inputs in multimodal machine translation. |
| Outcome: | The proposed model can recover masked tokens in the source sentences . the proposed model is based on a model with a visual modality . |
Multimodal Neural Machine Translation: A Survey of the State of the Art (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal neural machine translation (MNMT) is a task that aims to translate text into the target language using neural networks. |
| Approach: | They propose to integrate other modalities with textual data to enhance translation performance. |
| Outcome: | The proposed task aims to integrate visual modality with textual data to improve translation quality. |
Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work in multimodal machine translation (MT) has shown that ambiguity can be resolved using accompanying context such as images. |
| Approach: | They propose a multimodal machine translation approach based on a strong text-only MT model and a novel guided self-attention mechanism to train it. |
| Outcome: | The proposed model outperforms existing models on EnglishFrench, EnglishGerman and EnglishCzech benchmarks and is freely available. |
Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal machine translation models outperform text-only models when visual context is available, but recent studies have shown that the performance of MMT models is only marginally impacted when the associated image is replaced with an unrelated image or noise. |
| Approach: | They propose to use visual data to highlight the importance of visual inputs in MMT models to enhance their leverage. |
| Outcome: | The proposed models outperform text-only models when visual context is available, but the results show that the visual context might not be exploited by the models at all. |
A Visual Attention Grounding Neural Model for Multimodal Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to multimodal machine translation do not integrate visual information into the translation process. |
| Approach: | They propose a multimodal machine translation model that utilizes parallel visual and textual information. |
| Outcome: | The proposed model outperforms existing methods on the Multi30K and Ambiguous COCO datasets. |
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate multiple systems are expensive and require human evaluators. |
| Approach: | They propose a novel online learning approach that dynamically converges to the top-3 ranked systems for the language pairs considered by taking advantage of human feedback. |
| Outcome: | The proposed approach converges to the top-3 ranked systems for the language pairs considered despite the lack of human feedback for many translations. |
Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View (2023.acl-long)
Copied to clipboard
| Challenge: | Special attention is paid to the cross-modal misalignment in text-image datasets which may mislead the learning. |
| Approach: | They propose a multimodal back-translation method which uses diffusion-based generative models for pseudo-paralleled pairs and a divergence estimator to construct a high-resource corpora as a bridge for low-ressource learners. |
| Outcome: | The proposed method outperforms 14 state-of-the-art methods in both entity and relation extraction tasks. |
Learning Translations via Images with a Massively Multilingual Image Dataset (P18-1)
Copied to clipboard
| Challenge: | Existing datasets for learning translations of words are limited to a few high-resource languages and unrealistically easy settings. |
| Approach: | They propose a large-scale multilingual corpus of images labeled with the word they represent to facilitate translation research. |
| Outcome: | The proposed method improves on an unsupervised technique that has been limited to a few languages and unrealistic settings. |
Probing Image-Language Transformers for Verb Understanding (2021.findings-acl)
Copied to clipboard
| Challenge: | Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning. |
| Approach: | They collect a dataset of image-sentence pairs consisting of 421 verbs . they evaluate pretrained image-language transformers and find they fail more in situations that require verb understanding compared to other parts of speech. |
| Outcome: | The proposed model trains on a manually-annotated and smaller dataset does better on the task. |
Assessing Multilingual Fairness in Pre-trained Multimodal Representations (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent pre-trained multimodal models have shown exceptional capabilities towards connecting images and natural language. |
| Approach: | They propose two new fairness notions for pre-trained multimodal models that consider language as the fairness recipient. |
| Outcome: | The proposed models can be generalized to multilingualism by cross-lingual alignment . the results show that the models are individually fair across languages . |