| Challenge: | Existing work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient. |
| Approach: | They propose a translate-and-refine approach to multimodal machine translation where images are only used by a second stage decoder to generate a good first draft translation and to improve over this draft. |
| Outcome: | The proposed approach generates a good translation and improves over the draft by making better use of the target language textual context and making use of visual context. |
Similar Papers
Visual Cues and Error Correction for Translation Robustness (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing robustness techniques fail when faced with unseen types of noise and their performance degrades on clean texts. |
| Approach: | They propose visual context to improve translation robustness for noisy texts . they also propose an error correction training regime that can be used as an auxiliary task . |
| Outcome: | The proposed training regime improves translation robustness on noisy texts while maintaining translation quality on clean texts. |
Generative Imagination Elevates Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing multimodal neural machine translation methods require triplets of bilingual sentence - image for training and tuples of source sentence . Existing methods require truncated images for inference, but ImagiT uses both source sentence and “imagined representation” to produce a target translation. |
| Approach: | They propose a multimodal machine translation method using visual imagination to generate a target translation from a sentence in a source language. |
| Outcome: | The proposed method significantly outperforms the existing text-only neural machine translation baselines and improves translation quality. |
Translating Translationese: A Two-Step Approach to Unsupervised Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Using a dictionary, given a rough, target language natives can uncover the latent, fully-fluent rendering of the translation. |
| Approach: | They propose a method that breaks translation into two steps by generating a dictionary and then ‘translating’ the resulting pseudo-translation into a fully fluent translation. |
| Outcome: | The proposed method 'gets better translation results on high-resource languages than previously published unsupervised MT studies' |
Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View (2023.acl-long)
Copied to clipboard
| Challenge: | Special attention is paid to the cross-modal misalignment in text-image datasets which may mislead the learning. |
| Approach: | They propose a multimodal back-translation method which uses diffusion-based generative models for pseudo-paralleled pairs and a divergence estimator to construct a high-resource corpora as a bridge for low-ressource learners. |
| Outcome: | The proposed method outperforms 14 state-of-the-art methods in both entity and relation extraction tasks. |
Paying More Attention to Source Context: Mitigating Unfaithful Translations from Large Language Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models lack explicit alignment between source and target contexts, leading to unfaithful translations. |
| Approach: | They propose three learning strategies to encourage LLMs to pay more attention to source context . they use a dataset to test the effectiveness of their model across multiple language pairs . |
| Outcome: | The proposed model reduces hallucinatory translation and improves fidelity across multiple languages. |
Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work in multimodal machine translation (MT) has shown that ambiguity can be resolved using accompanying context such as images. |
| Approach: | They propose a multimodal machine translation approach based on a strong text-only MT model and a novel guided self-attention mechanism to train it. |
| Outcome: | The proposed model outperforms existing models on EnglishFrench, EnglishGerman and EnglishCzech benchmarks and is freely available. |
Probing the Need for Visual Context in Multimodal Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Current work on multimodal machine translation (MMT) suggests that the visual modality is either unnecessary or only marginally beneficial. |
| Approach: | They propose to use the visual modality to combine visual and textual information to generate better translations by partially depriving models from source-side textual context. |
| Outcome: | The proposed model can combine visual and textual information to generate better translations under limited textual context. |
A Visual Attention Grounding Neural Model for Multimodal Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to multimodal machine translation do not integrate visual information into the translation process. |
| Approach: | They propose a multimodal machine translation model that utilizes parallel visual and textual information. |
| Outcome: | The proposed model outperforms existing methods on the Multi30K and Ambiguous COCO datasets. |
Multimodal Lexical Translation (L18-1)
Copied to clipboard
| Challenge: | Multimodal Lexical Translation (MLT) is a task that aims to translate ambiguous words given their context -an image and a sentence in the source language. |
| Approach: | They introduce a task to translate an ambiguous word given its context -an image and a sentence in the source language. |
| Outcome: | The proposed task is based on the Multi30K dataset and uses word-alignment followed by human inspection to select subsets of the dataset which are difficult to translate. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |