UMTIT: Unifying Recognition, Translation, and Generation for Multimodal Text Image Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Current Image machine translation (IMT) relies on a cascaded system that combines Optical Character Recognition (OCR) and a complex process of rendering the translated text back onto the source image. |
| Approach: | They propose a multimodal image-text translation model that generates consistent target images . they use two image-to-text conversion steps to convert images to text to recognize source text . |
| Outcome: | The proposed model outperforms existing methods and surpasses state-of-the-art methods in text recognition tasks. |
Similar Papers
Translation-Enhanced Multilingual Text-to-Image Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing models for text-to-image generation are mostly based on the English language due to the lack of annotated image-caption data in other languages. |
| Approach: | They propose to use a multilingual multi-modal encoder to bootstrap mTTI systems that can be translated into other languages. |
| Outcome: | The proposed approach mitigates the language gap and improves on standard mTTI datasets. |
On Vision Features in Multimodal Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is given to the quality of vision models. |
| Approach: | They develop a selective attention model to study the patch-level contribution of an image in multimodal machine translation. |
| Outcome: | The proposed model is able to learn translation from the visual modality on probing tasks and is compared with existing models. |
Multimodal Transformer for Multimodal Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods to incorporate information from other modality, usually static images, are not considered relative to multimodal machine translation. |
| Approach: | They propose a multimodal self-attention method which learns the representation of images based on the text, which avoids encoding irrelevant information in images. |
| Outcome: | The proposed model outperforms previous studies and competitive baselines in terms of various metrics. |
Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models require associated image with input sentence, which is difficult to satisfy at inference. |
| Approach: | They propose to use synthetic and authentic images to generate translations using text-to-image generation models. |
| Outcome: | The proposed model achieves state-of-the-art performance on En-De and En-Fr datasets while remaining independent of authentic images during inference. |
In-Image Machine Translation. A Preliminary Modular Approach (2026.eacl-srw)
Copied to clipboard
| Challenge: | In-image machine translation is a sub-task of Image-Based Machine Translation that aims to substitute text embedded in images with its translation into another language. |
| Approach: | They propose a simple task that renders parallel text over a plain background and a pipeline that obtains the transcript of the original image, translates it, and generates a new image similar to the original one. |
| Outcome: | The proposed approach outperforms existing models including an end-to-end approach and is competitive with other similar approaches. |
Multimodal Machine Translation with Text-Image In-depth Questioning (2025.findings-acl)
Copied to clipboard
| Challenge: | Multimodal machine translation (MMT) models focus on intermodal interactions, but focus on simple interactions between nouns and entities in image, overlooking global semantic alignment. |
| Approach: | They propose a Text-Image In-depth Questioning method to deepen interactions and optimize translations by utilizing visual data to capture global semantic alignment. |
| Outcome: | The proposed method achieves state-of-the-art results on five translation directions of Multi30K and AmbigCaps, with +2.35 BLEU on the challenging MSCOCO benchmark. |
Document Image Machine Translation with Dynamic Multi-pre-trained Models Assembling (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing TIMT tasks focus on text-line-level images. |
| Approach: | They propose to extend the existing TIMT task and introduce a new framework to translate a source document image to markdown-formatted target translation. |
| Outcome: | The proposed task aims to translate a source document image with long context and complex layout structure to markdown-formatted target translation. |
Exploring Better Text Image Translation with Multimodal Codebook (2023.acl-long)
Copied to clipboard
| Challenge: | Current studies on text image translation face bottlenecks due to lack of a publicly available dataset and poor optical character recognition. |
| Approach: | They propose a text image translation model with a multimodal codebook and an OCR dataset for Chinese-English translation. |
| Outcome: | The proposed model can associate the image with relevant texts, providing useful supplementary information for translation. |
Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review (2025.findings-acl)
Copied to clipboard
Pei Fu, Tongkun Guan, Zining Wang, Zhentao Guo, Chen Duan, Hao Sun, Boming Chen, Qianyi Jiang, Jiayao Ma, Kai Zhou, Junfeng Luo
| Challenge: | Recent advances in vision-language models have unified perception and understanding tasks within Visual Question Answering paradigms. |
| Approach: | They propose to outline timeline, architecture, and pipeline of nearly all TIU MLLMs and review their performance on mainstream benchmarks. |
| Outcome: | The proposed models perform well on mainstream benchmarks and are compared with other models. |
Generative Imagination Elevates Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing multimodal neural machine translation methods require triplets of bilingual sentence - image for training and tuples of source sentence . Existing methods require truncated images for inference, but ImagiT uses both source sentence and “imagined representation” to produce a target translation. |
| Approach: | They propose a multimodal machine translation method using visual imagination to generate a target translation from a sentence in a source language. |
| Outcome: | The proposed method significantly outperforms the existing text-only neural machine translation baselines and improves translation quality. |