Exploring In-Image Machine Translation with Real-World Background (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models for IIMT focus on simplified scenarios, which is far from reality and impractical for applications in the real world. |
| Approach: | They propose a model that separates the background and text-image from the source image and performs translation on the text- image directly. |
| Outcome: | The proposed model improves translation quality and visual effect in complex scenarios . it separates background and text-image from source image and performs translation on the text- image directly . |
Similar Papers
PRIM: Towards Practical In-Image Multilingual Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current research on in-image machine translation focuses on synthetic data with simple background, single font, fixed text position, and bilingual translation. |
| Approach: | They propose an end-to-end model to handle the challenge of practical conditions in PRIM . they annotate a real-world one-line text image with complex background, fonts, diverse text positions . |
| Outcome: | The proposed model improves translation quality and visual effect compared to other models. |
In-Image Machine Translation. A Preliminary Modular Approach (2026.eacl-srw)
Copied to clipboard
| Challenge: | In-image machine translation is a sub-task of Image-Based Machine Translation that aims to substitute text embedded in images with its translation into another language. |
| Approach: | They propose a simple task that renders parallel text over a plain background and a pipeline that obtains the transcript of the original image, translates it, and generates a new image similar to the original one. |
| Outcome: | The proposed approach outperforms existing models including an end-to-end approach and is competitive with other similar approaches. |
Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | In-image machine translation (IIMT) aims to translate an image containing texts in source language into an image with translations in target language. |
| Approach: | They propose an end-to-end IIMT model with four modules that translate images . they propose a two-stage training framework to assist the model in learning alignment across languages . |
| Outcome: | The proposed model outperforms cascaded models with only 70.9% of parameters and is highly accurate. |
AnyTrans: Translate AnyText in the Image with Large Scale Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in natural language processing and computer vision have made it possible to translate images with text in one language into equivalent images displaying that text translated into another language. |
| Approach: | They propose an all-encompassing framework for the task–In-Image Machine Translation (IIMT) that incorporates contextual cues from both textual and visual elements during translation. |
| Outcome: | The proposed framework can be constructed using open-source models and requires no training, making it highly accessible and expandable. |
In-Image Neural Machine Translation with Segmented Pixel Sequence-to-Sequence Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-Image Machine Translation (IIMT) aims to convert images containing texts from one language to another. |
| Approach: | They propose an end-to-end model instead of the traditional cascade methods which use optical character recognition followed by neural machine translation and text rendering. |
| Outcome: | The proposed model outperforms both cascade methods and current model in translation quality and robustness across various dimensions. |
Multimodal Neural Machine Translation Using Synthetic Images Transformed by Latent Diffusion Model (2023.acl-srw)
Copied to clipboard
| Challenge: | Existing methods to translate source language sentences using images are not optimal for machine translation. |
| Approach: | They propose a new multimodal neural machine translation model using synthetic images transformed by a latent diffusion model. |
| Outcome: | The proposed model improves translation performance on English-German translation tasks using the Multi30k dataset. |
LaTeXMT: Machine Translation for LaTeX Documents (2025.emnlp-demos)
Copied to clipboard
| Challenge: | LaTeXMT is a software solution for structure-preserving, source-to-source translation of LaTex documents. |
| Approach: | They propose a software solution for structure-preserving, source-to-source translation of LaTeX documents . authors propose transformer-based language models which can be trained on plain text . |
| Outcome: | The proposed software is available under the LGPL-3.0 open-source licence and a web version is publicly available. |
INMT: Interactive Neural Machine Translation Prediction (D19-3)
Copied to clipboard
| Challenge: | Existing MT systems are only useful for information assimilation, and require substantial manual post processing. |
| Approach: | They propose an Interactive Machine Translation interface that assists human translators with on-the-fly hints and suggestions. |
| Outcome: | The proposed interface makes the end-to-end translation process faster, more efficient and creates high-quality translations. |
UMTIT: Unifying Recognition, Translation, and Generation for Multimodal Text Image Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Current Image machine translation (IMT) relies on a cascaded system that combines Optical Character Recognition (OCR) and a complex process of rendering the translated text back onto the source image. |
| Approach: | They propose a multimodal image-text translation model that generates consistent target images . they use two image-to-text conversion steps to convert images to text to recognize source text . |
| Outcome: | The proposed model outperforms existing methods and surpasses state-of-the-art methods in text recognition tasks. |
Exploiting Monolingual Data at Scale for Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) is a well-known and expensive task. |
| Approach: | They propose a method to use target-side monolingual data for neural machine translation and propose 'synthetic bitext' they propose generating synthetic bitext by translating monolingual into the other domain using models pretrained on genuine bitext. |
| Outcome: | The proposed approach achieves state-of-the-art results on WMT16, WMT17, WTM18 EnglishGerman translations and WTM19 GermanFrench translations. |