Context-Informed Machine Translation of Manga using Multimodal Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Automated manga translation is a promising potential solution, but it is underdeveloped due to the need to incorporate visual elements into the translation process to resolve ambiguities. |
| Approach: | They propose a method that leverages the vision component of multimodal large language models to improve translation quality and evaluate the impact of translation unit size, context length, and propose 'token efficient' approach for manga translation. |
| Outcome: | The proposed method achieves state-of-the-art results for Japanese-English translation and sets a new standard for Japanese and Polish translation. |
Similar Papers
M2C: Towards Automatic Multimodal Manga Complement (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal manga analysis focuses on enhancing manga understanding with visual and textual features. |
| Approach: | They propose a task to enhance manga understanding with visual and textual features by providing a shared semantic space for vision and language understanding. |
| Outcome: | The proposed task provides a shared semantic space for vision and language understanding. |
MangaVQA and MangaLMM: A Benchmark and Specialized Model for Multimodal Manga Understanding (2026.findings-eacl)
Copied to clipboard
Jeonghun Baek, Kazuki Egashira, Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Hikaru Ikuta, Kiyoharu Aizawa
| Challenge: | Manga is a richly multimodal narrative form that blends images and text in complex ways. |
| Approach: | They propose two benchmarks for multimodal manga understanding: mangaOCR and mangaVQA . mangaVQ consists of 526 high-quality, manually constructed question-answer pairs . |
| Outcome: | The proposed model is finetuned from the open-source LMM Qwen2.5-VL . it compares with proprietary models such as GPT-4o and Gemini 2.5 to evaluate its performance . |
Utilizing Longer Context than Speech Bubbles in Automated Manga Translation (2024.lrec-main)
Copied to clipboard
Hiroto Kaino, Soichiro Sugihara, Tomoyuki Kajiwara, Takashi Ninomiya, Joshua B. Tanner, Shonosuke Ishiwatari
| Challenge: | Existing methods to capture contextual information for manga machine translation are difficult to perform . unofficially translated pirated copies of manga are circulating overseas in large numbers . |
| Approach: | They propose two new ways to capture broader contextual information in manga machine translation . scene-based translation considers previous scene and broader context information . detailed analysis reveals the effect of zero-anaphora resolution in translation - highlighting the usefulness of longer contextual information if manga is translated in Japanese . |
| Outcome: | The proposed methods improve translation quality for manga (Japanese-style comics) the results show that the combined methods achieve the highest quality. |
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | MultiModal Large Language Models (MM-LLMs) have undergone significant advances in the past year . traditional MM models incur substantial computational costs, especially when trained from scratch . |
| Approach: | They propose a taxonomy encompassing 126 MM-LLMs and summarize key training recipes to enhance their potency. |
| Outcome: | The proposed models preserve the reasoning and decision-making capabilities of LLMs and empower diverse range of MM tasks. |
Can We Edit Multimodal Large Language Models? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to edit multimodal models have been used to incrementally infuse a language model with a new set of facts. |
| Approach: | They construct a benchmark for editing multimodal Large Language Models and establish metrics for evaluation. |
| Outcome: | The proposed benchmarks show that editing multimodal models is not as difficult as editing single-modal models. |
Harnessing PDF Data for Improving Japanese Large Multimodal Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) have demonstrated strong performance in English, but their effectiveness in Japanese remains limited due to the lack of high-quality training data. |
| Approach: | They propose a pipeline that leverages pretrained models to extract image-text pairs from PDFs . they use layout analysis, OCR, and vision-language pairing to enrich the training data . |
| Outcome: | The proposed pipeline extracts image-text pairs from Japanese PDFs, eliminating manual annotations. |
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis (2024.findings-naacl)
Copied to clipboard
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, Lei Li
| Challenge: | Existing studies show that large language models (LLMs) can handle multilingual machine translation (MMT) However, the multilingual translation ability of LLMs remains under-explored. |
| Approach: | They evaluate eight popular LLMs including ChatGPT and GPT-4 to determine their performance in multilingual machine translation. |
| Outcome: | The proposed model can generate moderate translation even on zero-resource languages and cross-lingual exemplars can provide better task guidance for low-resourced translation than exemplar in the same language pairs. |
Benchmarking and Improving Long-Text Translation with Large Language Models (2024.findings-acl)
Copied to clipboard
Longyue Wang, Zefeng Du, Wenxiang Jiao, Chenyang Lyu, Jianhui Pang, Leyang Cui, Kaiqiang Song, Derek Wong, Shuming Shi, Zhaopeng Tu
| Challenge: | Recent studies have illuminated the promising capabilities of large language models (LLMs) in handling long texts. |
| Approach: | They construct a benchmark dataset specifically designed for the finetuning and evaluation of large language models (LLMs) they compare LLMs with MT models and find they exhibit shortcomings in long-text domains . |
| Outcome: | The proposed model performs better in long-text translation, and its performance diminishes as document size increases. |
A Visual Attention Grounding Neural Model for Multimodal Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to multimodal machine translation do not integrate visual information into the translation process. |
| Approach: | They propose a multimodal machine translation model that utilizes parallel visual and textual information. |
| Outcome: | The proposed model outperforms existing methods on the Multi30K and Ambiguous COCO datasets. |
m3P: Towards Multimodal Multilingual Translation with Multimodal Prompt (2024.lrec-main)
Copied to clipboard
Jian Yang, Hongcheng Guo, Yuwei Yin, Jiaqi Bai, Bing Wang, Jiaheng Liu, Xinnian Liang, LinZheng Chai, Liqun Yang, Zhoujun Li
| Challenge: | Existing multimodal neural machine translation models focus on bilingual translation, but experimental results show that they outperform the text-only baselines and multilingual multimodal methods by a large margin. |
| Approach: | They propose a framework to leverage the multimodal prompt to guide the Multimodal Multilingual Neural Machine Translation (m3P) this framework aligns the representations of different languages with the same meaning and generates the conditional vision-language memory for translation. |
| Outcome: | The proposed framework outperforms previous text-only baselines and multilingual multimodal methods by a large margin. |