VISA: An Ambiguous Subtitles Dataset for Visual Scene-aware Machine Translation (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing multimodal machine translation datasets contain images and video captions or general subtitles which rarely contain linguistic ambiguity. |
| Approach: | They propose a dataset that consists of Japanese-English parallel sentence pairs and corresponding video clips. |
| Outcome: | The proposed dataset is challenging for the latest MMT system and can facilitate MMT research. |
Similar Papers
Video-Helpful Multimodal Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal machine translation datasets contain images and video captions or instructional video subtitles, which rarely contain linguistic ambiguity. |
| Approach: | They propose an MMT dataset that contains ambiguous subtitles and a video-helpful evaluation set. |
| Outcome: | The proposed model performs significantly better than existing models on ambiguous subtitles dataset . it is based on a training set and video-helpful evaluation set . |
3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset (2024.lrec-main)
Copied to clipboard
Xinyu Ma, Xuebo Liu, Derek F. Wong, Jun Rao, Bei Li, Liang Ding, Lidia S. Chao, Dacheng Tao, Min Zhang
| Challenge: | Existing studies have shown that visual information in existing MMT datasets is insufficient, causing models to disregard it and overestimate their capabilities. |
| Approach: | They propose to use 3AM to create an ambiguity-aware multimodal machine translation dataset. |
| Outcome: | The proposed dataset includes more ambiguity and a greater variety of captions and images than other MMT datasets. |
Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work in multimodal machine translation (MT) has shown that ambiguity can be resolved using accompanying context such as images. |
| Approach: | They propose a multimodal machine translation approach based on a strong text-only MT model and a novel guided self-attention mechanism to train it. |
| Outcome: | The proposed model outperforms existing models on EnglishFrench, EnglishGerman and EnglishCzech benchmarks and is freely available. |
BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation (2023.findings-acl)
Copied to clipboard
Liyan Kang, Luyang Huang, Ningxin Peng, Peihao Zhu, Zewei Sun, Shanbo Cheng, Mingxuan Wang, Degen Huang, Jinsong Su
| Challenge: | Existing datasets focus on captions describing images or videos, which are not large and diverse enough. |
| Approach: | They propose a large-scale video subtitle translation dataset to facilitate multi-modality machine translation. |
| Outcome: | The proposed dataset is 10 times larger than the widely used *How2* and *VaTeX* datasets. |
Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Multimodal machine translation (MMT) aims to improve translation quality by incorporating information from other modalities, such as vision. |
| Approach: | They propose a framework for multimodal machine translation that utilizes large-scale non-triple data and a multimodal translation dataset. |
| Outcome: | The proposed method can significantly improve translation performance with more non-triple data. |
MUCAR: Benchmarking Multilingual Cross-Modal Ambiguity Resolution for Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
Xiaolong Wang, Zhaolu Kang, Wangyuxuan Zhai, Xinyue Lou, Yunghwei Lai, Ziyue Wang, Yawen Wang, Kaiyu Huang, Yile Wang, Peng Li, Yang Liu
| Challenge: | Existing multimodal benchmarks overlook linguistic and visual ambiguities, authors say . ambiguity resolution between modalities is lacking in multimodal large language models . |
| Approach: | They propose a benchmark to evaluate multimodal ambiguity resolution across multilingual and cross-modal scenarios. |
| Outcome: | a new benchmark evaluates multimodal ambiguity resolution across multilingual and cross-modal scenarios . the benchmark shows that MLLMs can resolve ambiguities in image-text alignment . however, existing benchmarks often overlook linguistic and visual ambiguties . |
Multimodal Lexical Translation (L18-1)
Copied to clipboard
| Challenge: | Multimodal Lexical Translation (MLT) is a task that aims to translate ambiguous words given their context -an image and a sentence in the source language. |
| Approach: | They introduce a task to translate an ambiguous word given its context -an image and a sentence in the source language. |
| Outcome: | The proposed task is based on the Multi30K dataset and uses word-alignment followed by human inspection to select subsets of the dataset which are difficult to translate. |
Detect, Disambiguate, and Translate: On-Demand Visual Reasoning for Multimodal Machine Translation with Large Vision-Language Models (2025.naacl-long)
Copied to clipboard
Danyang Liu, Fanjie Kong, Xiaohang Sun, Dhruva Patil, Avijit Vajpayee, Zhu Liu, Vimal Bhat, Najmeh Sadoughi
| Challenge: | Multimodal machine translation (MMT) aims to leverage additional modalities beyond text . current MMT systems rely heavily on monolingual English captioning data . |
| Approach: | They propose a reasoning-based framework to leverage large-scale vision-language models for MMT . they propose Detect, Disambiguate, and Translate framework to detect ambiguity in input sentence . |
| Outcome: | The proposed framework outperforms state-of-the-art models in disambiguation accuracy and translation quality. |
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies report improvements when equipping models with multimodal information, but it remains unclear whether such improvements actually come from the multimodal part. |
| Approach: | They propose to extend conventional text-only translation models with multimodal information by extending them with visual input. |
| Outcome: | The proposed models replicate similar gains as recently developed multimodal-integrated systems achieved, but learn to ignore multimodal information. |
Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal machine translation models outperform text-only models when visual context is available, but recent studies have shown that the performance of MMT models is only marginally impacted when the associated image is replaced with an unrelated image or noise. |
| Approach: | They propose to use visual data to highlight the importance of visual inputs in MMT models to enhance their leverage. |
| Outcome: | The proposed models outperform text-only models when visual context is available, but the results show that the visual context might not be exploited by the models at all. |