SHIFT: Selected Helpful Informative Frame for Video-guided Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Video-guided machine translation (VMT) aims to improve translation quality by integrating contextual information from paired short video clips. |
| Approach: | They propose a plug-and-play framework for video-guided machine translation with multimodal large language models. |
| Outcome: | The proposed framework improves performance of MLLMs while reducing computational cost. |
Similar Papers
DART: Disambiguation-Aware Reasoning for Video-guided Machine Translation (2026.acl-long)
Copied to clipboard
| Challenge: | Video-guided Machine Translation (VMT) uses short video clips to enhance translation quality, but many samples are text-sufficient. |
| Approach: | They propose a framework that integrates multimodal large language models’ multimodal reasoning into video-guided machine translation by using a pipeline for constructing training data based on multimodal relevance to translation. |
| Outcome: | The proposed framework improves multimodal information utilization in video-guided machine translation, yielding gains in translation quality and computational efficiency. |
CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation (2025.findings-emnlp)
Copied to clipboard
Emilio Villa-Cueva, Sholpan Bolatzhanova, Diana Turmakhan, Kareem Elzeky, Henok Biadglign Ademtew, Alham Fikri Aji, Vladimir Araujo, Israel Abebe Azime, Jinheon Baek, Frederico Belcavello, Fermin Cristobal, Jan Christian Blaise Cruz, Mary Dabre, Raj Dabre, Toqeer Ehsan, Naome A Etori, Fauzan Farooqui, Jiahui Geng, Guido Ivetta, Thanmay Jayakumar, Soyeong Jeong, Zheng Wei Lim, Aishik Mandal, Sofía Martinelli, Mihail Minkov Mihaylov, Daniil Orel, Aniket Pramanick, Sukannya Purkayastha, Israfel Salazar, Haiyue Song, Tiago Timponi Torrent, Debela Desalegn Yadeta, Injy Hamed, Atnafu Lambebo Tonja, Thamar Solorio
| Challenge: | a human-curated benchmark of over 5,800 triples of images is used to evaluate multimodal translation systems. |
| Approach: | They introduce a human-curated benchmark of over 5,800 triples of images along with parallel captions in English and regional languages. |
| Outcome: | The results show that visual context improves translation quality in culturally-specific items . |
Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models (2026.acl-long)
Copied to clipboard
Woody Haosheng Gan, Deqing Fu, Julian Asilis, Ollie Liu, Vatsal Sharan, Robin Jia, Willie Neiswanger
| Challenge: | Steering methods have emerged as effective tools for guiding large language models’ behavior, yet multimodal large language model lacks comparable techniques due to architectural diversity and limited availability of multimodal steering vectors. |
| Approach: | They validate steering vectors derived solely from text-only LLM backbones and use a cross-modal transfer technique to reuse existing interpretability tools. |
| Outcome: | The proposed steering vectors can guide and enhance multimodal models using SPAR, Mean Shift, and Linear Probing. |
VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Video-language models excel at understanding video content but struggle with spatial relationships, temporal ordering, and cross-frame continuity. |
| Approach: | They propose a framework that trains video-LLMs to distinguish accurate representations from carefully crafted adversarial examples. |
| Outcome: | Experiments show that VideoPASTA improves performance without human annotation or captioning . the framework can be used on various state-of-the-art video-LLMs with no human annotation . |
Multimodal Neural Machine Translation: A Survey of the State of the Art (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal neural machine translation (MNMT) is a task that aims to translate text into the target language using neural networks. |
| Approach: | They propose to integrate other modalities with textual data to enhance translation performance. |
| Outcome: | The proposed task aims to integrate visual modality with textual data to improve translation quality. |
Video-guided Machine Translation with Spatial Hierarchical Attention Network (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing studies use pretrained motion detection models as verb sense ambiguity representations to solve the verb sense problem. |
| Approach: | They propose to use video contents as auxiliary information to address the word sense ambiguity problem in machine translation. |
| Outcome: | Experiments on the VATEX dataset show that the proposed system achieves 35.86 BLEU-4 score, which is 0.51 score higher than the single model of the SOTA method. |
GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Experiments show that GrammaMT enhances translation performance on open-source instruction-tuned LLMs for various low- to high-resource languages across three benchmarks: (1) largest corpus, (2) challenging 2023 SIGMORPHON Shared Task data, (3) even in an out-of-domain setting with FLORES. |
| Approach: | They propose a grammatically-aware prompting approach that uses Interlinear Glossed Text . they propose gloss-shot, chain-gloss and model-glooss prompting strategies that are training-free . |
| Outcome: | Experiments show that GrammaMT improves translation performance on open-source instruction-tuned LLMs for various low- to high-resource languages across three benchmarks. |
Video-Helpful Multimodal Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal machine translation datasets contain images and video captions or instructional video subtitles, which rarely contain linguistic ambiguity. |
| Approach: | They propose an MMT dataset that contains ambiguous subtitles and a video-helpful evaluation set. |
| Outcome: | The proposed model performs significantly better than existing models on ambiguous subtitles dataset . it is based on a training set and video-helpful evaluation set . |
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues (2026.findings-acl)
Copied to clipboard
Yaning Pan, Qianqian Xie, Guohui Zhang, Zekun Moore Wang, Yongqian Wen, Yuanxing Zhang, Haoxuan Hu, Zhiyu Pan, Yibing Huang, Zhidong Gan, Yonghong Lin, An Ping, Shihao Li, Yanghai Wang, Tianhao Peng, Jiaheng Liu
| Challenge: | Existing evaluation benchmarks for Multimodal Large Language Models (MLLMs) focus on single-turn question answering, overlooking the complexity of multi-turn dialogues in real-world scenarios. |
| Approach: | They propose a video understanding benchmark for MLLMs in multi-turn dialogues that assesses six core competencies that focus on perceptivity and interactivity. |
| Outcome: | The MT-Video-Bench evaluates 1,000 multi-turn dialogues from diverse domains and reveals significant performance discrepancies and limitations in handling multi-turned video dialogues. |
On Vision Features in Multimodal Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is given to the quality of vision models. |
| Approach: | They develop a selective attention model to study the patch-level contribution of an image in multimodal machine translation. |
| Outcome: | The proposed model is able to learn translation from the visual modality on probing tasks and is compared with existing models. |