Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models are trained on single-turn vision question-answering tasks, which do not accurately reflect real-world human conversations. |
| Approach: | They propose a large-scale multi-turn multimodal dialogue dataset that uses rules and GPT assistance to generate a multi-turned multimodal dialog dataset. |
| Outcome: | The proposed dataset is a strong benchmark for multi-turn multimodal dialogue learning . it features complex dialogues with contextual dependencies that force models to track, ground, and recall information across multiple turns and disparate visual regions. |
Similar Papers
MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation (2025.acl-long)
Copied to clipboard
Haochen Xue, Feilong Tang, Ming Hu, Yexin Liu, Qidong Huang, Yulong Li, Chengzhi Liu, Zhongxing Xu, Chong Zhang, Chun-Mei Feng, Yutong Xie, Imran Razzak, Zongyuan Ge, Jionglong Su, Junjun He, Yu Qiao
| Challenge: | Existing multimodal large language models lack the ability to memorize, recall, and reason in sustained interactions. |
| Approach: | They propose a multimodal real-world conversation benchmark for evaluating open-ended abilities of multimodal large language models. |
| Outcome: | The proposed benchmarks show that the models perform better in open-ended conversations. |
CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships? (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on assessing factual and logical correctness in downstream tasks with limited emphasis on evaluating MLLMs’ ability to interpret pragmatic cues and intermodal relationships. |
| Approach: | They propose to use Coherence Relations to assess MLLMs' ability to perform multimodal discourse analysis using different prompting strategies. |
| Outcome: | The proposed model fails to match the performance of simple classifier-based benchmarks on 10+ MLLMs using different prompting strategies. |
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct (2025.findings-acl)
Copied to clipboard
Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Yongbin Li, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Hamid Alinejad-Rokny, Xiaobo Xia, Jingkuan Song, Fei Huang
| Challenge: | a new framework for image-text instruction data evolution improves MLLM performance . lack of high-quality instruction data remains a major bottleneck in ML modeling . |
| Approach: | They propose a multimodal instruction data evolution framework that iteratively enhances data quality through fine-grained perception, cognitive reasoning, and interaction evolution. |
| Outcome: | The proposed approach improves MLLM performance in nine vision-language tasks while using significantly less data. |
Grounding Multilingual Multimodal LLMs With Cultural Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new data-centric approach could address cultural gaps in multimodal large language models . despite being trained on billions of image-text pairs, today's models are biased towards English and Western data. |
| Approach: | They propose a data-centric approach that directly grounds MLLMs in cultural knowledge. |
| Outcome: | The proposed approach outperforms open-source models on cultural-focused benchmarks without degrading results on mainstream vision–language tasks. |
MM-ChatAlign: A Novel Multimodal Reasoning Framework based on Large Language Models for Entity Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing MMEA methods rely on knowledge representation learning (KRL) to measure the similarity of entity embeddings. |
| Approach: | They propose a framework that utilizes the visual reasoning abilities of MLLMs for multimodal entity alignment. |
| Outcome: | The proposed framework integrates the visual reasoning abilities of MLLMs for multimodal entity alignment. |
Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing Multimodal Large Language Models (MLLMs) are predominantly trained on consistent visual-textual inputs, leaving open the question of whether they can handle semantic mismatches in layout-rich content. |
| Approach: | They propose to use multimodal inconsistency reasoning to assess MLLMs' ability to reason about semantic mismatches in webpages, presentation slides, and posters. |
| Outcome: | The proposed model outperforms open-source models in detecting inconsistencies in webpages, presentation slides, and posters while remaining vulnerable to inconsistent errors. |
MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues (2024.acl-long)
Copied to clipboard
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, Wanli Ouyang
| Challenge: | Large Language Models (LLMs) have greatly enhanced dialogue systems, but evaluation of their capabilities remains a challenge. |
| Approach: | They propose a model to evaluate the fine-grained abilities of Large Language Models in multi-turn dialogues. |
| Outcome: | The proposed model evaluates 21 popular chatbots based on MT-Bench-101 . it includes 3 overarching abilities and 13 distinct tasks within multi-turn dialogue scenarios. |
Multimodal Causal Reasoning Benchmark: Challenging Multimodal Large Language Models to Discern Causal Links Across Modalities (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing MLLMs lack robustness in multimodal causal reasoning compared to their performance in textual settings. |
| Approach: | They propose a novel multimodal chain-of-thought (CoT) reasoning benchmark that leverages siamese images and text pairs to challenge MLLMs. |
| Outcome: | The proposed benchmark leverages siamese images and text pairs to challenge MLLMs. |
MMCoQA: Conversational Question Answering over Text, Tables, and Images (2022.acl-long)
Copied to clipboard
| Challenge: | Existing conversational QA systems only use a single knowledge source, e.g., paragraphs or knowledge graph, and assume it contains enough evidence to extract answers to users' questions. |
| Approach: | They propose a task to answer users' questions with multimodal knowledge sources via multi-turn conversations using a multimodal dataset. |
| Outcome: | The proposed task brings a series of research challenges, including but not limited to priority, consistency, and complementarity of multimodal knowledge. |
Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models (2026.acl-long)
Copied to clipboard
Woody Haosheng Gan, Deqing Fu, Julian Asilis, Ollie Liu, Vatsal Sharan, Robin Jia, Willie Neiswanger
| Challenge: | Steering methods have emerged as effective tools for guiding large language models’ behavior, yet multimodal large language model lacks comparable techniques due to architectural diversity and limited availability of multimodal steering vectors. |
| Approach: | They validate steering vectors derived solely from text-only LLM backbones and use a cross-modal transfer technique to reuse existing interpretability tools. |
| Outcome: | The proposed steering vectors can guide and enhance multimodal models using SPAR, Mean Shift, and Linear Probing. |