CONSTRUCTURE: Benchmarking CONcept STRUCTUre REasoning for Multimodal Large Language Models

Challenge:	Existing multimodal large language models lack the ability to perceive the visual world with a deep concept structure cognition.
Approach:	They propose a concept-level benchmark to assess MLLMs’ hierarchical concept understanding and reasoning abilities.
Outcome:	The proposed model outperforms state-of-the-art models in concept structure reasoning evaluation.

Similar Papers

Overview

Challenge

Approach

Outcome

MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria (2025.naacl-long)

Copied to clipboard

Wentao Ge, Shunian Chen, Hardy Chen, Nuo Chen, Junying Chen, Zhihong Chen, Wenya Xie, Shuo Yan, ChenghaoZhu ChenghaoZhu, Ziyue Lin, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang

Multimodality

Challenge:	Existing evaluation methodologies for multimodal large language models are limited in evaluating objective queries without considering real-world user experiences.
Approach:	They propose to evaluate multimodal large language models with per-sample criteria using potent MLLM as the judge.
Outcome:	The proposed evaluation paradigm shows that it can be used to evaluate multimodal large language models with per-sample criteria.

CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships? (2025.acl-long)

Copied to clipboard

Aashish Anantha Ramakrishnan, Aadarsh Anantha Ramakrishnan, Dongwon Lee

Semantic Text Processing

Challenge:	Existing benchmarks focus on assessing factual and logical correctness in downstream tasks with limited emphasis on evaluating MLLMs’ ability to interpret pragmatic cues and intermodal relationships.
Approach:	They propose to use Coherence Relations to assess MLLMs' ability to perform multimodal discourse analysis using different prompting strategies.
Outcome:	The proposed model fails to match the performance of simple classifier-based benchmarks on 10+ MLLMs using different prompting strategies.

From Multimodal LLM to Human-level AI: Modality, Instruction, Reasoning, Efficiency and beyond (2024.lrec-tutorials)

Copied to clipboard

Hao Fei, Yuan Yao, Zhuosheng Zhang, Fuxiao Liu, Ao Zhang, Tat-Seng Chua

Responsible & Trustworthy NLP

Challenge:	This tutorial aims to deliver a comprehensive review of cutting-edge research in MLLMs.
Approach:	This tutorial will review cutting-edge research in MLLMs and examine the impact of ML in learning and reasoning.
Outcome:	This course will review cutting-edge research in MLLMs and examine the impact of ML models on learning, learning, and multimodal reasoning.

Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models (2025.findings-acl)

Copied to clipboard

Qianqi Yan, Yue Fan, Hongquan Li, Shan Jiang, Yang Zhao, Xinze Guan, Ching-Chen Kuo, Xin Eric Wang

Multimodality

Challenge:	Existing Multimodal Large Language Models (MLLMs) are predominantly trained on consistent visual-textual inputs, leaving open the question of whether they can handle semantic mismatches in layout-rich content.
Approach:	They propose to use multimodal inconsistency reasoning to assess MLLMs' ability to reason about semantic mismatches in webpages, presentation slides, and posters.
Outcome:	The proposed model outperforms open-source models in detecting inconsistencies in webpages, presentation slides, and posters while remaining vulnerable to inconsistent errors.

Probing Audio-Visual Reasoning in Multimodal Language Models through the Lens of Audio (2026.acl-long)

Copied to clipboard

Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, Xiangyu Yue

Multimodality

Challenge:	Recent multimodal large language models lack robust audio-visual integration ability and performance on DeafTest is highly correlated with AV-Odyssey accuracy.
Approach:	They propose a benchmarking tool that integrates audio-visual reasoning with audio-video cues to infer solutions.
Outcome:	The proposed model performs well on DeafTest, but lacks audio perception in simple audio tasks.

MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception (2024.acl-long)

Copied to clipboard

Yuhao Wang, Yusheng Liao, Heyang Liu, Hongcheng Liu, Yanfeng Wang, Yu Wang

Multimodality

Challenge:	Recent advances in multimodal large language models (MLLMs) have demonstrated exceptional capabilities in visual perception and understanding, but they also suffer from hallucinations, which limit their reliability as AI systems.
Approach:	They propose a benchmark to evaluate self-awareness in perception for multimodal large language models (MLLMs) by integrating image information with knowledge quadrants, and propose MM-SAP to evaluate this capability.
Outcome:	The proposed benchmark offers detailed analysis of MLLMs with self-awareness in perception.

Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Models (2025.findings-emnlp)

Copied to clipboard

Boyu Jia, Junzhe Zhang, Huixuan Zhang, Xiaojun Wan

Language Models

Challenge:	MLLMs have achieved significant breakthroughs in understanding across text and vision, but current models still face inconsistencies in reasoning outcomes.
Approach:	They propose to evaluate multimodal large language models using a multimodal knowledge reasoning dataset to examine the extent of consistency degradation.
Outcome:	The proposed evaluation tasks show that MLLMs are inefficient at integrating knowledge across modalities .

Multimodal Causal Reasoning Benchmark: Challenging Multimodal Large Language Models to Discern Causal Links Across Modalities (2025.findings-acl)

Copied to clipboard

Zhiyuan Li, Heng Wang, Dongnan Liu, Chaoyi Zhang, Ao Ma, Jieting Long, Weidong Cai

Multimodality

Challenge:	Existing MLLMs lack robustness in multimodal causal reasoning compared to their performance in textual settings.
Approach:	They propose a novel multimodal chain-of-thought (CoT) reasoning benchmark that leverages siamese images and text pairs to challenge MLLMs.
Outcome:	The proposed benchmark leverages siamese images and text pairs to challenge MLLMs.

WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code (2025.findings-acl)

Copied to clipboard

Zhiyu Lin, Zhengda Zhou, Zhiyuan Zhao, Tianrui Wan, Yilun Ma, Junyu Gao, Xuelong Li

Multimodality

Challenge:	Existing benchmarks for large language models focus on webpage generation outcomes.
Approach:	They propose a multi-view evaluation framework to evaluate MLLMs in four key areas: WebUI Perception, HTML Programming, WebUI-HTML Understanding, and WebUI to code.
Outcome:	The proposed framework evaluates MLLMs in four key areas: WebUI Perception, HTML Programming, WebUI-HTML Understanding, and WebUI to code.

Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task (2025.findings-acl)

Copied to clipboard

Yanbei Jiang, Yihao Ding, Chao Lei, Jiayang Ao, Jey Han Lau, Krista A. Ehinger

Multimodality

Challenge:	Existing AVR benchmarks focus on single-step reasoning, emphasizing the end result but neglecting the multi-stage nature of reasoning process.
Approach:	They propose a multi-stage AVR benchmark based on RAVEN to assess reasoning across varying levels of complexity.
Outcome:	The proposed metric considers the correctness of intermediate steps in addition to the final outcomes.