Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks for multimodal large language models (MLLMs) are language-centric or heavily reliant on simulated environments, rarely probing the nuanced, knowledge-intensive reasoning essential for practical, real-world scenarios. |
| Approach: | They propose a task of Sparsely Grounded Visual Navigation to evaluate MLLM-driven agents in city navigation in four diverse global cities. |
| Outcome: | The proposed benchmark encompassing four diverse global cities evaluates agents' decision-making abilities in city navigation. |
Similar Papers
VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models have demonstrated remarkable capabilities across vision-language tasks, but their performance as embodied agents needs further exploration. |
| Approach: | They propose a framework to evaluate multimodal large language models as zero-shot agents . they find that enhancing prevalent agents with Chain-of-Thought reasoning and self-reflection leads to an unexpected performance decrease. |
| Outcome: | The proposed framework enables comparisons and component-level ablations across diverse MLLM architectures, agent designs, and navigation tasks. |
VIVA+: Human-Centered Situational Decision-Making (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) show promising results in complex, human-centered environments, yet evaluating their capacity for nuanced, humanlike reasoning and decision-making remains challenging. |
| Approach: | They introduce VIVA+, a cognitively grounded benchmark for evaluating the reasoning and decision-making of MLLMs in human-centered situations. |
| Outcome: | The VIVA+ model is based on 1,317 real-world situations paired with 6,373 multiple-choice questions . it consists of three core abilities for decision-making: (1) Foundational Situation Comprehension, (2) Context-Driven Action Justification, and (3) Reflective Reasoning. |
UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces (2025.acl-long)
Copied to clipboard
Baining Zhao, Jianjie Fang, Zichao Dai, Ziyou Wang, Jirong Zha, Weichen Zhang, Chen Gao, Yue Wang, Jinqiang Cui, Xinlei Chen, Yong Li
| Challenge: | Large multimodal models exhibit remarkable intelligence, yet their embodied cognitive abilities during motion in open-ended urban aerial spaces remain to be explored. |
| Approach: | They propose a benchmark to evaluate whether large multimodal models can process continuous first-person visual observations like humans. |
| Outcome: | The proposed model can process first-person visual observations like humans, enabling recall, perception, reasoning, and navigation. |
Understanding GUI Agent Localization Biases through Logit Sharpness (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal large language models often exhibit hallucinations that compromise reliability . despite promising performance, these models often display systematic localization errors . |
| Approach: | They propose a framework that categorizes model predictions into four distinct types . they propose metric that evaluates alignment between semantic continuity and logits distribution . |
| Outcome: | The proposed framework categorizes model predictions into four different types . it reveals nuanced failure modes beyond traditional accuracy metrics . |
PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain (2024.findings-acl)
Copied to clipboard
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, Baobao Chang
| Challenge: | a new multimodal decision-making benchmark evaluates the integrated capabilities of multimodal large language models. |
| Approach: | They propose a multimodal decision-making benchmark for evaluating MLLMs . they propose an automatic evaluation protocol to assess 10 prevalent ML models . |
| Outcome: | The proposed benchmark improves performance of multimodal large language models in three scenarios . the model is required to integrate multiple capabilities to make accurate decisions . |
Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios (2026.findings-acl)
Copied to clipboard
Xiaomin Li, Tala Wang, Zichen Zhong, Ying Zhang, Zirui Zheng, Takashi Isobe, Dezhuang Li, Huchuan Lu, You He, Xu Jia
| Challenge: | Existing benchmarks focus on evaluating MLLMs’ pre-existing knowledge or perceptual understanding, often neglecting the critical capability of reasoning. |
| Approach: | They propose a benchmark designed for visual clue-driven reasoning in daily scenarios that combines rigorous grounding in authentic daily activities and challenging query design that necessitates more than surface-level perception. |
| Outcome: | The proposed benchmark identifies visual clues and their ability to provide robust reasoning in daily scenarios. |
Multimodal Language Models Show Evidence of Embodied Simulation (2024.lrec-main)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) are gaining popularity as partial solutions to the “symbol grounding problem” faced by language models trained on text alone. |
| Approach: | They propose to use multimodal large language models to integrate linguistic representations with data from other modalities to investigate whether they are integrated into a model. |
| Outcome: | The proposed models are sensitive to visual features like object shape when it is implied by a verbal description of an event. |
From Multimodal LLM to Human-level AI: Modality, Instruction, Reasoning, Efficiency and beyond (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | This tutorial aims to deliver a comprehensive review of cutting-edge research in MLLMs. |
| Approach: | This tutorial will review cutting-edge research in MLLMs and examine the impact of ML in learning and reasoning. |
| Outcome: | This course will review cutting-edge research in MLLMs and examine the impact of ML models on learning, learning, and multimodal reasoning. |
Can Multimodal Large Language Models Understand Spatial Relations? (2025.acl-long)
Copied to clipboard
| Challenge: | Spatial relation reasoning is a crucial task for multimodal large language models to understand the objective world. |
| Approach: | They propose a human-annotated spatial relation reasoning benchmark based on COCO2017 to improve MLLMs' spatial relation thinking. |
| Outcome: | The proposed benchmark achieves 48.14% accuracy, far below the human-level accuracy of 98.40%. |
Probing Multimodal Large Language Models for Global and Local Semantic Representations (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the ability of MLLMs to generate single tokens one by one, while lacking studies about how their representation vectors can encode global multimodal information. |
| Approach: | They propose to use image-caption corpus to train Multimodal Large Language Models (MLLMs) . they find that the topmost layers encode more global semantic information . |
| Outcome: | The proposed models can encode more global semantic information, rather than the topmost layers, and perform better on visual-language entailment tasks. |