DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain (2026.acl-long)
Copied to clipboard
| Challenge: | Existing vision-language models lack fine-grained classification, single-view imagery, and inaccurate metadata. |
| Approach: | They propose a hierarchical, multi-view benchmark to evaluate VLMs across three levels of cognitive complexity. |
| Outcome: | The proposed benchmark evaluates vision-language models across three levels of complexity . it systematically identifies five primary failure modes . the proposed benchmarks are available on https://github.com/meituan/DiningBench. |
Similar Papers
AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on basic abilities using nonverbal methods, such as yes-no and multiple-choice questions. |
| Approach: | They propose a benchmark that provides more nuanced evaluations of alignment capabilities for large Vision-Language Models (VLMs) they use a rule-calibrated evaluator that exceeds GPT-4's evaluation ability and a “alignment score” to assess the robustness and stability of models across diverse prompts. |
| Outcome: | The proposed benchmark covers 13 tasks across three categories and includes both single-turn and multi-turn dialogue scenarios. |
Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models (2026.findings-acl)
Copied to clipboard
Mohammad Mahdi Abootorabi, Omid Ghahroodi, Anas Madkoor, Marzia Nouri, Doratossadat Dastgheib, Ehsaneddin Asgari
| Challenge: | Existing evaluations focus on piecemeal or disconnected tasks, obscuring critical cognitive weaknesses and providing little insight for targeted improvement. |
| Approach: | They propose a bilingual, cognitively human-grounded multimodal benchmark for VLMs that evaluates six levels of cognition through carefully designed image–question–answer tasks. |
| Outcome: | The proposed framework ensures scalability, cultural inclusivity, and linguistic fidelity. |
NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models have demonstrated their strong performance on IQ test questions, achieving high scores across many languages. |
| Approach: | They propose a dataset to evaluate cognitive multimodal reasoning and problem-solving skills of large models. |
| Outcome: | The proposed dataset contains 2,728 multiple-choice questions and 4,642 images spanning 26 categories. |
RealHiTBench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM-Based Table Analysis (2025.findings-acl)
Copied to clipboard
Pengzuo Wu, Yuhang Yang, Guangcheng Zhu, Chao Ye, Hong Gu, Xu Lu, Ruixuan Xiao, Bowen Bao, Yijing He, Liangyu Zha, Wentao Ye, Junbo Zhao, Haobo Wang
| Challenge: | Existing benchmarks for large language models focus on simple, flat table structures. |
| Approach: | They propose a benchmark to evaluate the performance of both Large Language Models and Multimodal LLMs across a variety of input formats for complex tabular data, including LaTeX, HTML, and PNG. |
| Outcome: | The proposed benchmark evaluates the performance of LLMs and Multimodal LLM models across a variety of input formats for complex tabular data, including LaTeX, HTML, and PNG. |
WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines (2025.naacl-long)
Copied to clipboard
Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Wang Yutong, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, Anar Rzayev, Anirban Das, Ashmari Pramodya, Aulia Adila, Bryan Wilie, Candy Olivia Mawalim, Cheng Ching Lam, Daud Abolade, Emmanuele Chersoni, Enrico Santus, Fariz Ikhwantri, Garry Kuwanto, Hanyang Zhao, Haryo Akbarianto Wibowo, Holy Lovenia, Jan Christian Blaise Cruz, Jan Wira Gotama Putra, Junho Myung, Lucky Susanto, Maria Angelica Riera Machin, Marina Zhukova, Michael Anugraha, Muhammad Farid Adilazuarda, Natasha Christabelle Santosa, Peerat Limkonchotiwat, Raj Dabre, Rio Alexander Audino, Samuel Cahyawijaya, Shi-Xiong Zhang, Stephanie Yulia Salim, Yi Zhou, Yinxuan Gui, David Ifeoluwa Adelani, En-Shiun Annie Lee, Shogo Okada, Ayu Purwarianti, Alham Fikri Aji, Taro Watanabe, Derry Tanti Wijaya, Alice Oh, Chong-Wah Ngo
| Challenge: | Vision Language Models struggle with cultural-specific knowledge, especially in languages other than English and in underrepresented cultural contexts. |
| Approach: | They propose a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects and a training dataset. |
| Outcome: | The proposed model performs better with correct location context, but struggles with adversarial contexts and predicting specific regional cuisines and languages. |
Benchmarking Deflection and Hallucination in Large Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks overlook conflicts between visual and textual evidence and the importance of generating deflections when incomplete knowledge is retrieved. |
| Approach: | They propose a dynamic curation pipeline that preserves benchmark difficulty over time . they propose 'vlm-DeflectionBench' benchmark to probe model behaviour under conflicting evidence . |
| Outcome: | The proposed benchmarks overlook conflicts between visual and textual evidence and are prone to obsolescence . the proposed benchmark is based on 2,775 samples spanning diverse retrieval settings . |
MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Medical Vision-Language Models are predominantly trained on professional literature, limiting their ability to communicate findings in the lay register required for patient-centered care. |
| Approach: | They propose a multimodal benchmark dedicated to expert-lay semantic alignment that enforces strict semantic equivalence by integrating unified medical language system (UMS) Concept Unique Identifiers (CUIs) with micro-level entity constraints. |
| Outcome: | The proposed benchmark enforces strict semantic equivalence by integrating unified medical language system (UMLS) Concept Unique Identifiers (CUIs) with micro-level entity constraints. |
From Recognition to Reasoning: Benchmarking and Enhancing MLLMs on Real-World Receipt Document Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing models for visual information extraction suffer from limitations in scale and realism . ReceiptBench is a large-scale, human-annotated benchmark for receipts . |
| Approach: | They propose a large-scale, human-annotated benchmark for visual information extraction . the method organizes information extraction into four hierarchical sub-tasks . |
| Outcome: | The proposed method surpasses proprietary models on complex reasoning tasks. |
Hospitality-VQA: Decision-Oriented Informativeness Evaluation for Vision–Language Models (2026.eacl-srw)
Copied to clipboard
Jeongwoo Lee, Baek Duhyeong, Eungyeol Han, Soyeon Shin, Gukin Han, Seungduk Kim, Jaehyun Jeon, Taewoo Jeong
| Challenge: | Existing VQA benchmarks focus on factual correctness but rarely capture what information users actually find useful. |
| Approach: | They propose a framework to quantify how much information an image–question pair provides . they conduct experiments with several state-of-the-art VLMs to determine their reliability . |
| Outcome: | The proposed framework quantifies how much information an image–question pair provides in hospitality contexts. |
MVP-Bench: Can Large Vision-Language Models Conduct Multi-level Visual Perception Like Humans? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing LVLMs perform visual perception at multiple levels, but they are not able to perform multi-level tasks. |
| Approach: | They propose a visual–language benchmark to evaluate LVLMs' perceptions . they use manipulated images to examine how LVLs can perform multi-level tasks . |
| Outcome: | The proposed model performs poorly on high-level perception tasks, the authors show . they also show that current models do not generalize in understanding semantics of synthetic images . |