Can MLLMs Understand the Deep Implication Behind Chinese Images? (2025.acl-long)
Copied to clipboard
Chenhao Zhang, Xi Feng, Yuelin Bai, Xeron Du, Jinchang Hou, Kaixin Deng, Guangzeng Han, Qinrui Li, Bingli Wang, Jiaheng Liu, Xingwei Qu, Yifei Zhang, Qixuan Zhao, Yiming Liang, Ziqiang Liu, Feiteng Fang, Min Yang, Wenhao Huang, Chenghua Lin, Ge Zhang, Shiwen Ni
| Challenge: | MLLMs perform poorly on traditional culture images, indicating limitations in understanding high-level semantics and lacking a deep knowledge base of Chinese traditional culture. |
| Approach: | They propose to use Chinese images to assess MLLMs' higher-order perception and understanding of Chinese visual content. |
| Outcome: | The proposed model incorporates images that represent Chinese traditional culture, such as famous Chinese traditional paintings, to ensure the authenticity of the Chinese context. |
Similar Papers
MCS-Bench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in Chinese Classical Studies (2025.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) have advanced visual and language understanding, but their potential in Chinese Classical Studies (CCS) remains underexplored due to the lack of specialized benchmarks. |
| Approach: | They propose to develop a multimodal benchmark specifically designed for Chinese Classical Studies across multiple subdomains to bridge this gap. |
| Outcome: | The proposed benchmark spans seven core subdomains with a total of 45 meticulously designed tasks. |
Dunhuang-Bench: How Well Do MLLMs Understand Cultural Heritage? (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Multimodal Large Language Models (MLLMs) have led to extensive evaluations on Chinese cultural benchmarks. |
| Approach: | They construct a large-scale benchmark comprising 486 images and 22,970 QA pairs to evaluate MLLMs' cultural understanding. |
| Outcome: | The proposed benchmark incorporates three task formats to evaluate MLLMs’ cultural understanding: Question Answering with Text Description, Multi-turn Dialogue, and Question Answers with Choices. |
Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review (2025.findings-acl)
Copied to clipboard
Pei Fu, Tongkun Guan, Zining Wang, Zhentao Guo, Chen Duan, Hao Sun, Boming Chen, Qianyi Jiang, Jiayao Ma, Kai Zhou, Junfeng Luo
| Challenge: | Recent advances in vision-language models have unified perception and understanding tasks within Visual Question Answering paradigms. |
| Approach: | They propose to outline timeline, architecture, and pipeline of nearly all TIU MLLMs and review their performance on mainstream benchmarks. |
| Outcome: | The proposed models perform well on mainstream benchmarks and are compared with other models. |
MIBench: Evaluating Multimodal Large Language Models over Multiple Images (2024.emnlp-main)
Copied to clipboard
Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, Weiming Hu
| Challenge: | Existing benchmarks and MLLMs focus on single-image input scenarios, leaving performance of ML models when handling multiple images underexplored. |
| Approach: | They propose a benchmark to evaluate fine-grained abilities of multimodal large language models in multi-image scenarios. |
| Outcome: | The proposed benchmark categorizes the multi-image abilities into three scenarios: MII, MKS and MIC. |
BoYaEval: Evaluating Multimodal Large Language Models on Understanding Ancient Chinese Musical Scores (2026.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models excel in general tasks but struggle with specialized, structured cultural symbols. |
| Approach: | They evaluate 21 leading MLLMs and compare their performance to a benchmark for Ancient Chinese musical notation. |
| Outcome: | The benchmark evaluates 21 leading MLLMs on five types of ancient Chinese music notation systems. |
CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language (2026.acl-long)
Copied to clipboard
| Challenge: | CNSL-bench is the first comprehensive Chinese National Sign Language benchmark . current MLLMs are inferior to human performance, despite advances in multimodal modeling . |
| Approach: | They propose a Chinese National Sign Language benchmark to evaluate multimodal large language models in sign language understanding. |
| Outcome: | The proposed benchmark evaluates 21 open-source and proprietary MLLMs . results show that current models are inferior to human performance . |
RealBench: A Chinese Multi-image Understanding Benchmark Close to Real-world Scenarios (2025.findings-emnlp)
Copied to clipboard
Fei Zhao, Chengqiang Lu, Yufan Shen, Qimeng Wang, Yicheng Qian, Haoxin Zhang, Yan Gao, null Yiwu, Yao Hu, Zhen Wu, Shangyu Xing, Xinyu Dai
| Challenge: | RealBench is the first Chinese multimodal multi-image dataset . the dataset contains 9393 samples and 69910 images . |
| Approach: | They propose to create a Chinese multimodal multi-image dataset using 21 models . they use closed-source models that support multi-inputs as well as open-source visual and video models a . |
| Outcome: | The first Chinese multimodal multi-image dataset contains 9393 samples and 69910 images. |
TIU-Bench: A Benchmark for Evaluating Large Multimodal Models on Text-rich Image Understanding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-rich image understanding benchmarks lack scale and fragmented scenarios . a new full-image structured output format is proposed to enable fine-grained evaluation of perception and reasoning capabilities. |
| Approach: | They propose a large-scale, multilingual benchmark that includes over 100,000 annotations and 22,000 question-answer pairs. |
| Outcome: | The proposed framework provides a comprehensive platform for developing and evaluating next-generation multimodal AI systems. |
AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on basic abilities using nonverbal methods, such as yes-no and multiple-choice questions. |
| Approach: | They propose a benchmark that provides more nuanced evaluations of alignment capabilities for large Vision-Language Models (VLMs) they use a rule-calibrated evaluator that exceeds GPT-4's evaluation ability and a “alignment score” to assess the robustness and stability of models across diverse prompts. |
| Outcome: | The proposed benchmark covers 13 tasks across three categories and includes both single-turn and multi-turn dialogue scenarios. |
MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing scientific benchmarks lack human-annotated difficulty levels and structured taxonomies of scientific concepts. |
| Approach: | They propose a benchmark for evaluating mathematical and physical reasoning through text-only and text-image formats with human-annotated difficulty levels and detailed explanations. |
| Outcome: | The proposed model achieves only 63.77% accuracy and struggles with visual reasoning tasks. |