Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that understand image and text but also cross-reference in-between are lacking in evaluation data resources. |
| Approach: | They propose a multimodal evaluation pipeline to automatically generate question-answer pairs to test models’ understanding of the visual scene, text, and related knowledge. |
| Outcome: | The proposed model can answer the highly semantic VCR question correctly but fails to answer related visual question (Q2), textual question (q3), and background knowledge question ( Q4) as shallow mappings with language priors and unbalanced utilization of information between modalities. |
Similar Papers
What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge (2022.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation methods to measure what language models learn from multimodal training are lacking. |
| Approach: | They propose two evaluation tasks to measure commonsense knowledge in language models by using visual data to evaluate multimodal models and unimodal baselines. |
| Outcome: | The proposed evaluation tasks show that training on a visual modality improves on the visual commonsense knowledge in language models. |
Visual Commonsense in Pretrained Unimodal and Multimodal Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Fig. 1 shows how text-only and image-only models can capture commonsense visual attributes, but reporting bias affects their performance. |
| Approach: | They use a Visual Commonsense Tests dataset to validate their findings . they find multimodal models better reconstruct attribute distributions, but are still subject to reporting bias . |
| Outcome: | The proposed model improves on the unimodal and multimodal models, but is still subject to reporting bias. |
ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for visual commonsense reasoning (VCR) use pre-trained large language models and pre-training visionlanguage models. |
| Approach: | They propose a collaborative approach where pre-trained LLMs serve as problem classifiers to analyze problem category and either use VLMs to answer directly or actively instruct LLM to gather relevant visual elements to support potential commonsense inferences. |
| Outcome: | The proposed approach outperforms all other methods without in-domain fine-tuning on two VCR benchmark datasets. |
VCD: A Dataset for Visual Commonsense Discovery in Images (2025.findings-acl)
Copied to clipboard
| Challenge: | Visual commonsense data sets lack visual grounded representations of commonsensense . existing knowledge bases lack visual-based knowledge tied to actual visual scenes . |
| Approach: | They present a large-scale visual commonsense dataset with over 100,000 images and 14 million object-commonsense pairs that integrates both Seen (directly observable) and Unseen (inferrable) commonsens. |
| Outcome: | The proposed model integrates Seen (directly observable) and Unseen (inferrable) commonsense across Property, Action, and Space aspects. |
BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models (2026.eacl-long)
Copied to clipboard
Bryan Chen Zhengyu Tan, Weihua Zheng, Zhengyuan Liu, Nancy F. Chen, Hwaran Lee, Kenny Tsu Wei Choo, Roy Ka-Wei Lee
| Challenge: | Existing evaluations assess static recall or isolated visual grounding, leaving unanswered whether VLMs possess robust and transferable cultural understanding. |
| Approach: | They propose a multimodal, multicultural benchmark to evaluate the robustness of everyday cultural knowledge in vision-language models across linguistic rephrasings and visual modalities. |
| Outcome: | ‘BLEnD-Vis‘ constructs 313 culturally grounded question templates spanning 16 regions and generates three aligned multiple-choice formats. |
MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing Large Vision-Language Models (LVLMs) lack integrated commonsense knowledge . lack of integrated common knowledge limits their robustness and accuracy in VQA . |
| Approach: | They propose a framework to enhance multimodal inference by integrating commonsense reasoning. |
| Outcome: | MAGIC-VQA improves comprehensive benchmark datasets, surpassing existing models in tasks requiring advanced commonsense reasoning. |
Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) show reasoning promise, yet their visual perception is a bottleneck. |
| Approach: | They propose a visual perception benchmark to test the visual perception of MLLMs. |
| Outcome: | The proposed benchmark examines MLLMs' visual perception abilities with 1758 images and 2612 questions. |
TIU-Bench: A Benchmark for Evaluating Large Multimodal Models on Text-rich Image Understanding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-rich image understanding benchmarks lack scale and fragmented scenarios . a new full-image structured output format is proposed to enable fine-grained evaluation of perception and reasoning capabilities. |
| Approach: | They propose a large-scale, multilingual benchmark that includes over 100,000 annotations and 22,000 question-answer pairs. |
| Outcome: | The proposed framework provides a comprehensive platform for developing and evaluating next-generation multimodal AI systems. |
FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluation (2025.acl-demo)
Copied to clipboard
| Challenge: | FlagEvalMM is an evaluation framework designed to assess multimodal models . it is designed to be used for vision-language understanding and generation tasks . |
| Approach: | They propose an evaluation framework that decouples model inference from evaluation through an independent evaluation service. |
| Outcome: | The evaluation framework offers accurate and efficient insights into model strengths and limitations. |
Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment (2024.acl-long)
Copied to clipboard
| Challenge: | Recent Large Multimodal Models (LMMs) focus on visual knowledge-dimension alignment, but ignore visual knowledge. |
| Approach: | They propose a cognitive visual-language mapper that integrates visual-linguistic knowledge alignment with a fine-grained knowledge Adapter. |
| Outcome: | The proposed model significantly improves LMMs on knowledge-based visual question answering (VQA) it also improves the performance of other models, including GPT-4V and Gemini-Pro. |