AbsVis – Benchmarking How Humans and Vision-Language Models “See” Abstract Concepts in Images (2025.emnlp-main)
Copied to clipboard
| Challenge: | Abstract concepts like mercy and peace lack clear visual grounding, and therefore challenge humans and models to provide suitable image representations. |
| Approach: | They propose a dataset of 675 images annotated with 14,175 concept–explanation attributions from humans and two Vision-Language Models where each concept is accompanied by a textual explanation. |
| Outcome: | The proposed dataset compares human and VLM attributions in terms of diversity, abstractness, and alignment, and shows that overlapping concepts are most preferred. |
Similar Papers
From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have shown emerging capabilities through large-scale training that have made them gain popularity in recent years. |
| Approach: | They propose to perform retrieval across universals and cultural visual grounding tasks to assess cultural diversity across universal and culture-specific local concepts. |
| Outcome: | The proposed benchmarks show that the models perform significantly across cultures, underscoring the need for enhancing multicultural understanding in vision-language models. |
If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies assume that VLMs prioritize visual attributes to represent concepts. |
| Approach: | They propose a novel approach to characterize features important for VLMs using reinforcement learning. |
| Outcome: | The proposed approach characterizes features that are important for VLMs . it shows that spurious descriptions have a major role in VLM representations despite providing no helpful information. |
MVP-Bench: Can Large Vision-Language Models Conduct Multi-level Visual Perception Like Humans? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing LVLMs perform visual perception at multiple levels, but they are not able to perform multi-level tasks. |
| Approach: | They propose a visual–language benchmark to evaluate LVLMs' perceptions . they use manipulated images to examine how LVLs can perform multi-level tasks . |
| Outcome: | The proposed model performs poorly on high-level perception tasks, the authors show . they also show that current models do not generalize in understanding semantics of synthetic images . |
Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. |
| Approach: | They propose to use a multiple granularity attribute-centric benchmark and training mixture to evaluate LVLMs’ fine-grained visual comprehension ability. |
| Outcome: | The proposed model improves on LLaVa-1.5, InstructBLIP and GPT-4V and demonstrates that they struggle to generate descriptive visual attributes based on a concept that appears within an input image despite their prominent zero-shot image captioning ability. |
Naming, Describing, and Quantifying Visual Objects in Humans and LLMs (2024.acl-short)
Copied to clipboard
| Challenge: | Recent work has highlighted that speakers display a wide range of variability when asked to utter sentences, resulting in inter-speaker variability but also variability over time for the same speaker. |
| Approach: | They evaluate Vision & Language Large Language Models (VLLMs) on three categories where humans show great subjective variability concerning the distribution over plausible labels. |
| Outcome: | The proposed models can mimic human distributions over plausible labels, but fail to assign quantifiers, a task that requires more accurate, high-level reasoning. |
Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that deep vision-only and language-only models project inputs into a partially aligned representational space. |
| Approach: | They investigate whether a model's representational code is semantically shared . they find that alignment peaks in mid-to-late layers of both model types . |
| Outcome: | a forced-choice "Pick-a-Pic" task shows human preferences for image-caption matches are mirrored in embedding spaces across vision-language model pairs. |
AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on factual correctness, semantic grounding, visual reasoning, or multimodal large language models. |
| Approach: | They propose a benchmark to assess AICA, which integrates perception, reasoning, and generation into a unified framework. |
| Outcome: | The proposed framework corrects intensity errors and significantly enhances descriptive depth. |
Towards Artwork Explanation in Large-scale Vision Language Models (2024.acl-short)
Copied to clipboard
| Challenge: | Large-scale Vision-Language Models (LVLMs) output text from images and instructions, demonstrating advanced capabilities in text generation and comprehension. |
| Approach: | They propose to use artwork explanation generation task to quantitatively assess the understanding and utilization of artworks knowledge. |
| Outcome: | The proposed task evaluates the understanding and utilization of knowledge about artworks from images and titles and generates explanations using only images. |
CIVET: Systematic Evaluation of Understanding in VLMs (2025.findings-emnlp)
Copied to clipboard
Massimo Rizzoli, Simone Alghisi, Olha Khomyn, Gabriel Roccabruna, Seyed Mahed Mousavi, Giuseppe Riccardi
| Challenge: | Current Vision-Language Models can accurately recognize only a limited set of basic object properties; 3) they struggle to understand basic relations among objects. |
| Approach: | They propose a framework that evaluates VLMs on exhaustive sets of stimuli, free from annotation noise, dataset-specific biases, and uncontrolled scene complexity. |
| Outcome: | The proposed framework addresses the lack of standardized systematic evaluation for assessing VLMs’ understanding, enabling researchers to test hypotheses with statistical rigor. |
Concept-pedia: a Wide-coverage Semantically-annotated Multimodal Dataset (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current evaluations for Vision-language Models remain heavily anchored to ImageNet . |
| Approach: | They propose a large-scale semantically-annotated multimodal resource that extends the range of visual concepts, including diverse abstract categories. |
| Outcome: | The proposed model expands the range of visual concepts, including diverse abstract categories. |