| Challenge: | Existing studies have demonstrated that vision-language models can retain and generalize knowledge, but they do not measure their ability to retain it. |
| Approach: | They build an automatic pipeline to derive a knowledge resource for calibrating and probing vision-language models. |
| Outcome: | The proposed model outperforms the pretrained model on size and spatial tasks. |
Similar Papers
ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for visual commonsense reasoning (VCR) use pre-trained large language models and pre-training visionlanguage models. |
| Approach: | They propose a collaborative approach where pre-trained LLMs serve as problem classifiers to analyze problem category and either use VLMs to answer directly or actively instruct LLM to gather relevant visual elements to support potential commonsense inferences. |
| Outcome: | The proposed approach outperforms all other methods without in-domain fine-tuning on two VCR benchmark datasets. |
Can Language Models Understand Physical Concepts? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models do not understand basic physical concepts in the human world. |
| Approach: | They propose a method to transfer embodied knowledge from visual models to LMs . they use visual concepts and embodies concepts learned from interaction with the world . |
| Outcome: | The proposed method achieves comparable performance with scaling up parameters of LMs 134. |
Are Visual-Linguistic Models Commonsense Knowledge Bases? (2022.coling-1)
Copied to clipboard
| Challenge: | PTLMs are used to extract knowledge from text on demand. |
| Approach: | They compare visual-linguistic and language-only visual-language models in a zero-shot commonsense question answering inference task. |
| Outcome: | The proposed models are highly promising on certain types of commonsense knowledge associated with the visual world. |
Can VLMs Actually See and Read? A Survey on Modality Collapse in Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models integrate textual and visual information, enabling them to process visual inputs and generate predictions. |
| Approach: | They review work on modality collapse analysis to provide insights into the reason for this unintended behavior and review probing studies for fine-grained vision-language understanding. |
| Outcome: | The proposed models can achieve competitive performance in vision-language tasks despite relying heavily on textual information and ignoring visual information. |
Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Vision-language Models have been shown to be highly capable but lacking basic visual understanding skills. |
| Approach: | They propose to examine the limitations of vision-language models on visual tasks by constructing a series of tests that probe which components of design may be lacking. |
| Outcome: | The proposed tests compare VLMs to other models on visual encoders, intermediate vision-language projection and LLM-decoder outputs. |
Enhancing Advanced Visual Reasoning Ability of Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Vision-Language (VL) research have sparked new benchmarks for complex visual reasoning, challenging models’ advanced reasoning ability. |
| Approach: | They propose a novel multi-modal in-context learning methodology to enhance LLMs’ contextual understanding and reasoning. |
| Outcome: | The proposed model achieves SOTA performance among all visual reasoning tasks and achieves a 'higher level of accuracy' than previous models. |
NLKI: A Lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Small vision-language models lag behind their larger generative counterparts due to lack of knowledge. |
| Approach: | They propose a framework that integrates commonsense knowledge into small vision-language models . the framework retrieves natural language facts and prompts an LLM to craft natural language explanations . |
| Outcome: | The proposed framework retrieves natural language facts and prompts an LLM to craft natural language explanations. |
Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) demonstrate impressive reasoning ability and the maintenance of world knowledge in natural language tasks. |
| Approach: | They propose a framework that enables LLMs to ask relevant questions to uncover more details in the image, along with filters for refining the generated information. |
| Outcome: | The proposed framework boosts the performance of baseline methods by 2.15% on OK-VQA and achieves consistent improvements across different LLMs. |
SemVink: Advancing VLMs’ Semantic Understanding of Optical Illusions via Visual Global Thinking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Vision-language models excel in semantic tasks but fail at detecting hidden content . current architectures prioritize abstract reasoning over low-level visual operations . |
| Approach: | They propose a benchmark to test vision-language models that can detect hidden content . they propose HC-Bench to scale images to low resolutions to unlock 99% accuracy . |
| Outcome: | HC-Bench shows that leading VLMs achieve near-zero accuracy even with explicit prompting . et al.: current models prioritize abstract reasoning over low-level visual operations . they urge a shift toward hybrid models bridging gap between computational vision and human cognition . |
Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Despite significant progress in multimodal language models, it remains unclear whether visual grounding enhances their understanding of embodied knowledge compared to text-only models. |
| Approach: | They propose to assess vision-language models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |
| Outcome: | The proposed benchmark assesses the models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |