Papers with ViLT
ColorFoil: Investigating Color Blindness in Large Vision and Language Models (2025.naacl-srw)
Copied to clipboard
| Challenge: | Several studies indicate a lack of robustness of the models when dealing with complex linguistics and visual attributes. |
| Approach: | They propose a new V&L benchmark by creating color-related foils to assess the models’ perception ability to detect colors like red, white, green, etc. |
| Outcome: | The proposed benchmark evaluates seven state-of-the-art V&L models including CLIP, ViLT, GroupViT, and BridgeTower in a zero-shot setting and demonstrates that they have better color perception capabilities than CLIP and its variants and GroupVit. |
Multi-Modal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision–Language Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in self-supervised training have led to a new class of pretrained vision–language models. |
| Approach: | They propose a visual and textual bias benchmark to assess bias in self-supervised multimodal models using 3,800 images and phrases from 14 population subgroups. |
| Outcome: | The proposed model shows that it favors certain groups while maintaining the accuracy of the model. |
NLKI: A Lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Small vision-language models lag behind their larger generative counterparts due to lack of knowledge. |
| Approach: | They propose a framework that integrates commonsense knowledge into small vision-language models . the framework retrieves natural language facts and prompts an LLM to craft natural language explanations . |
| Outcome: | The proposed framework retrieves natural language facts and prompts an LLM to craft natural language explanations. |