LLMs Can Compensate for Deficiencies in Visual Representations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a strong language backbone in vision-language models compensates for weak visual features by contextualizing or enriching them. |
| Approach: | They investigate whether strong language backbone compensates for weak visual features . they use CLIP-based vision encoders to perform controlled self-attention ablations . |
| Outcome: | The proposed model compensates for weak visual features by contextualizing or enriching them. |
Similar Papers
Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that CLIP models struggle with visual reasoning tasks . despite the success of Contrastive Language-Image Pretraining, there are still limitations . |
| Approach: | They propose to use a visual encoder to train CLIP-like models for fine-grained visual reasoning tasks. |
| Outcome: | The proposed models outperform CLIP-like encoders in visual reasoning tasks . the study highlights the importance of VLM architectural choices . |
Advancing Vision-Language Models with Adapter Ensemble Strategies (2024.findings-emnlp)
Copied to clipboard
| Challenge: | CLIP revolutes vision-language pretraining by using contrastive learning on paired web data. |
| Approach: | They propose to combine a "adapter ensemble" with traditional machine learning techniques to augment large-scale pretrained vision-language models. |
| Outcome: | The proposed model outperforms baselines and derives improvement when the number of ensemble parameters increases. |
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing fine-tuning approaches for compositional understanding compromise performance in zero-shot multi-modal tasks. |
| Approach: | They propose a method to enhance compositional understanding in pre-trained vision and language models without sacrificing performance in zero-shot multi-modal tasks. |
| Outcome: | The proposed method achieves compositionality on par with state-of-the-art models and retains strong multi-modal capabilities. |
Preserving Language Capabilities in Vision-Language Models via Representation Regulation (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) provide a unified framework to process both text-only and vision-language tasks. |
| Approach: | They propose a method to reduce the distance between visual and textual representations by introducing a Representation Distribution Difference (RDD) loss. |
| Outcome: | Empirical evidence shows that finetuning VLMs on vision-language data has degraded language capabilities. |
Enhancing Advanced Visual Reasoning Ability of Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Vision-Language (VL) research have sparked new benchmarks for complex visual reasoning, challenging models’ advanced reasoning ability. |
| Approach: | They propose a novel multi-modal in-context learning methodology to enhance LLMs’ contextual understanding and reasoning. |
| Outcome: | The proposed model achieves SOTA performance among all visual reasoning tasks and achieves a 'higher level of accuracy' than previous models. |
RWKV-CLIP: A Robust Vision-Language Representation Learner (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using large image-text datasets, large-scale image-data sets have been used for visionlanguage pre-training. |
| Approach: | They propose a framework that leverages Large Language Models to combine and refine information from web-based image-text pairs, synthetic captions, and detection tags. |
| Outcome: | The proposed framework can combine and refine information from web-based image-text pairs, synthetic captions, and detection tags. |
Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models (2025.findings-acl)
Copied to clipboard
Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluis Marquez, Miguel Ballesteros, Yassine Benajiba
| Challenge: | LLaVA-7B demonstrated a decline in safety alignment ability on multi-modal inputs compared to its LLM backbone. |
| Approach: | They propose a method to recover alignment ability from LLM backbone while preserving functional capabilities of VLMs. |
| Outcome: | The proposed framework recovers alignment ability that is inherent in the LLM backbone with minimal impact on fluency and linguistic capabilities of pre-trained VLMs. |
Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Vision-language Models have been shown to be highly capable but lacking basic visual understanding skills. |
| Approach: | They propose to examine the limitations of vision-language models on visual tasks by constructing a series of tests that probe which components of design may be lacking. |
| Outcome: | The proposed tests compare VLMs to other models on visual encoders, intermediate vision-language projection and LLM-decoder outputs. |
Can VLMs Actually See and Read? A Survey on Modality Collapse in Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models integrate textual and visual information, enabling them to process visual inputs and generate predictions. |
| Approach: | They review work on modality collapse analysis to provide insights into the reason for this unintended behavior and review probing studies for fine-grained vision-language understanding. |
| Outcome: | The proposed models can achieve competitive performance in vision-language tasks despite relying heavily on textual information and ignoring visual information. |
Text encoders bottleneck compositionality in contrastive vision-language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal models are often unable to reason about simple spatial relations or attribute attachments. |
| Approach: | They first curate CompPrompts, a set of increasingly compositional image captions that VL models should be able to capture . then train text-only recovery probes that aim to reconstruct captions from single-vector text representations produced by several VL model. |
| Outcome: | The proposed model can reconstruct captions from single-vector text representations produced by several models on a broader range of scenes compared to previous models. |