Papers by Vedant Palit
What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have gained prominence due to their success in solving complex cross-modal tasks. |
| Approach: | They propose a Gaussian-Noise-free pipeline for mechanistic interpretability in VLMs that introduces Semantic Image Pairs corruption, the first visual counterpart to Symmetric Token Replacement for text. |
| Outcome: | The proposed pipeline identifies a set of “universal attention heads” in BLIP and LLaVA that consistently contribute across different tasks and modalities. |
Forgotten Polygons: Multimodal Large Language Models are Shape-Blind (2025.findings-acl)
Copied to clipboard
William Rudman, Michal Golovanevsky, Amir Bar, Vedant Palit, Yann LeCun, Carsten Eickhoff, Ritambhara Singh
| Challenge: | Multimodal Large Language Models struggle with visual reasoning, despite strong performance on vision-language tasks. |
| Approach: | They propose a visually cued chain-of-thought prompting that enhances multi-step mathematical reasoning by explicitly referencing visual annotations in diagrams. |
| Outcome: | The proposed model improves GPT-4o's accuracy on an irregular polygon side-counting task from 7% to 93%. |