Papers by William Rudman
What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have gained prominence due to their success in solving complex cross-modal tasks. |
| Approach: | They propose a Gaussian-Noise-free pipeline for mechanistic interpretability in VLMs that introduces Semantic Image Pairs corruption, the first visual counterpart to Symmetric Token Replacement for text. |
| Outcome: | The proposed pipeline identifies a set of “universal attention heads” in BLIP and LLaVA that consistently contribute across different tasks and modalities. |
Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts (2025.emnlp-main)
Copied to clipboard
Michal Golovanevsky, William Rudman, Michael A. Lepori, Amir Bar, Ritambhara Singh, Carsten Eickhoff
| Challenge: | Multimodal Large Language Models perform well on visual question answering tasks, but it remains unclear whether their reasoning relies more on memorized world knowledge or on visual information present in the input image. |
| Approach: | They propose a dataset of visual-realistic counterfactuals that put world knowledge priors into conflict with visual input. |
| Outcome: | The proposed dataset puts world knowledge priors into conflict with visual input . it shows that model predictions shift toward visual evidence in mid-to-late layers . |
Mechanisms of Prompt-Induced Hallucination in Vision–Language Models (2026.acl-long)
Copied to clipboard
William Rudman, Michal Golovanevsky, Dana Arad, Yonatan Belinkov, Carsten Eickhoff, Ritambhara Singh, Kyle Mahowald
| Challenge: | Large vision–language models (VLMs) often hallucinate by favoring textual prompts over visual evidence. |
| Approach: | They study the failure mode of large vision–language models by focusing on textual prompts over visual evidence. |
| Outcome: | The proposed model overestimates the number of objects in an image . it hallucinates additional waterlilies when asked to describe a mismatched number of items . the model ablation reduces prompt-induced hallucinosities by at least 40% without additional training . |
Outlier Dimensions Encode Task Specific Knowledge (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that fine-tuning outlier dimensions is detrimental to the representational quality of embeddings. |
| Approach: | They investigate how fine-tuning impacts outlier dimensions by testing their hypothesis that a single outlier dimension can complete downstream tasks with a minimal error rate. |
| Outcome: | The proposed model can encode crucial task-specific knowledge and the value of a representation in a single outlier dimension drives downstream model decisions. |
Forgotten Polygons: Multimodal Large Language Models are Shape-Blind (2025.findings-acl)
Copied to clipboard
William Rudman, Michal Golovanevsky, Amir Bar, Vedant Palit, Yann LeCun, Carsten Eickhoff, Ritambhara Singh
| Challenge: | Multimodal Large Language Models struggle with visual reasoning, despite strong performance on vision-language tasks. |
| Approach: | They propose a visually cued chain-of-thought prompting that enhances multi-step mathematical reasoning by explicitly referencing visual annotations in diagrams. |
| Outcome: | The proposed model improves GPT-4o's accuracy on an irregular polygon side-counting task from 7% to 93%. |
IsoScore: Measuring the Uniformity of Embedding Space Utilization (2022.findings-acl)
Copied to clipboard
| Challenge: | Several studies suggest that contextualized word embedding models do not isotropically project tokens into vector space. |
| Approach: | They propose to use a tool to measure isotropy to quantify the degree to which a point cloud uniformly utilizes the ambient vector space. |
| Outcome: | The proposed tool is the only available tool that accurately measures how uniformly distributed variance is across dimensions in vector space. |