Papers by Vasudev Lal
NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge (2024.findings-naacl)
Copied to clipboard
| Challenge: | Comparative knowledge is an essential component of our world knowledge, yet understudied in prior literature. |
| Approach: | They propose a framework for comparative knowledge distillation overgenerated from language models . they use a corpus of 8.8M comparisons over 1.74M entity pairs to acquire comparative information . |
| Outcome: | The proposed framework acquires comparative knowledge between everyday objects . human evaluations show that it outperforms existing resources in terms of validity . |
ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning (2023.acl-long)
Copied to clipboard
Xiao Xu, Bei Li, Chenfei Wu, Shao-Yen Tseng, Anahita Bhiwandiwalla, Shachar Rosenman, Vasudev Lal, Wanxiang Che, Nan Duan
| Challenge: | Two-Tower Vision-Language models suffer from ineffective layer-by-layer utilization of uni-modal representations and cannot flexibly exploit different levels of unil-modal knowledge. |
| Approach: | They propose a model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels to facilitate more comprehensive cross-modal alignment and fusion. |
| Outcome: | The proposed model outperforms baselines with and without Vision-Language Pre-training (VLP) with 4M VLP data. |
NeuroPrompts: An Adaptive Framework to Optimize Prompts for Text-to-Image Generation (2024.eacl-demo)
Copied to clipboard
| Challenge: | Recent advances in text-to-image diffusion models have made it difficult to obtain high-quality images. |
| Approach: | They propose an adaptive framework that automatically enhances a user's prompt to improve the quality of generation models. |
| Outcome: | The proposed framework generates prompts similar to those produced by human prompt engineers and provides user control over stylistic features via constraint set specification. |
NeuroCounterfactuals: Beyond Minimal-Edit Counterfactuals for Richer Data Augmentation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to produce counterfactuals rely on small perturbations via minimal edits, resulting in simplistic changes. |
| Approach: | They propose a novel approach to produce counterfactuals that allow for larger edits and linguistic diversity while still bearing similarity to the original document. |
| Outcome: | The proposed approach outperforms existing methods for generalizing natural language models under select settings. |
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing vision-and-language pretraining approaches rely on external object detectors to encode images in a multi-modal transformer framework. |
| Approach: | They propose an object-aware end-to-end VLP framework which feeds image grid features from CNNs into the Transformer and learns the multi-modal representations jointly. |
| Outcome: | The proposed framework achieves competitive or superior performances on vision-language tasks. |
Probing Semantic Routing in Large Mixture-of-Expert Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | large mixture-of-expert models have become increasingly common in the open domain . prior work has explored functional differentiation through routing behavior . |
| Approach: | They investigate whether expert routing in large mixture-of-expert models is influenced by the semantics of the inputs. |
| Outcome: | The results show that expert routing is influenced by the semantics of the inputs. |
Why do LLaVA Vision-Language Models Reply to Images in English? (2024.findings-emnlp)
Copied to clipboard
Musashi Hinck, Carolin Holtermann, Matthew Olson, Florian Schneider, Sungduk Yu, Anahita Bhiwandiwalla, Anne Lauscher, Shao-Yen Tseng, Vasudev Lal
| Challenge: | Including an image in a multimodal query significantly increases the likelihood of the model returning an English response regardless of the language of the query. |
| Approach: | They propose a two-pronged approach that combines extensive ablation of the design space with a mechanistic analysis of the models’ internal representations of image and text inputs. |
| Outcome: | The proposed approach reduces the multilingual error by switching the language backbone for a bilingual language model. |
InterpreT: An Interactive Visualization Tool for Interpreting Transformers (2021.eacl-demos)
Copied to clipboard
Vasudev Lal, Arden Ma, Estelle Aflalo, Phillip Howard, Ana Simoes, Daniel Korat, Oren Pereg, Gadi Singer, Moshe Wasserblat
| Challenge: | Using Transformer-based models for NLU/NLP tasks is a growing interest . but there are many open questions regarding the behavior of these models . |
| Approach: | They present an interactive visualization tool for interpreting Transformer-based models. |
| Outcome: | The tool can track and visualize token embeddings through each layer of a Transformer, highlight distances between certain token embeds, and identify task-related functions of attention heads using new metrics. |
Pruning the Paradox: How CLIP’s Most Informative Heads Enhance Performance While Amplifying Bias (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale vision-language models such as CLIP have advanced state-of-the-art performance in vision tasks . however, as they gain prominence in real-world applications, their embedded social biases can be harmful . et al., 2021: 103-104. |
| Approach: | They propose an interpretability metric that measures how consistently attention heads align with specific concepts in CLIP-like models. |
| Outcome: | The proposed interpretability metric measures how consistently attention heads align with specific concepts. |
LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression (2025.findings-naacl)
Copied to clipboard
Souvik Kundu, Anahita Bhiwandiwalla, Sungduk Yu, Phillip Howard, Tiep Le, Sharath Nittur Sridhar, David Cobbley, Hao Kang, Vasudev Lal
| Challenge: | LVLMs have been shown to perform well on simple uni-modal benchmarks, but their detailed study on multi-modal models is still lacking. |
| Approach: | They propose a framework to analyze the impact of compression on LVLMs on multi-modal input driven tasks. |
| Outcome: | The proposed framework analyzes the impact of compression on generative performance of large vision language models on multi-modal input driven tasks. |