Papers by Ranjay Krishna
DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback (2025.naacl-long)
Copied to clipboard
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd Van Steenkiste, Ranjay Krishna, Cyrus Rashtchian
| Challenge: | Text-to-Image models (T2I) still struggle to produce images that are both aesthetically pleasing and faithful to the user’s input text. |
| Approach: | They propose a training algorithm that trains T2I models to be faithful to the input text. |
| Outcome: | The proposed model improves both the semantic alignment and aesthetic appeal of two diffusion-based T2I models, evidenced by multiple benchmarks (+1.7% on TIFA, +2.9% on DSG1K, +3.4% on VILA aesthetic). |
Found in the middle: Calibrating Positional Attention Bias Improves Long Context Utilization (2024.findings-acl)
Copied to clipboard
Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, Tomas Pfister
| Challenge: | Large language models struggle to capture relevant information located in the middle of their input. |
| Approach: | They propose a calibration mechanism that allows the model to attend to contexts faithfully according to their relevance even when they are in the middle. |
| Outcome: | The proposed calibration mechanism mitigates this positional bias and improves retrieval-augmented generation performance. |
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023.findings-acl)
Copied to clipboard
Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister
| Challenge: | Deploying large language models (LLMs) is difficult because they are memory inefficient and compute-intensive for practical applications. |
| Approach: | They propose a mechanism that fine tunes or distills small models that outperform LLMs . they use human labels to fine tune models or LLM-generated labels to train models . |
| Outcome: | The proposed method outperforms LLMs by using fewer training examples compared to few-shot prompted models using substantially smaller model sizes. |
Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Currently, language-equipped vision systems such as VizWiz, TapTapSee, BeMyEyes, and CamFind are actively being deployed across a broad spectrum of users. |
| Approach: | They propose to identify collective outliers in active learning methods that are hard and often impossible for models to learn . they also propose to use visual inputs to identify these outlier examples as examples assigned low model confidence and prediction variability during training. |
| Outcome: | The proposed methods outperform random selection on visual question answering tasks. |
Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning (2024.emnlp-main)
Copied to clipboard
Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, Shiwei Liu
| Challenge: | Existing approaches to prune LLMs rely on the C4 dataset as calibration data . arithmetic datasets perform better than pre-training datasets for pruning, whereas chain-of-thought is only useful on certain tasks. |
| Approach: | They evaluate the selection of calibration data for LLM pruning across a wide range of datasets . they find that C4 is not the optimal calibration data, and that CoT is only useful on certain tasks. |
| Outcome: | The chosen calibration data significantly impacts the performance of pruned LLMs, the authors found . their results shed light on the importance of carefully selecting calibration data for LLM pruning . |
LATTE: Learning to Think with Vision Specialists (2025.emnlp-main)
Copied to clipboard
Zixian Ma, Jianguo Zhang, Zhiwei Liu, Jieyu Zhang, Juntao Tan, Manli Shu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Caiming Xiong, Ranjay Krishna, Silvio Savarese
| Challenge: | Open-source vision-language models excel on simple question-answering tasks, but struggle with complex questions that require both perception and reasoning. |
| Approach: | They propose a family of vision-language models that have LeArned to Think wiTh vision spEcialists by offloading perception to state-of-the-art vision models. |
| Outcome: | The proposed model achieves 4-5% gains over baselines across 6 benchmarks covering both perception and reasoning abilities. |
Wait, We Don’t Need to “Wait”! Removing Thinking Tokens Improves Reasoning Efficiency (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large reasoning models often introduce significant overthinking . this leads to verbose and redundant outputs that hinder efficiency. |
| Approach: | They propose a plug-and-play solution that disables explicit self-reflection . it suppresses tokens such as "Wait" and "Hmm" during inference . |
| Outcome: | The proposed approach reduces chain-of-thought trajectory length by up to 27%–51% in five R1-style model series without compromising model utility. |
ImageInWords: Unlocking Hyper-Detailed Image Descriptions (2024.emnlp-main)
Copied to clipboard
Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, Radu Soricut
| Challenge: | generating accurate hyper-detailed image descriptions is challenging for vision-language models trained on web-scraped image-text. |
| Approach: | They propose a data-centric framework for generating hyper-detailed image descriptions using web-scraped image-text. |
| Outcome: | The proposed framework improves on human evaluations on the data, even with only 9k samples. |
Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation (2025.acl-long)
Copied to clipboard
Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, Christopher Clark
| Challenge: | Vision-language models struggle to understand text-rich images due to the scarcity of diverse text-only large language data. |
| Approach: | They propose a framework that leverages the coding capabilities of text-only large language models to create synthetic text-rich multimodal data. |
| Outcome: | The proposed framework can generate high-quality instruction-tuning data using Python, HTML, LaTeX and other languages. |
Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps (2024.emnlp-main)
Copied to clipboard
| Challenge: | Despite the utility and impressive capabilities of large language models, their tendency to generate hallucinations presents a significant challenge in their deployment. |
| Approach: | They propose a simple hallucination detection model based on the ratio of attention weights on the context versus newly generated tokens. |
| Outcome: | The proposed model reduces the amount of hallucinations by 9.6% in a summarization task. |