Papers by Trevor Darrell
Do What? Teaching Vision-Language-Action Models to Reject the Impossible (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that VLAs can recognize, interpret, and respond to false-premise instructions. |
| Approach: | They propose a framework that detects when an instruction cannot be executed due to a false premise and engages in language-based clarification or correction. |
| Outcome: | The proposed framework detects when an instruction cannot be executed due to a false premise and engages in language-based clarification or correction. |
From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating insightful explanations with limited annotations are limited. |
| Approach: | They propose a method that iteratively computes visual features, an answer, and an explanation to improve the explanation quality step by step until the answer converges. |
| Outcome: | The proposed method outperforms previous methods while utilizing 5% of the human-annotated explanations across 10 metrics, showing up to 4.2 and 1.3 increases in BLEU-1 score on the VCR and VQA-X datasets. |
Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs (2023.emnlp-main)
Copied to clipboard
Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, Amir Globerson
| Challenge: | Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS) performance in a variety of tasks. |
| Approach: | They propose to integrate structured annotations into visual and textual representations to improve VLMs' understanding of compositional scenes. |
| Outcome: | The proposed method improves VLMs on multiple VL datasets with only a mild degradation in ZS capabilities. |
Modular Networks for Compositional Instruction Following (2021.naacl-main)
Copied to clipboard
| Challenge: | Standard instruction following models struggle on novel compositions of subgoals observed during training. |
| Approach: | They propose a modular architecture that follows natural language instructions that describe sequences of diverse subgoals. |
| Outcome: | The proposed architecture improves generalization to novel subgoals and environments unseen in training. |
Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation (2022.naacl-main)
Copied to clipboard
| Challenge: | Detecting out-of-context media is a problem in domains of public significance . a method that leverages automatically generated hard image-text mismatches is proposed . |
| Approach: | They propose a method that leverages automatically generated hard image-text mismatches to detect out-of-context media . they analyze tweets relevant to topics such as COVID-19, Climate Change and Military Vehicles . |
| Outcome: | The proposed method improves detection accuracy over a strong baseline on a set of fakes created by humans. |
Modular Visual Question Answering via Code Generation (2023.acl-short)
Copied to clipboard
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, Dan Klein
| Challenge: | a framework for visual question answering is based on modular code generation . the scope of reasoning needed for visual questions is vast, and requires many skills . |
| Approach: | They propose a framework that formulates visual question answering as modular code generation. |
| Outcome: | The proposed framework improves accuracy on COVR and GQA datasets by 3% and 2% compared to the few-shot baseline that does not employ code generation. |
G3: Geolocation via Guidebook Grounding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a new task uses explicit knowledge from human-written guidebooks to improve geolocation accuracy . a state-of-the-art image-only method is unable to predict the location of an image . |
| Approach: | They propose a task that uses streetview images and a guidebook to predict a country for each image . they add clues from the guidebook and supervise attention with country-level pseudo labels . |
| Outcome: | The proposed method outperforms state-of-the-art image-only geolocation methods with 5% improvement in Top-1 accuracy. |
CLAIR: Evaluating Image Captions with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing measures for image caption evaluation fail to capture dimensions of similarity . a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. |
| Approach: | They propose a method that leverages the zero-shot language modeling capabilities of large language models to evaluate captions. |
| Outcome: | The proposed method shows a stronger correlation with human judgments of caption quality compared to other measures. |
Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for identifying object referents of language expressions consider target and distractor objects independently and pool multiple views before grounding. |
| Approach: | They propose a model that selects an object referent based on language that distinguishes between two similar objects and a multi-view approach to grounding in context model which reduces the relative error by 12.9% . |
| Outcome: | The proposed model improves on the SNARE object reference task with a relative error reduction of 12.9% and an absolute improvement of 2.7%. |
ALOHa: A New Measure for Hallucination in Captioning Models (2024.naacl-short)
Copied to clipboard
Suzanne Petryk, David Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph Gonzalez, Trevor Darrell
| Challenge: | Existing metric for object hallucination, CHAIR, is limited to MS COCO objects and synonyms. |
| Approach: | They propose a new open-vocabulary metric, ALOHa, which leverages large language models to measure object hallucinations. |
| Outcome: | The proposed metric correctly identifies 13.6% more hallucinated objects than CHAIR on HAT and 30.8% more on nocaps. |
ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension (2022.acl-long)
Copied to clipboard
| Challenge: | Visual referring expression comprehension (ReC) models can be trained for a domain, but it remains unclear if they can be applied in a zero-shot manner to more complex tasks like ReC. |
| Approach: | They propose a method that repurposes CLIP, a state-of-the-art large-scale model, for training a referring expression comprehension model for a new visual domain. |
| Outcome: | The proposed model reduces the gap between zero-shot baselines from prior work and supervised models by as much as 29% on RefCOCOg, and on ReFGTA (video game imagery), and its relative improvement over supervised ReC models is 8%. |
Re-evaluating the Need for Visual Signals in Unsupervised Grammar Induction (2024.findings-naacl)
Copied to clipboard
Boyi Li, Rodolfo Corona, Karttikeya Mangalam, Catherine Chen, Daniel Flaherty, Serge Belongie, Kilian Weinberger, Jitendra Malik, Trevor Darrell, Dan Klein
| Challenge: | Recent studies show multimodal inputs can improve grammar induction, but weak textual baselines are needed for training. |
| Approach: | They use a fixed grammar family to compare multimodal grammar induction methods . they find multimodal inputs can improve grammar induction by grounding textual inputs to the visual world . |
| Outcome: | The proposed model outperforms weaker baselines on four benchmark datasets. |
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation (P19-1)
Copied to clipboard
| Challenge: | Existing models that ground language into visual appearance and route structure are outperforming their visual counterparts in unseen new environments. |
| Approach: | They propose to decompose the grounding procedure into a set of expert models with access to different modalities and ensemble them at prediction time. |
| Outcome: | The proposed model outperforms models with only route structure and visual features on the benchmark Room-to-Room dataset. |
Activation Reward Models for Few-Shot Model Alignment (2026.findings-acl)
Copied to clipboard
Tianning Chai, Chancharik Mitra, Brandon Huang, Gautam Rajendrakumar Gare, Zhiqiu Lin, Assaf Arbelle, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, Deva Ramanan, Roei Herzig
| Challenge: | A common approach is to use reward models that enable reinforcement-learning post-training. |
| Approach: | They propose a method that steers LLM activations to align with few-shot preference data without finetuning. |
| Outcome: | The proposed method surpasses zero-shot, few-shot and voting-based benchmarks on reward hacking and noise signals. |
Scaling Vision-Language Models with Sparse Mixture of Experts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study explores the effectiveness of mixture-of-experts (MoE) techniques in scaling vision-language models . alayrac and colleagues demonstrate the effectiveness and performance of MoE in scaling VLMs . |
| Approach: | They propose to use sparsely-gated mixture-of-experts techniques to scale vision-language models . they show that MoE can achieve state-of the-art performance over dense models a range of benchmarks . |
| Outcome: | The proposed approach achieves state-of-the-art performance over dense models of equivalent computational cost. |
TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods that can find relevant information, extract it, and answer video questions in a single pass are not able to adapt if insufficient or incorrect information is collected. |
| Approach: | They propose a modular multi-LMM agent framework that can find relevant information, extract it, and answer the question simultaneously. |
| Outcome: | The proposed model improves performance on several VideoQA benchmarks without fine-tuning on specific datasets. |
Puzzled by Puzzles: When Vision-Language Models Can’t Take a Hint (2025.emnlp-main)
Copied to clipboard
| Challenge: | rebus puzzles encode language through imagery, spatial arrangement, and symbolic substitution. |
| Approach: | They construct a benchmark of rebus puzzles in english language to test their ability to interpret and solve them. |
| Outcome: | The proposed model performs well on a set of english-language rebus puzzles. |
Exposing the Limits of Video-Text Models through Contrast Sets (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent video-text models can retrieve relevant videos based on text with high accuracy, but to what extent do they comprehend the semantics of the text? |
| Approach: | They propose a framework that probes video-text models with hard negatives . they leverage a pre-trained language model and a set of heuristics to create verb and person entity focused contrast sets. |
| Outcome: | The proposed framework erases the performance gap between CLIP-based methods and the earlier methods. |
Enough Coin Flips Can Make LLMs Act Bayesian (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models exhibit the ability to generalize given few-shot examples in their input prompt, an emergent capability known as in-context learning. |
| Approach: | They investigate whether large language models use in-context learning to generalize given few-shot examples in their input prompt. |
| Outcome: | The proposed model can generalize given few-shot examples in their input prompt, an emergent capability known as in-context learning. |
Aligning Large Multimodal Models with Factually Augmented RLHF (2024.findings-acl)
Copied to clipboard
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell
| Challenge: | Large Multimodal Models (LMMs) are built across modalities and the misalignment between two modality can result in "hallucination" . developing LMMs faces challenges such as a lack of data and a limited number of data sets. |
| Approach: | They propose a new algorithm that augments the reward model with additional factual information such as image captions and ground-truth multi-choice options. |
| Outcome: | The proposed approach improves on the LLaVA-Bench dataset with the 96% performance level of the text-only GPT-4 and an improvement of 60% on MMHAL-BENCH over other baselines. |
Localizing Moments in Video with Temporal Language (D18-1)
Copied to clipboard
| Challenge: | a novel model for localizing moments in a longer video using natural language queries is challenging . moment localization is similar to other language and vision tasks, but it offers an interesting opportunity to model temporal dependencies and reasoning in text. |
| Approach: | They propose a model that explicitly reasons about different temporal segments in a video . their dataset includes a dataset with real videos and template sentences . |
| Outcome: | The proposed model explicitly reasons about different temporal segments in a video . it shows that temporal context is important for localizing phrases which include temporal language . |
Object Hallucination in Image Captioning (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning metrics do not capture image relevance . current metrics only measure similarity to ground truth captions . |
| Approach: | They propose a new image relevance metric to evaluate captioning models with veridical visual labels and assess their rate of object hallucination. |
| Outcome: | The proposed metrics show that models with veridical visual labels have higher hallucination rates than models with lower hallucinosity. |
Voxel-informed Language Grounding (2022.acl-short)
Copied to clipboard
| Challenge: | Embodied robotic agents can be used to ground objects using 3D geometry . despite typically being paired with 2D images, natural language describes a fundamentally 3D world . |
| Approach: | They propose a model that leverages 3D geometric information to ground natural language . they show that VLG significantly improves grounding accuracy on SNARE . |
| Outcome: | The proposed model significantly improves grounding accuracy on SNARE, an object reference game task. |
NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media (2021.emnlp-main)
Copied to clipboard
| Challenge: | a threat scenario where an image is used out of context to support a narrative is proposed. |
| Approach: | They propose a dataset where both image and text are unmanipulated but mismatched . they benchmark several state-of-the-art multimodal models on their dataset . |
| Outcome: | The proposed dataset shows that machine-driven image repurposing is now a realistic threat . it provides samples that represent challenging instances of mismatch between text and image . |
Disentangled Action Recognition with Knowledge Bases (2022.naacl-main)
Copied to clipboard
| Challenge: | a new method for compositional action recognition is proposed to address the problem of zero-shot learning. |
| Approach: | They propose a method to generalize compositional action recognition models to new verbs and nouns . they use knowledge graphs to extract disentangled feature representations for verbs, noun and type constraint . |
| Outcome: | The proposed approach improves generalization ability of the compositional action recognition model to novel verbs and nouns that are unseen during training time. |