Papers by David Chan
Distribution Aware Metrics for Conditional Natural Language Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing metrics for conditional natural language generation rely on pairwise comparisons between a single generated text and the best-matching reference. |
| Approach: | They propose a family of meta-metrics that build on existing pairwise distance functions to evaluate conditional natural language generation models. |
| Outcome: | The proposed method evaluates the ability of a model to generate text matching diversity in references in visual description and summarization. |
CLAIR: Evaluating Image Captions with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing measures for image caption evaluation fail to capture dimensions of similarity . a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. |
| Approach: | They propose a method that leverages the zero-shot language modeling capabilities of large language models to evaluate captions. |
| Outcome: | The proposed method shows a stronger correlation with human judgments of caption quality compared to other measures. |
ALOHa: A New Measure for Hallucination in Captioning Models (2024.naacl-short)
Copied to clipboard
Suzanne Petryk, David Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph Gonzalez, Trevor Darrell
| Challenge: | Existing metric for object hallucination, CHAIR, is limited to MS COCO objects and synonyms. |
| Approach: | They propose a new open-vocabulary metric, ALOHa, which leverages large language models to measure object hallucinations. |
| Outcome: | The proposed metric correctly identifies 13.6% more hallucinated objects than CHAIR on HAT and 30.8% more on nocaps. |
Understanding the Behaviors of Environment-aware Information Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Recent retrieval-augmented generation approaches have demonstrated strong capability in handling complex queries. |
| Approach: | They propose a branching-based rollout technique that improves training stability . they find different retrievers exhibit distinct optimal query styles . |
| Outcome: | The proposed method improves training stability and improves retrieval-aware systems. |
Virtual Personas for Language Models via an Anthology of Backstories (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are trained from vast repositories of text authored by millions of distinct authors, reflecting an enormous diversity of human traits. |
| Approach: | They propose a method for conditioning LLMs to particular virtual personas by harnessing open-ended life narratives, which they refer to as backstories, and demonstrate that it improves consistency and reliability of experimental outcomes. |
| Outcome: | The proposed method improves consistency and reliability of experimental outcomes while ensuring better representation of diverse sub-populations. |
IC3: Image Captioning by Committee Consensus (2023.emnlp-main)
Copied to clipboard
| Challenge: | Traditionally, image captioning models are trained to generate a single “best’ (most like a reference) image caption. |
| Approach: | They propose a method to generate a single caption that captures high-level details from several annotator viewpoints. |
| Outcome: | The proposed method outperforms baseline SOTA models and improves the performance of automated recall systems by up to 84%. |