Papers by John Canny
Distribution Aware Metrics for Conditional Natural Language Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing metrics for conditional natural language generation rely on pairwise comparisons between a single generated text and the best-matching reference. |
| Approach: | They propose a family of meta-metrics that build on existing pairwise distance functions to evaluate conditional natural language generation models. |
| Outcome: | The proposed method evaluates the ability of a model to generate text matching diversity in references in visual description and summarization. |
CLAIR: Evaluating Image Captions with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing measures for image caption evaluation fail to capture dimensions of similarity . a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. |
| Approach: | They propose a method that leverages the zero-shot language modeling capabilities of large language models to evaluate captions. |
| Outcome: | The proposed method shows a stronger correlation with human judgments of caption quality compared to other measures. |
ALOHa: A New Measure for Hallucination in Captioning Models (2024.naacl-short)
Copied to clipboard
Suzanne Petryk, David Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph Gonzalez, Trevor Darrell
| Challenge: | Existing metric for object hallucination, CHAIR, is limited to MS COCO objects and synonyms. |
| Approach: | They propose a new open-vocabulary metric, ALOHa, which leverages large language models to measure object hallucinations. |
| Outcome: | The proposed metric correctly identifies 13.6% more hallucinated objects than CHAIR on HAT and 30.8% more on nocaps. |
What’s The Latest? A Question-driven News Chatbot (2020.acl-demos)
Copied to clipboard
| Challenge: | a news chatbot that draws content from news articles creates conversations with a user about the news. |
| Approach: | They describe an automatic news chatbot that draws content from a diverse set of news sources and creates conversations with a user about the news. |
| Outcome: | The proposed system engages news readers in multi-turn conversations about specific stories. |
The Summary Loop: Learning to Write Abstractive Summaries Without Examples (2020.acl-main)
Copied to clipboard
| Challenge: | Unsupervised abstractive summarization is important for news headlines and research papers . a novel method that encourages the inclusion of key terms from the original document into the summary is presented . |
| Approach: | They propose a method that encourages the inclusion of key terms from the original document into the summary by a coverage model along with a fluency model. |
| Outcome: | The proposed method outperforms existing methods on news summarization datasets and is competitive with existing methods. |
Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactions (2025.findings-acl)
Copied to clipboard
Taedong Yun, Eric Yang, Mustafa Safdari, Jong Ha Lee, Vaishnavi Vinod Kumar, S. Sara Mahdavi, Jonathan Amar, Derek Peyton, Reut Aharony, Andreas Michaelides PhD, Logan Douglas Schneider, Isaac Galatzer-Levy, Yugang Jia, John Canny, Arthur Gretton, Maja Mataric
| Challenge: | Structured data is generated grounded in health and lifestyle factors and full profiles of synthetic users are developed conditioned on the structured data. |
| Approach: | They propose an end-to-end framework for generating synthetic users for evaluating interactive agents designed to encourage positive behavior changes, such as in health and lifestyle coaching. |
| Outcome: | The proposed framework is validated in the domains of sleep and diabetes coaching using two independently-developed agents for sleep and diabetic coaching as case studies. |
IC3: Image Captioning by Committee Consensus (2023.emnlp-main)
Copied to clipboard
| Challenge: | Traditionally, image captioning models are trained to generate a single “best’ (most like a reference) image caption. |
| Approach: | They propose a method to generate a single caption that captures high-level details from several annotator viewpoints. |
| Outcome: | The proposed method outperforms baseline SOTA models and improves the performance of automated recall systems by up to 84%. |
Moral Foundations of Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Moral foundations theory (MFT) is a psychological assessment tool that decomposes human moral reasoning into five factors, including care/harm, liberty/oppression, and sanctity/degradation. |
| Approach: | They propose to use moral foundations theory to analyze whether popular LLMs have acquired a bias towards a particular set of moral values. |
| Outcome: | The proposed model can be adversarially selected to exhibit a particular moral foundations and can affect downstream tasks. |