Papers by Jack Hessel
Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Effective interlocutors account for the uncertain goals, beliefs, and emotions of others. |
| Approach: | They propose to calibrate language models to better represent outcome uncertainty . they propose to use two methods to calibrated small open-source models . |
| Outcome: | The proposed fine-tuning strategies can calibrate smaller open-source models to beat pre-trained models 10x their size. |
Text encoders bottleneck compositionality in contrastive vision-language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal models are often unable to reason about simple spatial relations or attribute attachments. |
| Approach: | They first curate CompPrompts, a set of increasingly compositional image captions that VL models should be able to capture . then train text-only recovery probes that aim to reconstruct captions from single-vector text representations produced by several VL model. |
| Outcome: | The proposed model can reconstruct captions from single-vector text representations produced by several models on a broader range of scenes compared to previous models. |
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube (2020.emnlp-main)
Copied to clipboard
| Challenge: | a representative pretraining model is fit to a diverse YouTube8M dataset . a priori, this domain is relatively easy for instructional videos . |
| Approach: | They fit a representative pretraining model to a YouTube8M dataset and examine its success and failure cases. |
| Outcome: | The proposed model can be trained on more diverse video corpora and achieve high performance on many video understanding tasks. |
Something’s Brewing! Early Prediction of Controversy-causing Posts from Discussion Features (N19-1)
Copied to clipboard
| Challenge: | Using data from several different communities on reddit.com, we predict the ultimate controversiality of posts. |
| Approach: | They analyze reddit.com data to predict the ultimate controversiality of posts . they use textual content and tree structure of early comments to predict content . |
| Outcome: | The proposed model predicts the ultimate controversiality of posts using features drawn from textual content and tree structure of early comments. |
Selective “Selective Prediction”: Reducing Unnecessary Abstention in Vision-Language Reasoning (2024.findings-acl)
Copied to clipboard
Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, Khyathi Chandu
| Challenge: | ReCoVERR reduces the over-abstention of a vision-language system with low tolerance for inaccurate predictions without increasing the error rate of the system’s predictions. |
| Approach: | They propose an inference-time algorithm to reduce the over-abstention of a selective vision-language system without increasing the error rate of the system’s predictions. |
| Outcome: | ReCoVERR reduces the over-abstention of a vision-language system without increasing the error rate of the system’s predictions. |
How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models (2024.findings-emnlp)
Copied to clipboard
Jaeyoung Lee, Ximing Lu, Jack Hessel, Faeze Brahman, Youngjae Yu, Yonatan Bisk, Yejin Choi, Saadia Gabriel
| Challenge: | a growing influx of misinformation across news and social media is hampered by outdated foundation model training data. |
| Approach: | They propose to use large language models to scale up online policing mechanisms . they evaluate foundation model performance without continual updating . |
| Outcome: | The proposed model can improve performance without continual updating . the proposed model improves on two widely used benchmarks . |
Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Visual arguments rely on images to persuade viewers to do or believe something . |
| Approach: | They propose three tasks for evaluating visual argument understanding . they use visual premises, commonsense premises and reasoning trees to analyze visual arguments . |
| Outcome: | The proposed tasks evaluate visual argument understanding using a dataset of 1,611 images annotated with 5,112 visual premises (with regions), 5,574 commonsense premises, and reasoning trees connecting them into structured arguments. |
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for learning audio-text connections rely on parallel audio- text data . a new approach allows for the representation of environmental soundscapes without using parallel data - a challenge for many applications . |
| Approach: | They propose a model that induces Audio-Text alignment without using parallel audio-text data. |
| Outcome: | The proposed model outperforms the current state-of-the-art for audio classification tasks with no audio-text data by 2.2% on the ESC50 and US8K tasks. |
NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
Peter West, Ronan Bras, Taylor Sorensen, Bill Lin, Liwei Jiang, Ximing Lu, Khyathi Chandu, Jack Hessel, Ashutosh Baheti, Chandra Bhagavatula, Yejin Choi
| Challenge: | a new commonsense knowledge model, NovaCOMET, combines knowledge and general task models. |
| Approach: | They propose an open commonsense knowledge model that combines knowledge and general task models. |
| Outcome: | The proposed model matches or exceeds existing knowledge models on commonsense reasoning tasks. |
Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets (N18-1)
Copied to clipboard
| Challenge: | Existing work suggests that concepts with concrete visual manifestations are easier to learn than abstract ones. |
| Approach: | They propose an algorithm for automatically computing the visual concreteness of words and topics within multimodal datasets. |
| Outcome: | The proposed algorithm predicts the capacity of machine learning algorithms to learn textual/visual relationships. |
What’s “up” with vision-language models? Investigating their struggle with spatial reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has re-surfaced a concern that has long plagued vision-language models: poor performance on simple tasks like attribute attachment, counting, etc. |
| Approach: | They evaluate 18 vision-language models and find they perform poorly on VQAv2 . they find that popular vision-linguistic pretraining corpora lack reliable data for learning spatial relationships . |
| Outcome: | The new models are compared with existing datasets on what'sup and visual-language models . they achieve 56% accuracy on the new benchmarks compared to 99% for humans . |
Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest (2023.acl-long)
Copied to clipboard
Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, Yejin Choi
| Challenge: | Large neural networks can generate jokes, but do they really “understand” humor? a new challenge challenges AI models to match a joke to a cartoon, identify a winning caption, and explain why a winner is funny. |
| Approach: | They propose three tasks based on the New Yorker Cartoon Caption Contest . they aim to match a joke to a cartoon, identify a winning caption and explain why it's funny . |
| Outcome: | The proposed tasks are based on the New Yorker Cartoon Caption Contest . they include matching a joke to a cartoon, identifying a winning caption, and explaining why a funny caption is funny. |
Unsupervised Discovery of Multimodal Links in Multi-image, Multi-sentence Documents (D19-1)
Copied to clipboard
| Challenge: | a structured training objective based on identifying whether collections of images and sentences co-occur in documents can suffice to predict links between specific images and specific sentences. |
| Approach: | They propose algorithms that discover image-sentence relationships without explicit annotation . they experiment on seven datasets of varying difficulty . |
| Outcome: | The proposed algorithms can predict links between images and sentences without explicit multimodal annotation. |
WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wild (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Currently, the volume and complexity of chat logs makes it difficult to analyze individual conversations. |
| Approach: | They propose a tool that enables fast, versatile, and large-scale conversation analysis by combining search and visualization capabilities with a list of criteria. |
| Outcome: | The proposed tool can be extended to handle millions of chat logs and other datasets. |
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Image captioning relies on reference-based automatic evaluations, but references are expensive to collect and comparing against multiple human-authored captions is insufficient. |
| Approach: | They propose a reference-free metric that can be used for automatic caption evaluation without references. |
| Outcome: | The proposed model outperforms existing metrics on image-text compatibility and a reference-augmented version achieves even higher correlation with human judgements. |
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization (2023.emnlp-main)
Copied to clipboard
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, Yejin Choi
| Challenge: | a dataset of 1.5 million conversations distilled from everyday spoken situations is limited in scale due to its associated costs. |
| Approach: | They propose to make SODA a publicly available, million-scale high-quality social dialogue dataset . they contextualize social commonsense knowledge from a knowledge graph to distill broad spectrum of social interactions . |
| Outcome: | The proposed dataset is the first publicly available, million-scale high-quality social dialogue dataset. |
Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms (2023.emnlp-main)
Copied to clipboard
Seungju Han, Junhyeok Kim, Jack Hessel, Liwei Jiang, Jiwan Chung, Yejin Son, Yejin Choi, Youngjae Yu
| Challenge: | NormLens is a visual-grounded framework for understanding commonsense norms . state-of-the-art models are not well-aligned with human annotation, we show . |
| Approach: | They propose a visual-grounded framework to study commonsense norms by NormLens . they find that models are not well-aligned with human annotation . |
| Outcome: | The proposed model judgments and explanations are not well-aligned with human annotations. |
Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think! (2020.emnlp-main)
Copied to clipboard
| Challenge: | a new tool for evaluating expressive cross-modal interactions is needed . empirical multimodally-additive function projection is a tool for isolating unimodal structure . |
| Approach: | They propose a tool that modifies model predictions so that cross-modal interactions are eliminated . they propose to use EMAP to evaluate models' ability to leverage cross-module interactions . |
| Outcome: | The proposed tool can be used to evaluate models on image+text classification tasks . it finds that removing cross-modal interactions results in little to no performance degradation . |
Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step (2023.acl-long)
Copied to clipboard
| Challenge: | Symbolic Chain-of-thought Distillation (SCoTD) is a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model. |
| Approach: | They propose a method to train a smaller student model on rationalizations from a larger teacher model. |
| Outcome: | The proposed method improves the performance of a student model in supervised and few-shot settings and especially for challenge sets. |
UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations (2024.naacl-long)
Copied to clipboard
Wenting Zhao, Justin Chiu, Jena Hwang, Faeze Brahman, Jack Hessel, Sanjiban Choudhury, Yejin Choi, Xiang Li, Alane Suhr
| Challenge: | Existing work evaluating commonsense reasoning focuses on making inferences about common, everyday situations. |
| Approach: | They propose to use an English language corpus to investigate commonsense reasoning . they characterize performance differences between human explainers and best-performing large language models . |
| Outcome: | The proposed method reduces the loss rate of human-written explanations on commonsense reasoning compared with the vanilla supervised fine-tuning approach . |
How effective is BERT without word ordering? Implications for language understanding and data privacy (2021.acl-short)
Copied to clipboard
| Challenge: | Ordered word sequences contain the rich structures that define language. |
| Approach: | They show that token representations and self-attention activations within BERT are resilient to shuffling the order of input tokens. |
| Outcome: | The proposed model is able to handle shuffled token representations and self-attention activations . the model can handle GLUE language understanding tasks with bag-of-words data . |
Reframing Human-AI Collaboration for Generating Free-Text Explanations (2022.naacl-main)
Copied to clipboard
| Challenge: | Large language models are capable of generating fluent-appearing text with little task-specific supervision. |
| Approach: | They propose a pipeline that combines GPT-3 with a supervised filter that incorporates binary acceptability judgments from humans in the loop. |
| Outcome: | The proposed model can generate freetext explanations in a fewshot setting with human-written examples. |
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2022.naacl-main)
Copied to clipboard
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, Yejin Choi
| Challenge: | Prior studies suggested pre-trained language models possess limited understanding of commonsense knowledge despite otherwise stellar performance on leaderboards. |
| Approach: | They propose a framework that uses larger models to teach smaller models by distilling knowledge symbolically as text in addition to the neural model. |
| Outcome: | The proposed framework is based on a general language model teacher's commonsense knowledge graphs and a neural commonsensing model surpassing the teacher model's in all three criteria. |
Domain-Specific Lexical Grounding in Noisy Visual-Textual Documents (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing image-text grounding approaches require detailed annotations, authors say . existing methods are difficult to adapt to unlabeled multi-image, multi-sentence documents, they say . |
| Approach: | They propose a method that can learn contextual meanings from unlabeled documents . they demonstrate that a simple unsupervised clustering-based method can be useful . |
| Outcome: | The proposed method is particularly effective for local contextual meanings of a word . existing image-text grounding methods are difficult to adapt to unlabeled multi-image, multi-sentence documents . |