Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms (2023.emnlp-main)
Copied to clipboard
Seungju Han, Junhyeok Kim, Jack Hessel, Liwei Jiang, Jiwan Chung, Yejin Son, Yejin Choi, Youngjae Yu
| Challenge: | NormLens is a visual-grounded framework for understanding commonsense norms . state-of-the-art models are not well-aligned with human annotation, we show . |
| Approach: | They propose a visual-grounded framework to study commonsense norms by NormLens . they find that models are not well-aligned with human annotation . |
| Outcome: | The proposed model judgments and explanations are not well-aligned with human annotations. |
Similar Papers
Find Someone Who: Visual Commonsense Understanding in Human-Centric Grounding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Visual scenes often involve multiple people and humans can distinguish between them based on context descriptions about what happened before, their mental/physical states, and intentions. |
| Approach: | They propose a task that tests human-centric commonsense grounding models' ability to distinguish individuals given context descriptions about what happened before and their mental/physical states or intentions. |
| Outcome: | The proposed model outperforms pre-trained and non-pretrained models on 130k commonsense descriptions annotated on 67k images. |
Commonsense Reasoning for Natural Language Processing (2020.acl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we will outline the various types of commonsense knowledge and discuss techniques to gather and represent commonsence knowledge. |
| Approach: | This tutorial will provide researchers with the critical foundations and recent advances in commonsense representation and reasoning. |
| Outcome: | This tutorial will outline the various types of commonsense and discuss techniques to gather and represent commonsence knowledge while highlighting the challenges specific to this type of knowledge (e.g., reporting bias). |
What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge (2022.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation methods to measure what language models learn from multimodal training are lacking. |
| Approach: | They propose two evaluation tasks to measure commonsense knowledge in language models by using visual data to evaluate multimodal models and unimodal baselines. |
| Outcome: | The proposed evaluation tasks show that training on a visual modality improves on the visual commonsense knowledge in language models. |
EgoNormia: Benchmarking Physical-Social Norm Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing VLMs lack robust grounded norm understanding, a new study finds . current VLM models lack robust grounding, despite a high score for safety and privacy . |
| Approach: | They propose a pipeline to generate grounded MCQs from ego-centric videos of human interactions. |
| Outcome: | The proposed pipeline can generate grounded MCQs from egocentric video . it shows that current VLMs lack robust grounded norm understanding . |
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (D18-1)
Copied to clipboard
| Challenge: | a new dataset presents a task of grounded commonsense inference, unifying natural language inference and commonsensical reasoning. |
| Approach: | They propose a procedure that constructs a de-biased dataset by iteratively training stylistic classifiers and using them to filter the data. |
| Outcome: | The proposed procedure oversamples a de-biased dataset using state-of-the-art language models . human models struggle on the proposed procedure, indicating significant opportunities for future research. |
Visual Commonsense in Pretrained Unimodal and Multimodal Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Fig. 1 shows how text-only and image-only models can capture commonsense visual attributes, but reporting bias affects their performance. |
| Approach: | They use a Visual Commonsense Tests dataset to validate their findings . they find multimodal models better reconstruct attribute distributions, but are still subject to reporting bias . |
| Outcome: | The proposed model improves on the unimodal and multimodal models, but is still subject to reporting bias. |
Contextualizing Language Models for Norms Diverging from Social Majority (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies on transformer-based language models have shown that there seems to be a 'moral dimension' to LMs, as they show high accuracy in related downstream tasks such as moral reasoning and action classification. |
| Approach: | They propose a mechanism based on deontic logic to allow for a flexible adaptation of individual norms by de-biasing training data sets and a task-reduction to textual entailment. |
| Outcome: | The proposed mechanism de-biases training data sets and reduces tasks to textual entailment. |
Things not Written in Text: Exploring Spatial Commonsense from Visual Signals (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained language models fail in many NLP tasks, but are ineffective in spatial commonsense reasoning. |
| Approach: | They propose a spatial commonsense benchmark that focuses on relative scales of objects and the positional relationship between people and objects under different actions. |
| Outcome: | The proposed framework outperforms pretrained models in answering spatial questions. |
What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations (2023.findings-emnlp)
Copied to clipboard
Kavel Rao, Liwei Jiang, Valentina Pyatkin, Yuling Gu, Niket Tandon, Nouha Dziri, Faeze Brahman, Yejin Choi
| Challenge: | Moral or ethical judgments rely heavily on the contexts in which they occur . a student model that produces defeasible contexts with improved validity, diversity, and defasibility is superior to intermediate student models . |
| Approach: | a new study uses a student model to provide contextualizations that make an action morally acceptable . the model is based on a dataset of 115K defeasible moral actions rated highly by human annotators . |
| Outcome: | The proposed model outperforms all intermediate models in a high-quality dataset . the model is based on 1.2M entries of contextualizations and rationales for 115K moral actions . |
On the Consistency of Commonsense in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluations of commonsense for large language models focus on downstream knowledge tasks, failing to probe whether LLMs truly understand and utilize knowledge or merely memorize it. |
| Approach: | They propose to automatically construct a large benchmark named CoCo which measures LLMs’ knowledge memorization, comprehension, and application and examines the consistency between these tasks. |
| Outcome: | The proposed benchmark systematically assesses LLMs’ knowledge memorization, comprehension, and application and examines the consistency between these tasks. |