| Challenge: | Recent language models have succeeded in tasks that do not require contextual reasoning, such as size comparison and prediction of event frequency. |
| Approach: | They propose to use a dataset to measure physical commonsense to infer size relationships between objects in contexts that fit physical common sense. |
| Outcome: | The proposed dataset consists of contexts that fit physical commonsense and those that don't . results show that language models fail to use verbs and make incorrect judgments . |
Similar Papers
How Pre-trained Word Representations Capture Commonsense Physical Comparisons (D19-60)
Copied to clipboard
| Challenge: | Pre-trained word representations capture common sense on physical properties such as size and weight. |
| Approach: | They investigate whether pre-trained representations capture comparisons and find they have higher accuracy than previous approaches. |
| Outcome: | The proposed models learn a consistent ordering over all the objects in the comparisons. |
PROST: Physical Reasoning about Objects through Space and Time (2021.findings-acl)
Copied to clipboard
| Challenge: | a new dataset is available to test pretraining of physical reasoning models . state-of-the-art models are inadequate at reasoning about physical interactions, authors say . |
| Approach: | They present a dataset that contains 18,736 multiple-choice questions from 14 templates . they propose to use the dataset to probe both causal and masked language models . |
| Outcome: | The proposed dataset contains 18,736 multiple-choice questions covering 10 physical reasoning concepts. |
Code over Words: Overcoming Semantic Inertia via Code-Grounded Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models struggle with semantic inertia, a problem that is often attributed to natural language encoding, which entangles descriptive semantics and logical rules, leading to persistent hallucinations of familiar physics despite explicit contradictory rules. |
| Approach: | They propose a framework that decouples logical dynamics from visual priors via amortized theory induction and counterfactual contrastive alignment. |
| Outcome: | The proposed framework outperforms expensive inference-time search methods in both efficiency and accuracy. |
POSQA: Probe the World Models of LLMs with Size Comparisons (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Embodied language comprehension emphasizes that language understanding is not only mental processing in the brain but also involves interactions with the physical and social environment. |
| Approach: | They propose to use a physical object size question to examine the extremity of large language models to test their embodied comprehension. |
| Outcome: | The proposed dataset shows that even the largest LLMs perform poorly under the zero-shot setting. |
Things not Written in Text: Exploring Spatial Commonsense from Visual Signals (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained language models fail in many NLP tasks, but are ineffective in spatial commonsense reasoning. |
| Approach: | They propose a spatial commonsense benchmark that focuses on relative scales of objects and the positional relationship between people and objects under different actions. |
| Outcome: | The proposed framework outperforms pretrained models in answering spatial questions. |
Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size (2026.findings-acl)
Copied to clipboard
Dikshant Kukreja, Kshitij Sah, Gautam Gupta, Avinash Anand, Rajiv Ratn Shah, Zhengkui Wang, Aik Beng Ng, Erik Cambria
| Challenge: | Larger language models become better and worse at handling contextual information . et al. (2017) formalized contextual entrainment as a tendency to favor tokens in context . |
| Approach: | They formalize the first scaling laws for contextual entrainment . they find large models are four times more resistant to counterfactual misinformation . |
| Outcome: | The largest models are four times more resistant to counterfactual misinformation than the smallest, but twice as prone to copying arbitrary tokens. |
A Multi-layered Approach to Physical Commonsense Understanding: Creation and Evaluation of an Italian Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Using a multilingual model, we examine the ability of large language models to perform reasoning tasks. |
| Approach: | They propose to use a multilingual model to analyze commonsense reasoning in large language models for Italian and to provide a semi-automated system to complete the annotation. |
| Outcome: | The proposed model performs at high-level classification tasks but its easoning is inconsistent and unverifiable, since it does not capture intermediate evidence. |
Commonsense Reasoning for Natural Language Processing (2020.acl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we will outline the various types of commonsense knowledge and discuss techniques to gather and represent commonsence knowledge. |
| Approach: | This tutorial will provide researchers with the critical foundations and recent advances in commonsense representation and reasoning. |
| Outcome: | This tutorial will outline the various types of commonsense and discuss techniques to gather and represent commonsence knowledge while highlighting the challenges specific to this type of knowledge (e.g., reporting bias). |
Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing (D19-60)
Copied to clipboard
| Challenge: | Workshop on Commonsense Inference in Natural Language Processing focuses on commonsense knowledge representation and application in NLP tasks. |
| Approach: | COIN is a workshop on commonsense inference in natural language processing . workshop included two shared tasks on reading comprehension using commonsensense knowledge . |
| Outcome: | the workshop focused on modeling commonsense knowledge and commonsensing in natural language processing tasks. |
Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities (2025.findings-acl)
Copied to clipboard
Wenyue Hua, Kaijie Zhu, Lingyao Li, Lizhou Fan, Mingyu Jin, Shuhang Lin, Haochen Xue, Zelong Li, Jindong Wang, Yongfeng Zhang
| Challenge: | Using large language models, large language model models can be used to evaluate reasoning abilities in context-rich scenarios. |
| Approach: | They construct datasets for both propositional logic and abductive logic reasoning with four difficulty levels across 12 distinct domains based on Wikipedia categorization and those with purely abstract variables. |
| Outcome: | The proposed model can be used to benchmark LLMs in real-world scenarios, but not in context-rich scenarios. |