PROST: Physical Reasoning about Objects through Space and Time (2021.findings-acl)
Copied to clipboard
| Challenge: | a new dataset is available to test pretraining of physical reasoning models . state-of-the-art models are inadequate at reasoning about physical interactions, authors say . |
| Approach: | They present a dataset that contains 18,736 multiple-choice questions from 14 templates . they propose to use the dataset to probe both causal and masked language models . |
| Outcome: | The proposed dataset contains 18,736 multiple-choice questions covering 10 physical reasoning concepts. |
Similar Papers
How Pre-trained Word Representations Capture Commonsense Physical Comparisons (D19-60)
Copied to clipboard
| Challenge: | Pre-trained word representations capture common sense on physical properties such as size and weight. |
| Approach: | They investigate whether pre-trained representations capture comparisons and find they have higher accuracy than previous approaches. |
| Outcome: | The proposed models learn a consistent ordering over all the objects in the comparisons. |
NEWTON: Are Large Language Models Capable of Physical Reasoning? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have been shown to encapsulate syntactic, semantic, word sense, and common-sense knowledge, but limited exploration of their physical reasoning abilities has been conducted. |
| Approach: | They propose a repository and benchmark to evaluate LLMs' physical reasoning skills . they use a pipeline to generate a variant of the benchmark customized to the objects and attributes relevant for their application. |
| Outcome: | The proposed benchmark examines the reasoning capabilities of language models across reasoning tasks. |
Probing Physical Reasoning with Counter-Commonsense Context (2023.acl-short)
Copied to clipboard
| Challenge: | Recent language models have succeeded in tasks that do not require contextual reasoning, such as size comparison and prediction of event frequency. |
| Approach: | They propose to use a dataset to measure physical commonsense to infer size relationships between objects in contexts that fit physical common sense. |
| Outcome: | The proposed dataset consists of contexts that fit physical commonsense and those that don't . results show that language models fail to use verbs and make incorrect judgments . |
There’s a Time and Place for Reasoning Beyond the Image (2022.acl-long)
Copied to clipboard
| Challenge: | Currently, most work in this area is focused on reasoning with local evidence, but there is a gap between a state-of-the-art joint model and human performance. |
| Approach: | They propose a model that can be used to infer, associate, and reason with contextual information from other sources to establish a more complete picture. |
| Outcome: | The proposed model can identify the news articles related to the image through segment-wise understandings of the signs, buildings, crowds, and more. |
A Benchmark for Reasoning with Spatial Prepositions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Spatial reasoning is a fundamental building block of human cognition . large language models (LLMs) are not on par with advanced aspects of human cognitive domains . |
| Approach: | They propose a benchmark to assess inferential properties of statements with spatial prepositions . they use prompt engineering to test the performance of two large language models . |
| Outcome: | The proposed benchmark shows that none of the models reaches human performance. |
Probing Across Time: What Does RoBERTa Know and When? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches to natural language processing rely on fixed artifacts such as language models . current studies have focused on how these models acquire and demonstrate knowledge . |
| Approach: | They apply probing techniques to examine how language models acquire knowledge . they aim to inform future work on more efficient pretraining and understanding dependencies . |
| Outcome: | The proposed model learns linguistic abstractions, factual and commonsense knowledge, and reasoning abilities fast, stably, and robustly across domains. |
Spatial and Temporal Language Understanding: Representation, Reasoning, and Grounding (2024.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial provides an overview of cutting edge research on spatial and temporal language understanding. |
| Approach: | This tutorial provides an overview of cutting edge research on spatial and temporal language understanding. |
| Outcome: | This tutorial provides an overview of cutting edge research on spatial and temporal language understanding. |
Can Language Models Understand Physical Concepts? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models do not understand basic physical concepts in the human world. |
| Approach: | They propose a method to transfer embodied knowledge from visual models to LMs . they use visual concepts and embodies concepts learned from interaction with the world . |
| Outcome: | The proposed method achieves comparable performance with scaling up parameters of LMs 134. |
CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions (2022.findings-acl)
Copied to clipboard
Tayfun Ates, M. Ateşoğlu, Çağatay Yiğit, Ilker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun, Deniz Yuret
| Challenge: | Existing models with similar physical and causal understanding capabilities are still underdeveloped. |
| Approach: | They propose a video question answering dataset that requires causal reasoning about physical forces and object interactions. |
| Outcome: | The proposed dataset requires causal reasoning about physical forces and object interactions. |
Identifying Physical Object Use in Sentences (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior research has focused on learning the prototypical functions of physical objects . but many sentences refer to objects even when they are not used . |
| Approach: | They propose a task that determines whether a physical object mentioned in a sentence was used or likely will be used. |
| Outcome: | The proposed model exploits data augmentation methods and FrameNet to fine-tune a pre-trainedmodel. |