| Challenge: | Existing reasoning datasets saturate and fail to test abstract, multi-step problems, especially pathfinding and complex rule constraint satisfaction. |
| Approach: | They propose to use a spatial few-shot grid to evaluate spatial and rule-based reasoning with 1,000 2D grid puzzles. |
| Outcome: | The proposed model can be used to evaluate spatial reasoning and improve its accuracy. |
Similar Papers
SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) do not perform well on the datasets. |
| Approach: | They propose to use a Spatial Reasoning Characterization framework and a spatial reasoning path framework to study spatial reasoning. |
| Outcome: | The proposed framework and datasets outperform state-of-the-art models in spatial reasoning. |
SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the spatial reasoning capabilities of modern language models (LMs) however, there has been limited research into the spatial thinking capabilities of LMs. |
| Approach: | They propose a question-answering (QA) benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior work. |
| Outcome: | The proposed method significantly improves LMs' ability on spatial understanding, which in turn helps solve two external datasets, bAbI, and boolQ. |
SParC: Cross-Domain Semantic Parsing in Context (P19-1)
Copied to clipboard
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, Dragomir Radev
| Challenge: | Xu et al., 2017): a dataset for cross-domain semantic parsing in context with 4,298 question sequences. |
| Approach: | They present a dataset for cross-domainSemanticParsing inContext that consists of 4,298 coherent question sequences. |
| Outcome: | The proposed dataset demonstrates that it has greater semantic diversity and can be generalized to unseen domains due to its cross-domain nature and the unseened databases at test time. |
PROST: Physical Reasoning about Objects through Space and Time (2021.findings-acl)
Copied to clipboard
| Challenge: | a new dataset is available to test pretraining of physical reasoning models . state-of-the-art models are inadequate at reasoning about physical interactions, authors say . |
| Approach: | They present a dataset that contains 18,736 multiple-choice questions from 14 templates . they propose to use the dataset to probe both causal and masked language models . |
| Outcome: | The proposed dataset contains 18,736 multiple-choice questions covering 10 physical reasoning concepts. |
Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter? (2024.emnlp-main)
Copied to clipboard
Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin Rrv, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, Chitta Baral
| Challenge: | Existing studies evaluate only the final predicted answer of a puzzle, without providing any finer metrics to evaluate them. |
| Approach: | They propose to use a grid-based evaluation dataset to evaluate LLMs' reasoning abilities and a new error taxonomy to evaluate their reasoning chains. |
| Outcome: | The proposed model outperforms existing prompting methods on a wide range of natural language understanding tasks previously thought to be exclusive to humans. |
Step Guided Reasoning: Improving Mathematical Reasoning using Guidance Generation and Step Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve mathematical reasoning require extensive datasets for training or depend on few-shot methods that compromise computational accuracy. |
| Approach: | They propose a training-free adaptation framework that efficiently equips general-purpose pre-trained language models with enhanced mathematical reasoning capabilities. |
| Outcome: | The proposed framework outperforms Qwen2.5-72B-Math-Instruct on MMLU-STEM with a score of 90.9%, compared to 87.3%. |
Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning (2025.findings-emnlp)
Copied to clipboard
Yihong Tang, Ao Qu, Zhaokai Wang, Dingyi Zhuang, Zhaofeng Wu, Wei Ma, Shenhao Wang, Yunhan Zheng, Zhan Zhao, Jinhua Zhao
| Challenge: | Currently, vision-language models excel in many downstream tasks but struggle with spatial reasoning, which is crucial for navigation and interaction with physical environments. |
| Approach: | They propose a framework that generates synthetic data to provide targeted supervision for VLMs across these basic spatial capabilities. |
| Outcome: | The proposed framework disentangles 2D spatial reasoning into three core components: direction comprehension, distance estimation, and localization. |
Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models lack spatial reasoning capability, despite their ability to comprehend spatial arrangements and model structural relations. |
| Approach: | They propose a benchmark to evaluate vision-language models' spatial perception, structural understanding, and reasoning capabilities by minimizing reliance on domain-specific knowledge. |
| Outcome: | The proposed benchmark is based on 1,100 carefully curated real-world images with high spatial complexity. |
Answering Cross-Dimensional Geometric Visual Questions by Multi-constraint Spatial Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for solving complex visual questions are limited in their ability to represent in a cross-dimensional space. |
| Approach: | They propose a method that can answer complex visual questions using cross-dimensional reasoning. |
| Outcome: | The proposed method can answer complex visual questions in 2D to 3D space with great application value. |
TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large reasoning models (LRMs) have driven significant breakthroughs across various reasoning tasks including deductive, arithmetic, commonsense, relational, and symbolic reasoning. |
| Approach: | They propose a programmatic approach to evaluate basic strategic, spatial, and logical reasoning abilities in large reasoning models through four two-player Tic-Tac-Toe-style games that humans can effortlessly solve from a young age. |
| Outcome: | The proposed model performs 41% lower on TTT-Bench than MATH 500 and AIME 2024 models, while the larger models perform better on longer reasoning traces. |