Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter? (2024.emnlp-main)
Copied to clipboard
Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin Rrv, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, Chitta Baral
| Challenge: | Existing studies evaluate only the final predicted answer of a puzzle, without providing any finer metrics to evaluate them. |
| Approach: | They propose to use a grid-based evaluation dataset to evaluate LLMs' reasoning abilities and a new error taxonomy to evaluate their reasoning chains. |
| Outcome: | The proposed model outperforms existing prompting methods on a wide range of natural language understanding tasks previously thought to be exclusive to humans. |
Similar Papers
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)
Copied to clipboard
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral
| Challenge: | Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics. |
| Approach: | They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models. |
| Outcome: | The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting. |
Evaluating Step-by-step Reasoning Traces: A Survey (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation practices are inconsistent, resulting in fragmented progress across evaluator design and benchmark development. |
| Approach: | a survey provides a comprehensive overview of step-by-step reasoning evaluation . existing evaluation practices are inconsistent, resulting in fragmented progress . |
| Outcome: | The proposed evaluation criteria are based on four top-level categories . the results are presented in a systematic review of the literature. |
FineReason: Evaluating and Improving LLMs’ Deliberate Reasoning through Reflective Puzzle Solving (2025.acl-long)
Copied to clipboard
Guizhen Chen, Weiwen Xu, Hao Zhang, Hou Pong Chan, Chaoqun Liu, Lidong Bing, Deli Zhao, Anh Tuan Luu, Yu Rong
| Challenge: | Recent advances in large language models (LLMs) highlight an important shift from the “System 1” way of quick reactions to the “system 2” style of reflection-and-correction problem solving. |
| Approach: | They propose a logic-puzzle benchmark for systematic evaluation of large language models' reasoning capabilities that decomposes each puzzle into atomic steps. |
| Outcome: | The proposed model improves on state checking and state transition tasks and demonstrates gains in reasoning by up to 5.1%. |
Assessing LLM Reasoning Steps via Principal Knowledge Grounding (2025.findings-emnlp)
Copied to clipboard
Hyeon Hwang, Yewon Cho, Chanwoong Yoon, Yein Park, Minju Song, Kyungjae Lee, Gangwoo Kim, Jaewoo Kang
| Challenge: | Step-by-step reasoning has become a standard approach for large language models to tackle complex tasks. |
| Approach: | They propose a framework that assesses the knowledge grounding of intermediate reasoning by using a large-scale repository of atomic knowledge essential for reasoning. |
| Outcome: | The evaluation suite identifies missing or misapplied knowledge elements and provides crucial insights for uncovering fundamental reasoning deficiencies in LLMs. |
Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzles (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to eliminate implicit biases in LLMs do not eradicate underlying behavioral bias. |
| Approach: | They propose a framework that uses logic grid puzzles to probe the influence of social stereotypes on logical reasoning and decision making in LLMs. |
| Outcome: | The proposed framework systematically probes the influence of social stereotypes on logical reasoning and decision making in LLMs. |
Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) often display unstable behaviors, e.g., hallucinating unsupported premises, overthinking simple tasks, and displaying higher sensitivity to prompt variations. |
| Approach: | They propose a graph-based analytical framework that clusters long, verbose CoT outputs into semantically coherent reasoning steps, then constructs directed reasoning graphs to capture contextual and logical dependencies among these steps. |
| Outcome: | The proposed framework enables quantitative evaluation of internal reasoning structure and quality beyond conventional metrics and provides practical insights for prompt engineering and cognitive analysis of LLMs. |
RAFFLES: Reasoning-based Attribution of Faults for LLM Systems (2026.eacl-long)
Copied to clipboard
Chenyang Zhu, Spencer Hong, Jingyu Wu, Kushal Chawla, Yuhui Tang, Youbing Yin, Nathan Wolfe, Erin Babinsky, Daben Liu
| Challenge: | Existing evaluation frameworks focus on simple metrics and end-to-end outcomes, but they struggle with longer contexts. |
| Approach: | They propose an offline evaluation architecture that incorporates iterative reasoning to evaluate the quality of the candidate faults and rationales of the Judge. |
| Outcome: | The proposed architecture outperforms baseline evaluation frameworks with two datasets to identify step-level faults in multi-agent systems and ReasonEval datasets. |
Puzzle Solving using Reasoning of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their logical reasoning abilities across various domains. |
| Approach: | They propose to divide puzzles into rule-based and rule-less categories and critically assess LLMs' performance through various methodologies. |
| Outcome: | The proposed models have demonstrated capabilities in deductive reasoning and inductive reasoning, but they face limitations in inductive thinking. |
Exposing the Achilles’ Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations focus on final accuracy, neglecting the critical aspect of reasoning capabilities. |
| Approach: | They propose to evaluate LLMs’ abilities to detect and correct reasoning mistakes by using rule-based methods and smaller language models. |
| Outcome: | The proposed model outperforms existing models such as GPT-4o and GPT4 in both accuracy and accuracy, but lacks data contamination and memorization concerns. |
SATBench: Benchmarking LLMs’ Logical Reasoning via Automated Puzzle Generation from SAT Formulas (2025.emnlp-main)
Copied to clipboard
Anjiang Wei, Yuheng Wu, Yingjia Wan, Tarun Suresh, Huanmi Tan, Zhanke Zhou, Sanmi Koyejo, Ke Wang, Alex Aiken
| Challenge: | SATBench is a benchmark for evaluating the logical reasoning capabilities of large language models (LLMs) through logical puzzles derived from Boolean satisfiability (SAT) problems. |
| Approach: | They propose a benchmark to evaluate logical reasoning capabilities of large language models (LLMs) using logical puzzles derived from Boolean satisfiability problems. |
| Outcome: | The proposed model achieves 65.0% accuracy on hard UNSAT problems, close to the baseline of 50%. |