Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning (2025.findings-naacl)
Copied to clipboard
Venkatesh Mishra, Bimsara Pathiraja, Mihir Parmar, Sat Chidananda, Jayanth Srinivasa, Gaowen Liu, Ali Payani, Chitta Baral
| Challenge: | Reasoning abilities of LLMs have been a key focus in recent years. |
| Approach: | They propose to use a college-level Multiple Choice Question-Answering task to identify LLM errors and evaluate their performance. |
| Outcome: | The proposed framework can be used in detailed error analysis of reasoning chains for logic-intensive complex tasks. |
Similar Papers
Evaluating Step-by-step Reasoning Traces: A Survey (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation practices are inconsistent, resulting in fragmented progress across evaluator design and benchmark development. |
| Approach: | a survey provides a comprehensive overview of step-by-step reasoning evaluation . existing evaluation practices are inconsistent, resulting in fragmented progress . |
| Outcome: | The proposed evaluation criteria are based on four top-level categories . the results are presented in a systematic review of the literature. |
Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study (2025.findings-emnlp)
Copied to clipboard
Yujun Zhou, Jiayi Ye, Zipeng Ling, Yufei Han, Yue Huang, Haomin Zhuang, Zhenwen Liang, Kehan Guo, Taicheng Guo, Xiangqi Wang, Xiangliang Zhang
| Challenge: | Existing benchmarks that rely on final-answer accuracy fail to capture the quality of the reasoning process. |
| Approach: | They propose a fine-grained evaluation framework that assesses logical reasoning across three dimensions: overall accuracy, stepwise soundness, and representation-level probing. |
| Outcome: | The proposed framework assesses logical reasoning across three dimensions: overall accuracy, stepwise soundness, and representation-level probing. |
Current Advances in LLM Reasoning (2026.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial examines comprehensive evaluation strategies to assess the reasoning abilities of large language models (LLMs) advanced inference time methods and post-training methods that aim to make LLMs think more like humans are discussed in this tutorial. |
| Approach: | This tutorial explores comprehensive evaluation strategies to assess the reasoning abilities of large language models (LLMs) and discusses two types of methods to improve models’ reasoning: advanced inference time methods, structured and self-improvement inference methods, and post-training methods, such as RLHF, DPO, and GRPO. |
| Outcome: | This tutorial examines evaluation strategies to assess the reasoning abilities of large language models and discusses two types of methods to improve models’ reasoning. |
Too Fast, Too Shallow – LLMs, Including Reasoning LLMs, Are Unreliable Constitutional Reasoners (2026.findings-acl)
Copied to clipboard
| Challenge: | Using three different datasets, we assess LLMs’ constitutional reasoning abilities using three different constitutional frameworks. |
| Approach: | They propose to use the influential dual process theory of cognition to assess LLMs' constitutional reasoning abilities. |
| Outcome: | The LLMs label less than 70% correctly and open-weight reasoning LLM and gpt-4o outperform open- weight non-reasoning LLM. |
Verifying the Steps of Deductive Reasoning Chains (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models have been shown to improve the reasoning capabilities of the models. |
| Approach: | They propose to automate verification of individual reasoning steps in a logical deductive Chain-of-Thought. |
| Outcome: | The proposed method can detect unsound reasoning steps fairly well, but under-performs symbolic methods. |
A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated great potential for domain-specific applications, such as the law domain. |
| Approach: | They propose a framework to investigate LLMs' competence in the law domain by using similar cases and multi-choice options. |
| Outcome: | The proposed solutions can be extended to other domains to facilitate evaluations in other domain. |
Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics (2026.acl-long)
Copied to clipboard
| Challenge: | Evaluating the quality of LLM-generated reasoning traces in expert domains is essential for ensuring credibility and explainability, yet remains challenging due to the inherent complexity of such reasoning tasks. |
| Approach: | They propose a large-scale legal reasoning dataset with an emphasis on reasoning trace evaluation that converts court judgments into hierarchical trees of opposing parties’ arguments and the court’s conclusions. |
| Outcome: | The proposed model improves the quality of LLM-generated reasoning traces in legal domains, whereas RL improves correctness albeit with reduced coverage. |
Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to large language models focus on semantic similarity, neglecting the intricate logical structures and reasoning essential for addressing complex legal issues. |
| Approach: | They propose a Logical-Semantic Integration Model (LSIM) that bridges semantic and logical coherence and a supervised framework that integrates semantic features with in-context learning. |
| Outcome: | The proposed framework significantly improves accuracy and reliability on a real-world legal QA dataset. |
Assessing LLM Reasoning Steps via Principal Knowledge Grounding (2025.findings-emnlp)
Copied to clipboard
Hyeon Hwang, Yewon Cho, Chanwoong Yoon, Yein Park, Minju Song, Kyungjae Lee, Gangwoo Kim, Jaewoo Kang
| Challenge: | Step-by-step reasoning has become a standard approach for large language models to tackle complex tasks. |
| Approach: | They propose a framework that assesses the knowledge grounding of intermediate reasoning by using a large-scale repository of atomic knowledge essential for reasoning. |
| Outcome: | The evaluation suite identifies missing or misapplied knowledge elements and provides crucial insights for uncovering fundamental reasoning deficiencies in LLMs. |
Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above (2025.acl-long)
Copied to clipboard
| Challenge: | Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing. |
| Approach: | They argue for a reform of multiple choice question answering (MCQA) they argue for more generative formats based on human testing . |
| Outcome: | The proposed reforms improve the quality of MCQA, the authors argue . they show that even when MCQ is a useful format, its datasets suffer from leakage, unanswerability, shortcuts and saturation. |