Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study

Challenge:	Existing benchmarks that rely on final-answer accuracy fail to capture the quality of the reasoning process.
Approach:	They propose a fine-grained evaluation framework that assesses logical reasoning across three dimensions: overall accuracy, stepwise soundness, and representation-level probing.
Outcome:	The proposed framework assesses logical reasoning across three dimensions: overall accuracy, stepwise soundness, and representation-level probing.

Similar Papers

Overview

Challenge

Approach

Outcome

LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)

Copied to clipboard

Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral

Language Models

Challenge:	Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics.
Approach:	They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models.
Outcome:	The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting.

ReportLogic: Evaluating Logical Quality in Deep Research Reports (2026.acl-long)

Copied to clipboard

Jujia Zhao, Zhaoxin Huan, Zihan Wang, Xiaolu Zhang, Jun Zhou, Suzan Verberne, Zhaochun Ren

Robustness in NLP

Challenge:	Existing evaluation frameworks that evaluate large language models for Deep Research largely ignore this requirement.
Approach:	They propose a benchmark that quantifies report-level logical quality through a reader-centric lens of auditability.
Outcome:	The proposed model quantifies logical quality through a reader-centric lens of auditability.

Exploring Self-supervised Logic-enhanced Training for Large Language Models (2024.naacl-long)

Copied to clipboard

Fangkai Jiao, Zhiyang Teng, Bosheng Ding, Zhengyuan Liu, Nancy Chen, Shafiq Joty

Semantic Text Processing

Challenge:	Traditional attempts to enhance the logical reasoning abilities of language models often rely on supervised fine-tuning, limiting their generalization to new tasks or domains.
Approach:	They propose a framework for integrating logical reasoning capabilities into LLMs and activating them via in-context learning.
Outcome:	The proposed framework achieves comparable results to existing models on three language understanding benchmarks.

Structure Trumps Size: Rethinking Data Quality for LLM Reasoning (2025.findings-emnlp)

Copied to clipboard

Hu Xu, Zeyan Li, Rui Wang, Jianfeng Xu

Language Models

Challenge:	Existing methods for fine-tuning Large Language Models rely on heuristic strategies and lack systematic, quantitative frameworks for evaluating data quality.
Approach:	They propose a multi-dimensional quantitative framework for reasoning data management . they rigorously evaluate and optimize datasets along six orthogonal dimensions .
Outcome:	The proposed framework rigorously evaluates and optimizes datasets along six orthogonal dimensions.

There’s No Such Thing as Simple Reasoning for LLMs (2025.findings-acl)

Copied to clipboard

Nurul Fajrin Ariyani, Zied Bouraoui, Richard Booth, Steven Schockaert

Language Models

Challenge:	Existing work has focused on relatively complex “many-hop” reasoning problems.
Approach:	They analyse the performance of fine-tuned LLMs on simple reasoning problems . they find the models remain highly brittle, being susceptible to seemingly innocent perturbations .
Outcome:	The proposed models fail on simple reasoning problems, but are highly brittle . they are susceptible to seemingly innocent perturbations, such as adding duplicates to the set of premises and shuffling the order in which the premises are presented.

Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models (2025.naacl-long)

Copied to clipboard

Ziche Liu, Rui Ke, Yajiao Liu, Feng Jiang, Haizhou Li

Semantic Text Processing

Challenge:	Existing studies focus on data selection but lack a clear, unified framework . variability in experimental settings complicates systematic comparisons .
Approach:	They propose a three-stage scheme to standardize data selection for fine-tuning large language models . they propose unified comparison approach that incorporates ratio-based efficiency and ranking-based feasibility metrics to address inconsistencies across experiments.
Outcome:	The proposed scheme outperforms existing methods in a dozen key studies and identifies key challenges.

Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning (2025.findings-naacl)

Copied to clipboard

Venkatesh Mishra, Bimsara Pathiraja, Mihir Parmar, Sat Chidananda, Jayanth Srinivasa, Gaowen Liu, Ali Payani, Chitta Baral

Reasoning

Challenge:	Reasoning abilities of LLMs have been a key focus in recent years.
Approach:	They propose to use a college-level Multiple Choice Question-Answering task to identify LLM errors and evaluate their performance.
Outcome:	The proposed framework can be used in detailed error analysis of reasoning chains for logic-intensive complex tasks.

Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning (2025.naacl-long)

Copied to clipboard

Rujing Yao, Yang Wu, Chenghao Wang, Jingwei Xiong, Fang Wang, Xiaozhong Liu

Information Retrieval

Challenge:	Existing approaches to large language models focus on semantic similarity, neglecting the intricate logical structures and reasoning essential for addressing complex legal issues.
Approach:	They propose a Logical-Semantic Integration Model (LSIM) that bridges semantic and logical coherence and a supervised framework that integrates semantic features with in-context learning.
Outcome:	The proposed framework significantly improves accuracy and reliability on a real-world legal QA dataset.

Can LLMs Reason Like Doctors? Exploring the Limits of Large Language Models in Complex Medical Reasoning (2026.findings-eacl)

Copied to clipboard

Flavio Merenda, Jose Manuel Gomez-Perez, German Rigau

Language Models

Challenge:	Large language models (LLMs) have shown remarkable progress in reasoning across multiple domains, but it remains unclear whether their abilities reflect genuine reasoning or sophisticated pattern matching.
Approach:	They conduct one of the largest evaluations to date, assessing 77 LLMs . they select three medical question answering (QA) benchmarks targeting reasoning processes .
Outcome:	The results highlight the need to improve specific reasoning strategies to better reflect medical decision-making.

LLM Tropes: Revealing Fine-Grained Values and Opinions in Large Language Models (2024.findings-emnlp)

Copied to clipboard

Dustin Wright, Arnav Arora, Nadav Borenstein, Srishti Yadav, Serge Belongie, Isabelle Augenstein

Language Models

Challenge:	Existing approaches to evaluate latent values and opinions in large language models suffer from three notable shortcomings.
Approach:	They propose to analyze 156k LLM responses to 62 propositions of the Political Compass Test (PCT) generated by 6 LLMs using 420 prompt variations.
Outcome:	The proposed analysis of 156k LLM responses to the Political Compass Test (PCT) generated by 6 LLMs shows that tropes are recurrent and consistent across prompts.