Lukas Helff, Ahmad Omar, Felix Friedrich, Antonia Wüst, Hikaru Shindo, Rupert Mitchell, Tim Woydt, Patrick Schramowski, Wolfgang Stammer, Kristian Kersting
| Challenge: | Existing benchmarks intended to evaluate reasoning capabilities emphasize deductive reasoning, where conclusions necessarily follow from given premises. |
| Approach: | They propose an end-to-end framework for systematic evaluation and training of Large Language Models via Scalable Logical Reasoning. |
| Outcome: | The proposed framework doubles Llama-3-8B accuracy on SLR-Bench, achieving parity with Gemini-Flash-Thinking at a fraction of computational cost. |
Similar Papers
LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models (2026.acl-industry)
Copied to clipboard
Tianchun Li, Haochen Liu, Vishwa Pardeshi, Xingchen Wang, Tianci Liu, Huijun Zhao, Wei Fan, Jing Gao
| Challenge: | Small language models (SLMs) are promising for real-world deployment but struggle with high-stakes legal reasoning tasks. |
| Approach: | They propose a diagnostic-driven synthesis framework that extracts and refines reasoning trajectories from a capable teacher via fine-grained prompting and a self-reflective verification is employed to adaptively select the most effective data for the SLM student. |
| Outcome: | The proposed framework extracts and refines reasoning trajectories from a capable teacher via fine-grained prompting, then a self-reflective verification is employed to adaptively select the most effective data for the student. |
SCALER: Synthetic Scalable Adaptive Learning Environment for Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) is a principled way to enhance the reasoning capabilities of large language models, yet its effectiveness hinges on training signals that remain informative as models evolve. |
| Approach: | They propose a framework that sustains effective learning signals through adaptive environment design that transforms real-world programming problems into verifiable reasoning environments with controllable difficulty and unbounded instance generation. |
| Outcome: | The proposed framework outperforms baselines across diverse reasoning benchmarks and exhibits more stable, long-horizon training dynamics. |
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)
Copied to clipboard
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral
| Challenge: | Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics. |
| Approach: | They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models. |
| Outcome: | The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting. |
SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis (2026.findings-acl)
Copied to clipboard
| Challenge: | SynthRL synthesizes over 3.3K additional verifiable, challenging questions from approximately 8K seed samples. |
| Approach: | They propose a scalable and guaranteed pipeline for automatic data scaling in reasoning-oriented RL training. |
| Outcome: | The proposed pipeline synthesizes over 3.3K additional verifiable, challenging questions from approximately 8K seed samples. |
PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets with verifiable answers are limited in reliability, diversity, and scalability . a new approach to generate verifikatable data at scale is needed to improve models' performance . |
| Approach: | They propose a formal framework for synthesizing verifiable data at scale using a novel DSL-driven approach. |
| Outcome: | The proposed framework improves performance on a wide range of puzzles and logic benchmarks. |
Let’s Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing efforts to improve CoT prompting have limitations that require extensive human effort or performance needs to be improved. |
| Approach: | They propose a prompt approach for automatic reasoning called LBS3 inspired by curriculum learning which better reflects human learning habits. |
| Outcome: | The proposed approach achieves strongly competitive performance compared to baselines in reasoning-intensive tasks with varying open- and closed-source LLMs. |
PRover: Proof Generation for Interpretable Reasoning over Rules (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that transformers can act as “soft theorem provers” by answering questions over explicitly provided knowledge in natural language. |
| Approach: | They propose a transformer-based model that answers binary questions over rule-bases and generates the corresponding proofs. |
| Outcome: | The proposed model generates proofs with an accuracy of 87% while maintaining or improving performance on the QA task. |
Exploring Self-supervised Logic-enhanced Training for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Traditional attempts to enhance the logical reasoning abilities of language models often rely on supervised fine-tuning, limiting their generalization to new tasks or domains. |
| Approach: | They propose a framework for integrating logical reasoning capabilities into LLMs and activating them via in-context learning. |
| Outcome: | The proposed framework achieves comparable results to existing models on three language understanding benchmarks. |
RV-Syn: Rational and Verifiable Mathematical Reasoning Data Synthesis based on Structured Function Library (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for generating high-quality reasoning data are limited in quality and availability. |
| Approach: | They propose a method that constructs mathematical operations and generates verifiable graphs that are back-translated into complex problems. |
| Outcome: | The proposed method achieves a 6.3% performance gain over existing methods on LLaMA-3-8B and outperforms others with only half the training data (50k vs. 100k). |
FOL-Traces: Verified First-Order Logic Reasoning Traces at Scale (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing approaches to evaluate language models fail to provide structural clarity and verifiable inference. |
| Approach: | They propose to use a large-scale dataset of programmatically verified reasoning traces to evaluate structured logical inference. |
| Outcome: | The proposed model achieves 45.7% accuracy on masked operation prediction and 27% on two-step completion. |