LR²Bench: Evaluating Long-chain Reflective Reasoning Capabilities of Large Language Models via Constraint Satisfaction Problems (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in o1-like models have significantly enhanced the reasoning abilities of Large Language Models (LLMs). |
| Approach: | They propose a benchmark to evaluate the Long-chain Reflective Reasoning capabilities of Large Language Models. |
| Outcome: | The proposed benchmark evaluates the Long-chain Reflective Reasoning capabilities of Large Language Models (LLMs) it consists of 850 samples across six Constraint Satisfaction Problems (CSPs) |
Similar Papers
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2025.acl-long)
Copied to clipboard
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li
| Challenge: | Existing benchmarks on longcontext large language models fail to reflect their deep understanding capabilities across diverse tasks. |
| Approach: | They propose a benchmark to assess the ability of long-context large language models to handle long-text problems. |
| Outcome: | The proposed model achieves 50.1% accuracy when directly answering the questions . human experts achieve only 53.7% accuracy under a 15-minute time constraint . |
CFBench: A Comprehensive Constraints-Following Benchmark for LLMs (2025.acl-long)
Copied to clipboard
Tao Zhang, ChengLIn Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, Zenan Zhou
| Challenge: | Existing evaluations of Large Language Models (LLMs) focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user’s perspective. |
| Approach: | They propose a Chinese Comprehensive Constraints Following Benchmark for LLMs that compiles constraints from real-world instructions and constructs a systematic framework for constraint types. |
| Outcome: | The proposed framework integrates multi-dimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. |
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)
Copied to clipboard
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral
| Challenge: | Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics. |
| Approach: | They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models. |
| Outcome: | The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting. |
Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning? (2025.acl-long)
Copied to clipboard
Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Z.y. Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, Bo Zheng
| Challenge: | Recent advances in o1-like models have generated long Chain-of-Thought reasoning steps to improve the reasoning abilities of existing Large Language Models (LLMs). |
| Approach: | They propose a DeltaBench to analyze the quality and effectiveness of o1-like models and measure their ability to detect errors in long COT reasoning. |
| Outcome: | The proposed model can detect errors in long COT reasoning. |
LRBench and Judge-R1: Principled Evaluation and Training of LLM-Based Judges for Long-Context Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating large language models (LLMs) under long contexts are underexplored. |
| Approach: | They propose a large-scale benchmark for evaluating large language models (LLMs) that combines reinforcement learning with multi-turn search to enable grounded and principle-aware evaluation. |
| Outcome: | The proposed model outperforms single-turn baselines across domains and principles. |
CriticBench: Benchmarking LLMs for Critique-Correct Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | CriticBench is a benchmark designed to assess LLMs’ abilities to critique and refine their reasoning across a variety of tasks. |
| Approach: | They propose a benchmark to assess LLMs' ability to critique and correct reasoning across a variety of tasks. |
| Outcome: | The proposed benchmark examines the performance of 17 large language models in generation, critique, and correction reasoning. |
TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Grasping the concept of time is a fundamental facet of human cognition. |
| Approach: | They propose a hierarchical temporal reasoning benchmark that covers a broad spectrum of temporal phenomena. |
| Outcome: | The proposed benchmark shows that state-of-the-art LLMs are still far behind humans in temporal reasoning . |
Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that Legal-R1 delivers competitive performance across diverse tasks. |
| Approach: | They propose to evaluate 12 large language models across 17 legal tasks across statutory and case-law traditions to determine their general reasoning performance. |
| Outcome: | The proposed model performs well across 17 legal tasks across statutory and case-law traditions. |
RiddleBench: A New Generative Reasoning Benchmark for LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) show remarkable capabilities, but complex reasoning skills require deeper investigation. |
| Approach: | They propose a benchmark of 1,737 puzzles to test reasoning beyond simple pattern matching. |
| Outcome: | The proposed model performs poorly when faced with reordered constraints or irrelevant information. |
Are Your LLMs Capable of Stable Reasoning? (2025.findings-acl)
Copied to clipboard
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, Kai Chen
| Challenge: | Existing evaluation protocols and metrics do not capture the full spectrum of LLM capabilities, especially in complex reasoning tasks. |
| Approach: | They propose a new evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model’s potential capabilities and operational consistency. |
| Outcome: | The proposed evaluation metric measures model performance across multiple sampling attempts and provides comprehensive insights into their potential capabilities and operational consistency. |