Temporal Consistency for LLM Reasoning Process Error Identification (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Empirical evaluations show consistent performance improvements over baseline methods . 7B/8B distilled models outperform all 70B/72B models and GPT-4o on ProcessBench . |
| Approach: | They propose a temporal consistency method that leverages consistency in a sequence of self-reflection actions to improve verification accuracy. |
| Outcome: | The proposed method outperforms existing methods on three benchmarks . it leverages consistency in a sequence of self-reflection actions to improve accuracy . |
Similar Papers
Self-Consistency Boosts Calibration for Math Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing solutions for math reasoning tasks use semantic parsing or AST decoding, but performance can degrade dramatically even with slight changes to the questions. |
| Approach: | They propose three calibration methods based on self-consistency for math reasoning tasks. |
| Outcome: | The proposed methods bridge model confidence and accuracy better than existing methods based on p(True) or logit. |
ProcessBench: Identifying Process Errors in Mathematical Reasoning (2025.acl-long)
Copied to clipboard
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin
| Challenge: | Existing models fail to generalize to more challenging math problems, authors say . existing benchmarks related to assessing language models' reasoning process are limited . |
| Approach: | They propose a tool to measure language models' ability to identify erroneous steps in reasoning . they use two types of models: process reward models and critic models . |
| Outcome: | The proposed model outperforms existing models in evaluating language models' reasoning process . the best open-source model has demonstrated the critique capability competitive with the proprietary model . |
Assessing the Belief Consistency of Large Language Models on the Logical Conversation Process (2026.acl-long)
Copied to clipboard
Tomoki Tsujimura, Matīss Rikters, Masaki Asada, Shusaku Egami, Tatsuya Ishigaki, Ken Yano, Hiroya Takamura
| Challenge: | Large language models have been shown remarkable ability to understand given contexts. |
| Approach: | They propose a method to evaluate whether beliefs held by LLMs remain consistent . they propose to use multiple choice question answering format to assess belief consistency . |
| Outcome: | The proposed method evaluates the consistency of LLMs in a multiple-choice question answering format. |
Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for training effective PRMs focus on the first incorrect step and all preceding steps, assuming that all subsequent steps are incorrect. |
| Approach: | They propose a data annotation method specifically designed to score the long CoT reasoning process by using an LLM-based judger for annotation. |
| Outcome: | The proposed method improves PRMs' ability to identify effective self-correction behaviors and reasoning based on erroneous steps. |
Learning to Reason Over Time: Timeline Self-Reflection for Improved Temporal Reasoning in Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models struggle with temporal reasoning, which requires processing time-related information such as event sequencing, durations, and inter-temporal relationships. |
| Approach: | They propose a framework that enhances the temporal reasoning abilities of Large Language Models (LLMs) by combining timeline construction with iterative self-reflection. |
| Outcome: | The proposed framework improves the temporal reasoning abilities of large language models and improves traceability of the inference process. |
Benchmarking Temporal Reasoning and Alignment Across Chinese Dynasties (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing temporal reasoning benchmarks rely on rule-based construction and lack contextual depth . a recent study found existing LLMs struggle with nuanced temporal understanding . |
| Approach: | a benchmark is designed to evaluate LLMs on temporal reasoning in Chinese dynasties. |
| Outcome: | a new benchmark evaluates LLMs on temporal reasoning across Chinese dynasties . it emphasizes cross-entity relationships, pairwise temporal alignment, contextualized and culturally-grounded reasoning . results show existing LLM benchmarks struggle with nuanced temporal understanding . |
Do LLMs Catch Their Own Mistakes? A Comprehensive Benchmark for Reflective Tool Use LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks primarily evaluate planning and execution success, overlooking the self-reflective dimension of tool use. |
| Approach: | They propose a benchmark to assess LLMs’ self-reflective reasoning in tool-augmented multi-turn dialogues. |
| Outcome: | The proposed benchmark covers 10 domains with 88 distinct APIs and 968 annotated dialogues, systematically injecting diverse error types arising from both user and assistant behavior. |
Beyond Timestamps: Bridging Forward and Backward Reasoning in Temporal Numerical and Relational Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Temporal Numerical and Relational reasoning rely on single-task evaluation paradigms. |
| Approach: | They propose a benchmark to evaluate Temporal Numerical and Relational reasoning . they propose QA and verification, and a Consistency Rate to quantify robustness . |
| Outcome: | The proposed framework evaluates both Temporal Numerical and Relational reasoning . it measures the alignment between QA and FV and the Consistency Rate measures robustness across these directions. |
Temporal Token Matters: Investigating and Interpreting the Consistency of Temporal Ordering in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit notable deficiencies in temporal reasoning . phrasing changes can lead LLMs to produce inconsistent outputs . |
| Approach: | They investigate the mechanistic interpretability of temporal ordering within event temporal reasoning . they identify a sparse subset of attention heads that are causally responsible for reasoning outcomes . |
| Outcome: | The proposed model outperforms other models in a variety of tasks and is validated by intervention-based experiments. |
Tracing the Light of Thought: A Probabilistic Self- and Cross-Consistency Verification Mechanism Improving Mathematical Reasoning in LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for evaluating reasoning paths are not efficient, but they are prone to errors. |
| Approach: | They propose a probabilistic self- and cross-consistency framework for mathematical reasoning that employs an accept-reject mechanism to encourage high-quality reasoning paths. |
| Outcome: | The proposed framework improves on 9 LLMs across 4 challenging benchmarks. |