Scaling Evaluation-Time Compute with Reasoning Models as Evaluators (2026.findings-acl)
Copied to clipboard
Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Minkyeong Moon, Carolin Lawrence, Kiril Gashteovski, Julia Hockenmaier, Graham Neubig, Sean Welleck
| Challenge: | Language model (LM) evaluators that generate chain-of-thought reasoning are widely used for the assessment of LM responses. |
| Approach: | They investigate whether increasing LMs' "thinking" time through scaling test-time compute can improve an LM's evaluation capability. |
| Outcome: | The proposed reasoning models improve evaluation performance monotonically with the number of reasoning tokens generated, mirroring trends seen in LM reasoning. |
Similar Papers
Logical Reasoning with Outcome Reward Models for Test-Time Scaling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), but it is under-explored in deductive reasoning. |
| Approach: | They propose to use Chain-of-Thought to generate data using single and multiple samples to train ORMs. |
| Outcome: | The proposed model expands the type of errors covered in the training dataset, covering previously unexplored error types. |
ReEfBench: Quantifying the Reasoning Efficiency of LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for Chain-of-Thought evaluations do not distinguish between genuine reasoning and mere verbosity. |
| Approach: | They propose a framework for the non-intrusive, comprehensive process-centric evaluation of reasoning grounded in First-Order Logic. |
| Outcome: | The proposed framework identifies four distinct behavioral prototypes and diagnoses the failure modes. |
Thought calibration: Efficient and confident test-time scaling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for teaching language models to be economical with their token budgets have failed to achieve the desired results. |
| Approach: | They propose to calibrate a language model's growing body of thoughts to determine when new reasoning plateaus. |
| Outcome: | The proposed framework preserves model performance with up to 60% reduction in thinking tokens on in-distribution data, and up to 20% in out-of-difference data. |
ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods for test-time scaling are limited. |
| Approach: | They propose an adaptive resolution-aware scaling evaluation metric specifically designed to assess the test-time scaling effectiveness of large reasoning models. |
| Outcome: | The proposed metric provides a reliable and fine-grained measurement of test-time scaling capabilities, revealing significant variations in scaling efficiency across models. |
Current Advances in LLM Reasoning (2026.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial examines comprehensive evaluation strategies to assess the reasoning abilities of large language models (LLMs) advanced inference time methods and post-training methods that aim to make LLMs think more like humans are discussed in this tutorial. |
| Approach: | This tutorial explores comprehensive evaluation strategies to assess the reasoning abilities of large language models (LLMs) and discusses two types of methods to improve models’ reasoning: advanced inference time methods, structured and self-improvement inference methods, and post-training methods, such as RLHF, DPO, and GRPO. |
| Outcome: | This tutorial examines evaluation strategies to assess the reasoning abilities of large language models and discusses two types of methods to improve models’ reasoning. |
Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning? (2025.acl-long)
Copied to clipboard
Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Z.y. Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, Bo Zheng
| Challenge: | Recent advances in o1-like models have generated long Chain-of-Thought reasoning steps to improve the reasoning abilities of existing Large Language Models (LLMs). |
| Approach: | They propose a DeltaBench to analyze the quality and effectiveness of o1-like models and measure their ability to detect errors in long COT reasoning. |
| Outcome: | The proposed model can detect errors in long COT reasoning. |
When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing research implicitly assumes that longer thinking leads to better results . a recent study suggests that test-time compute scaling is more effective than model scaling . |
| Approach: | They challenge the assumption that longer thinking yields better results . they show that models exhibit overthinking and marginal returns diminish at higher budgets . |
| Outcome: | The proposed framework reduces computation significantly while maintaining comparable accuracy. |
Z1: Efficient Test-time Scaling with Code (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can achieve enhanced complex problem-solving through test-time computing scaling, but this often entails longer contexts and numerous reasoning token costs. |
| Approach: | They propose an efficient test-time scaling method that trains LLMs on code-related reasoning trajectories and a novel Shifted Thinking Window to mitigate overthinking overhead. |
| Outcome: | The proposed method reduces overthinking overhead while maintaining performance. |
Test-Time Scaling of Reasoning Models for Machine Translation (2026.eacl-long)
Copied to clipboard
| Challenge: | Using TTS, Reasoning Models (RMs) are able to perform tasks such as math and coding with limited results. |
| Approach: | They evaluate 12 Reasoning Models across a diverse suite of MT benchmarks, examining three scenarios: direct translation, forced-reasoning extrapolation, and post-editing. |
| Outcome: | The proposed approach improves translation quality on three domains, with inconsistent results for general-purpose RMs and performance plateauing. |
Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering (2025.acl-short)
Copied to clipboard
| Challenge: | Existing evaluations of test-time scaling assume that a reasoning system should always give an answer to any question provided. |
| Approach: | They propose to increase compute budget at inference time to increase confidence in correct responses by considering settings with non-zero levels of response risk. |
| Outcome: | The proposed model can answer more questions correctly and have higher confidence in correct responses. |