Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Models (2026.acl-long)
Copied to clipboard
Mehrzad Samadi, Aleksander Ficek, Sean Narenthiran, Siddhartha Jain, Wasi Uddin Ahmad, Somshubra Majumdar, Vahid Noroozi, Boris Ginsburg
| Challenge: | Competitive programming has become a rigorous benchmark for evaluating the reasoning and problem-solving capabilities of large language models (LLMs). |
| Approach: | They propose a scalable and reproducible test-time compute framework that achieves IOI gold-level performance using open-weight models. |
| Outcome: | The proposed framework achieves IOI gold-level performance using open-weight models . it scales consistently with available compute, narrowing the gap between open and closed systems. |
Similar Papers
Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming (2026.findings-acl)
Copied to clipboard
Sama Hadhoud, Alaa Elsetohy, Frederikus Hudi, Jan Christian Blaise Cruz, Steven Halim, Alham Fikri Aji
| Challenge: | Existing evaluations conflate algorithmic reasoning with code-level implementation. |
| Approach: | They propose to center editorials in both solution generation and evaluation . they propose to compare editorials to gold standards and validate an LLM-as-a-judge protocol . |
| Outcome: | The proposed approach improves solve rates on some LLMs with gold editorials . but the gap between gold and generated editorials shows bottlenecks in implementation . |
GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may encourage cherry-picking favored settings and for better results. |
| Approach: | They propose an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals that systematically evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
| Outcome: | The evaluation suite is built on top of OpenAI Evals and evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Assertions have been the de facto collateral for hardware for over a decade. |
| Approach: | They propose a benchmark to evaluate LLMs’ effectiveness for assertion generation quantitatively. |
| Outcome: | The proposed benchmark compares state-of-the-art LLMs with existing benchmarks and shows that they generate higher fractions of functionally correct assertions. |
Ranking Reasoning LLMs under Test-Time Scaling (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as general-purpose reasoning systems for tasks such as programming and mathematical problem solving. |
| Approach: | They formalize dense benchmark ranking under test-time scaling and introduce a library that implements statistical ranking methods such as paired-comparison models, item response theory, voting rules, graph- and spectral-based methods. |
| Outcome: | The proposed method is based on paired-comparison models, item response theory (IRT) models, voting rules, graph- and spectral-based methods. |
AMO-Bench: Large Language Models Still Struggle in High School Math Competitions (2026.findings-acl)
Copied to clipboard
Junlin Liu, Shengnan An, Shuang Zhou, Dan Ma, Yehao Lin, Xinxuan Lv, Xuanlin Wang, Xiaoyu Li, Ziwen Wang, Xuezhi Cao, Xunliang Cai
| Challenge: | Existing benchmarks for mathematical reasoning are becoming less effective due to performance saturation. |
| Approach: | They propose to use a mathematical reasoning benchmark with Olympiad difficulty to evaluate top-tier LLMs. |
| Outcome: | The proposed benchmarks are cross-validated by experts to meet IMO difficulty standards and entirely original problems to prevent performance leakages from data memorization. |
ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning (2026.acl-demo)
Copied to clipboard
Vladislav Smirnov, Quang-Chieu Nguyen, Sergey Senichev, Minh Ngoc Ta, Ekaterina Fadeeva, Artem Vazhentsev, Daria Galimzianova, Nikolai Rozanov, Viktor Mazanov, Jingwei Ni, Tianyi Wu, Igor Kiselev, Mrinmaya Sachan, Iryna Gurevych, Preslav Nakov, Timothy Baldwin, Artem Shelmanov
| Challenge: | Existing TTC scaling strategies and reasoning scorers are fragmented and evaluated under inconsistent protocols. |
| Approach: | They propose a framework for seamless test-time compute scaling of large language model reasoning . they use a modular Python library to implement state-of-the-art scaling strategy and scorer families . |
| Outcome: | The proposed framework evaluates performance and computational efficiency on mathematical and coding tasks. |
TESTEVAL: Benchmarking Large Language Models for Test Case Generation (2025.findings-naacl)
Copied to clipboard
Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, Lei Ma
| Challenge: | Existing methods to generate test cases using large language models are limited in their ability to generate unit test cases. |
| Approach: | They propose a test case generation benchmark that uses large language models to generate unit test cases. |
| Outcome: | The proposed test case generation benchmarks compare LLMs with commercial and open-source LLM platforms and find that they lack the ability to comprehend program logic and execution paths. |
The Program Testing Ability of Large Language Models for Code (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent development of large language models (LLMs) for code shows promise in achieving code intelligence. |
| Approach: | They explore the ability of large language models to generate automated test cases . they show +11.77% and +4.22% higher code pass rates on HumanEval+ . |
| Outcome: | The proposed models show higher pass rates on humanEval+ compared with the current state-of-the-art models. |
ICE-Score: Instructing Large Language Models to Evaluate Code (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in the field of natural language generation have facilitated the use of large language models to assess the quality of generated text. |
| Approach: | They propose a new evaluation metric by instructing large language models for code assessments using a set of programming languages. |
| Outcome: | The proposed evaluation metric surpasses state-of-the-art metrics for code generation, delivering high levels of accuracy and consistency across programming languages and tasks. |
CodeContests-O: Powering LLMs via Feedback-Driven Iterative Test Case Generation (2026.findings-acl)
Copied to clipboard
Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Kangwen Zhao, Dongyun Xue, Mingxiao Feng, Wengang Zhou, Houqiang Li
| Challenge: | Existing approaches to synthesize test cases using Large Language Models (LLMs) rely on the model’s intrinsic generation capabilities without external feedback, resulting in insufficiently diverse cases. |
| Approach: | They propose a feedback-driven iterative framework that leverages Large Language Models to generate initial test cases, execute them against known correct and incorrect solutions, and utilizes the failed results as feedback to guide the LLM in refining the test cases toward high fidelity and discriminability. |
| Outcome: | The proposed method outperforms the existing codecontests and codecontests+ models by 4.30% and 8.78%. |