CodeContests-O: Powering LLMs via Feedback-Driven Iterative Test Case Generation (2026.findings-acl)
Copied to clipboard
Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Kangwen Zhao, Dongyun Xue, Mingxiao Feng, Wengang Zhou, Houqiang Li
| Challenge: | Existing approaches to synthesize test cases using Large Language Models (LLMs) rely on the model’s intrinsic generation capabilities without external feedback, resulting in insufficiently diverse cases. |
| Approach: | They propose a feedback-driven iterative framework that leverages Large Language Models to generate initial test cases, execute them against known correct and incorrect solutions, and utilizes the failed results as feedback to guide the LLM in refining the test cases toward high fidelity and discriminability. |
| Outcome: | The proposed method outperforms the existing codecontests and codecontests+ models by 4.30% and 8.78%. |
Similar Papers
CodeContests+: High-Quality Test Case Generation for Competitive Programming (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Competitive programming has become a key task for training and evaluating large language models . but test cases of competitive programming problems are often difficult to obtain . |
| Approach: | They propose an LLM-based agent system that creates high-quality test cases for competitive programming problems. |
| Outcome: | The proposed system improves code tests on a CodeContests dataset with pass/fail labels. |
The Program Testing Ability of Large Language Models for Code (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent development of large language models (LLMs) for code shows promise in achieving code intelligence. |
| Approach: | They explore the ability of large language models to generate automated test cases . they show +11.77% and +4.22% higher code pass rates on HumanEval+ . |
| Outcome: | The proposed models show higher pass rates on humanEval+ compared with the current state-of-the-art models. |
S*: Test Time Scaling for Code Generation (2025.findings-emnlp)
Copied to clipboard
Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E. Gonzalez, Ion Stoica
| Challenge: | S* is the first hybrid test-time scaling framework that significantly improves the coverage and selection accuracy of generated code. |
| Approach: | They propose a hybrid test-time scaling framework that augments parallel scaling with sequential scaling to further increase the performance. |
| Outcome: | The proposed framework outperforms existing scaling approaches in large-scale modeling and reasoning models. |
DebateCoder: Towards Collective Intelligence of LLMs via Test Case Driven LLM Debate for Code Generation (2025.acl-long)
Copied to clipboard
Jizheng Chen, Kounianhua Du, Xinyi Dai, Weiming Zhang, Xihuai Wang, Yasheng Wang, Ruiming Tang, Weinan Zhang, Yong Yu
| Challenge: | Existing debate-based approaches to code generation are limited due to several reasons: 1) Reliance on different instances of the same LLM for debate, 2) under-utilization of test cases, and 3) reliance on third-party moderators for result consolidation and decision-making. |
| Approach: | They propose to use test cases to analyze code and identify bugs while opposing models generate test cases for each other to challenge each other's code during the debate process. |
| Outcome: | The proposed model collects intelligence of LLMs via test case-driven debate for code generation. |
TESTEVAL: Benchmarking Large Language Models for Test Case Generation (2025.findings-naacl)
Copied to clipboard
Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, Lei Ma
| Challenge: | Existing methods to generate test cases using large language models are limited in their ability to generate unit test cases. |
| Approach: | They propose a test case generation benchmark that uses large language models to generate unit test cases. |
| Outcome: | The proposed test case generation benchmarks compare LLMs with commercial and open-source LLM platforms and find that they lack the ability to comprehend program logic and execution paths. |
CodeJudge: Evaluating Code Generation with Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising performance in code generation, but how to reliably evaluate code generated by LLMs remains a challenging problem. |
| Approach: | They propose a framework that leverages Large Language Models to evaluate the semantic correctness of generated code without the need for test cases. |
| Outcome: | The proposed framework outperforms existing methods on four code generation datasets and five programming languages. |
Planning-Driven Programming: A Large Language Model Programming Workflow (2025.acl-long)
Copied to clipboard
| Challenge: | Recent research suggests continuous program refinements through visible tests to improve code generation accuracy in large language models (LLMs). |
| Approach: | They propose an LLM programming workflow to improve both initial code generation and subsequent refinements within a structured two-phase workflow. |
| Outcome: | The proposed workflow improves both initial code generation and subsequent refinements within a structured two-phase workflow. |
Turning the Tide: Repository-based Code Reflection (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Code large language models (LLMs) enhance programming by understanding and generating code across languages. |
| Approach: | a new benchmark evaluates code understanding and generation in repositories using code large language models. |
| Outcome: | The proposed model improves code understanding and generation in repositories by evaluating 1,888 test cases across 6 programming languages. |
Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure (2025.acl-short)
Copied to clipboard
| Challenge: | TestCase-Eval focuses on Fault Coverage and Fault Exposure tasks . authors provide insights into their strengths and limitations in generating effective test cases . correctness and robustness of algorithmic solutions hinge on quality of test suites . |
| Approach: | They introduce TestCase-Eval, a benchmark for systematic evaluation of LLMs in test-case generation. |
| Outcome: | The new benchmark measures the performance of LLMs in test-case generation. |
Dynamic Scaling of Unit Tests for Code Reward Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Existing large language models struggle to produce accurate responses on the first attempt for complex reasoning tasks like code generation. |
| Approach: | They propose a lightweight yet effective unit test generator that scales unit tests based on problem difficulty. |
| Outcome: | The proposed approach significantly improves performance on three benchmarks. |