CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Large Language Models often lack coverage for subtle corner cases . a substantial amount of effort has been applied to address this challenge . |
| Approach: | They propose a framework that generates adversarial test cases that expose latent vulnerabilities in code submissions. |
| Outcome: | The proposed framework improves the True Negative Rate (TNR) of existing datasets and generates superior adversarial cases on liveCodeBench. |
Similar Papers
CodeContests+: High-Quality Test Case Generation for Competitive Programming (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Competitive programming has become a key task for training and evaluating large language models . but test cases of competitive programming problems are often difficult to obtain . |
| Approach: | They propose an LLM-based agent system that creates high-quality test cases for competitive programming problems. |
| Outcome: | The proposed system improves code tests on a CodeContests dataset with pass/fail labels. |
UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models have enabled the development of coding agents for real-world code generation. |
| Approach: | They propose a novel LLM-driven test case generator that analyzes codebases and dependencies to generate test cases for real-world Python projects. |
| Outcome: | The proposed framework improves the performance of SWE-Bench by analyzing codebases and dependencies. |
RedCoder: Automated Multi-Turn Red Teaming for Code LLMs (2026.acl-long)
Copied to clipboard
Wenjie Jacky Mo, Qin Liu, Xiaofei Wen, Dongwon Jung, Hadi Askari, Wenxuan Zhou, Zhe Zhao, Muhao Chen
| Challenge: | Existing red-teaming approaches for code generation rely on extensive human effort and are prone to generating malicious code under adversarial environments. |
| Approach: | They propose a red-teaming agent that engages victim models in multi-turn conversations to elicit vulnerable code. |
| Outcome: | Experiments show that RedCoder outperforms red-teaming methods in inducing vulnerabilities in code generation. |
TESTEVAL: Benchmarking Large Language Models for Test Case Generation (2025.findings-naacl)
Copied to clipboard
Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, Lei Ma
| Challenge: | Existing methods to generate test cases using large language models are limited in their ability to generate unit test cases. |
| Approach: | They propose a test case generation benchmark that uses large language models to generate unit test cases. |
| Outcome: | The proposed test case generation benchmarks compare LLMs with commercial and open-source LLM platforms and find that they lack the ability to comprehend program logic and execution paths. |
CodeContests-O: Powering LLMs via Feedback-Driven Iterative Test Case Generation (2026.findings-acl)
Copied to clipboard
Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Kangwen Zhao, Dongyun Xue, Mingxiao Feng, Wengang Zhou, Houqiang Li
| Challenge: | Existing approaches to synthesize test cases using Large Language Models (LLMs) rely on the model’s intrinsic generation capabilities without external feedback, resulting in insufficiently diverse cases. |
| Approach: | They propose a feedback-driven iterative framework that leverages Large Language Models to generate initial test cases, execute them against known correct and incorrect solutions, and utilizes the failed results as feedback to guide the LLM in refining the test cases toward high fidelity and discriminability. |
| Outcome: | The proposed method outperforms the existing codecontests and codecontests+ models by 4.30% and 8.78%. |
LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs (2025.acl-long)
Copied to clipboard
Kaibo Liu, Zhenpeng Chen, Yiyang Liu, Jie M. Zhang, Mark Harman, Yudong Han, Yun Ma, Yihong Dong, Ge Li, Gang Huang
| Challenge: | TrickCatcher generates test cases that pass existing tests yet contain bugs . a recent study found that tricky bugs are not detected by test suites . |
| Approach: | They propose an LLM-powered approach to generating test cases for uncovering bugs in plausible programs . they use a PUT and specification to generate program variants, an input generator and an Llm to construct test inputs . |
| Outcome: | The proposed approach achieves recall, precision, and F1 scores that are 1.80, 2.65, and 1.66 . trickCatcher generates program variants based on the program under test and its specification . |
Toward Automated Robustness Evaluation of Mathematical Reasoning (2026.findings-acl)
Copied to clipboard
Yutao Hou, Zeguan Xiao, Fei Yu, Yihan Jiang, Ma Shuguang, Zhaoqian Dai, Hailiang Huang, Yun Chen, Guanhua Chen
| Challenge: | Existing robustness evaluations rely on hand-crafted templates or a limited set of perturbation rules, resulting in model failure. |
| Approach: | They propose a framework inspired by software stress testing that generates adversarial variants via a multi-round rewrite-verify loop, ensuring semantic consistency while successfully inducing model failure. |
| Outcome: | The proposed framework generates adversarial variants dynamically for each LLM, minimizing the risk of data contamination. |
Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure (2025.acl-short)
Copied to clipboard
| Challenge: | TestCase-Eval focuses on Fault Coverage and Fault Exposure tasks . authors provide insights into their strengths and limitations in generating effective test cases . correctness and robustness of algorithmic solutions hinge on quality of test suites . |
| Approach: | They introduce TestCase-Eval, a benchmark for systematic evaluation of LLMs in test-case generation. |
| Outcome: | The new benchmark measures the performance of LLMs in test-case generation. |
Beyond Superficial Tests: Adversarial Refinement for Reliable Property-Based Testing (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable proficiency in code generation, yet their application to Property-Based Testing (PBT) remains fraught with a superficiality gap. |
| Approach: | They propose an agentic framework that hardens software properties through Adversarial Refinement. |
| Outcome: | a new framework hardens software properties through Adversarial Refinement that detects and fixes bugs in top-tier libraries. |
AutoSUIT Bench - Automated Security UnIt Test Benchmark for LLM Coding (2026.findings-acl)
Copied to clipboard
Samuel Osebe, Fan Yang, Junyi Li, Yue Gu, Yongxin Wang, Satyapriya Krishna, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Weitong Ruan
| Challenge: | Large Language Models (LLMs) are evolving rapidly on code generation tasks. |
| Approach: | They propose to automate the vulnerability code benchmark creation with iterative auto validation. |
| Outcome: | The proposed benchmark covers 232 CWE categories across C/C++, Java, and Python languages. |