MBPP

Think Less, Code Better: Probing When Chain-of-Thought Hurts and How to Route Around It (2026.acl-srw)

Challenge:	Large Language Models excel in stand-alone code tasks but struggle with handling entire code repositories.
Approach:	They propose a system that integrates LLM agents with graph database interfaces extracted from code repositories.
Outcome:	The proposed system integrates LLM agents with graph database interfaces extracted from code repositories.

Copied to clipboard

Rajarshi Ghoshal, Debadri Basak, Salma Emad Mahmoud Abdelhalim, Pratibha Kaur Arora

Tree-of-Evolution: Tree-Structured Instruction Evolution for Code Generation in Large Language Models (2025.acl-long)

Challenge:	Chain-of-Thought (CoT) prompting is the dominant strategy for eliciting step-by-step reasoning in large language models, but its effect on code generation is poorly understood.
Approach:	They develop a chain-of-thought (CoT) prompting router that selects among 12 prompt styles via a single 84 ms forward pass.
Outcome:	The proposed model outperforms CoT in small models with a 84 ms forward pass.

Copied to clipboard

Ziyang Luo, Kaixin Li, Hongzhan Lin, Yuchen Tian, Mohan Kankanhalli, Jing Ma

LETI: Learning to Generate from Textual Interactions (2024.findings-naacl)

Challenge:	Data synthesis is a key research area in large language models (LLMs).
Approach:	They propose a framework that models code instruction synthesis process with a tree structure and optimization-driven evolution to alleviate constraints of unidirectional synthesis and randomness-driven generation.
Outcome:	The proposed framework outperforms open-weight code LLMs on five widely-used benchmarks.

Copied to clipboard

Xingyao Wang, Hao Peng, Reyhaneh Jabbarvand, Heng Ji

Reason-Code: Reliable Code Generation via Test-Driven Monte Carlo Tree Search (2026.acl-industry)

Challenge:	Existing techniques fine-tune on input-output pairs or with numerical rewards that gauge the output quality are not effective.
Approach:	They propose to fine-tune pre-trained language models with binary labels and a Python interpreter to get textual feedback from the inputs.
Outcome:	The proposed model outperforms the base model on unseen problems and achieves comparable or better performance on humanEval.

Copied to clipboard

Zixu Li, Zhiqi Peng

EvoAgentX: An Automated Framework for Evolving Agentic Workflows (2025.emnlp-demos)

Challenge:	Large Language Models (LLMs) are widely used for code generation, but their performance degrades on complex tasks.
Approach:	They propose an inference-time framework that formulates code generation as a search process guided by execution feedback.
Outcome:	The proposed framework improves reliability without paying full cost of additional sampling under strict latency budgets.

Copied to clipboard

Yingxu Wang, Siwei Liu, Jinyuan Fang, Zaiqiao Meng

Numerical Reasoning

Challenge:	Existing MAS frameworks often require manual workflow configuration and lack native support for dynamic evolution and performance optimization.
Approach:	They propose an open-source platform that automates generation, execution, and evolutionary optimization of multi-agent workflows.
Outcome:	The proposed platform automates generation, execution, and evolutionary optimization of multi-agent workflows.

Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step (2024.findings-acl)

Copied to clipboard

Li Zhong, Zilong Wang, Jingbo Shang

TaCIE: Enhancing Instruction Comprehension in Large Language Models through Task-Centred Instruction Evolution (2025.coling-main)

Challenge:	Large language models (LLMs) are leading progress in code generation, but they are underutilized in the literature.
Approach:	They propose a debugging framework that allows LLMs to refine their generated programs with the runtime execution information.
Outcome:	The proposed framework improves the baseline performance by 9.8% across the HumanEval, MBPP, and TransCoder benchmarks.

Copied to clipboard

Jiuding Yang, Shengyao Lu, Weidong Guo, Xiangyang Li, Kaitong Yang, Yu Xu, Di Niu

AMR-Evol: Adaptive Modular Response Evolution Elicits Better Knowledge Distillation for Large Language Models in Code Generation (2024.emnlp-main)

Challenge:	Existing methods for fine-tuning Large Language Models (LLMs) encounter performance limitations, impeding further enhancements in code generation tasks.
Approach:	They propose to combine two distinct prompts through a hybridization process to enhance the evolution of training prompts for code LLMs.
Outcome:	The proposed method significantly improves the performance of Code LLMs across five code generation benchmarks.

Copied to clipboard

Ziyang Luo, Xin Li, Hongzhan Lin, Jing Ma, Lidong Bing

Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024.acl-long)

Challenge:	proprietary large language models (LLMs) have demonstrated impressive code generation performance.
Approach:	They propose an adaptive module-based model that refines the direct response distillation process by modular decomposition and adaptive response evolution.
Outcome:	The proposed framework outperforms baseline model and code generation methods on three popular benchmarks.

Copied to clipboard

Baizhou Huang, Shuai Lu, Xiaojun Wan, Nan Duan

Self-Correcting Code Generation Using Small Language Models (2025.findings-emnlp)

Challenge:	Existing work utilizes verification properties to verify and re-rank solutions in a majority voting manner, but this assumption may not hold.
Approach:	They propose a multi-perspective self-consistency framework that incorporates both inter- and intra-consistency across outputs from multiple perspectives.
Outcome:	The proposed framework significantly boosts performance of foundation models on various benchmarks, including HumanEval (+15.91%), MBPP (+6.43%) and CodeContests (+9.37%).

Copied to clipboard

Jeonghun Cho, Deokhyung Kang, Hyounghun Kim, Gary Lee

CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models (2025.naacl-long)

Challenge:	a recent study has demonstrated that self-correction is a powerful tool for code generation, but whether it is effective for smaller models remains unexplored.
Approach:	They propose a method that trains small language models to maintain correct outputs while progressively correcting incorrect outputs as turns proceed.
Outcome:	The proposed approach improves the ability of small language models for multi-turn code correction.

Copied to clipboard

Jierui Li, Hung Le, Yingbo Zhou, Caiming Xiong, Silvio Savarese, Doyen Sahoo

Learning to Reason via Self-Iterative Process Feedback for Small Language Models (2025.coling-main)

Challenge:	coding tasks require generated code to be fully executable and functionally correct . current agentic approaches struggle with multi-stage planning, generating, and debugging .
Approach:	They propose a framework for LLM agents to efficiently explore the search space in different stages of the code generation process.
Outcome:	The proposed framework achieves top results on 7 code generation benchmarks and a 31.9% solving rate on the SWEBench benchmark.

Copied to clipboard

Kaiyuan Chen, Jin Wang, Xuejie Zhang

Instruction Fusion: Advancing Prompt Evolution through Hybridization (2024.acl-long)

Challenge:	Existing methods for enhancing SLMs’ reasoning depend on costly external signals, resulting in SLM overly confident with limited supervision signals.
Approach:	They propose to fine-tune and align SLMs using positive and negative feedback signals and introduce process supervision for rewards in preference alignment by sampling-based inference simulation and process reward models.
Outcome:	The proposed method improves Gemma-2B's performance on GSM8K and MBPP, and out-of-domain generalization capabilities on MMLU_Math and HumanEval.

Copied to clipboard

Weidong Guo, Jiuding Yang, Kaitong Yang, Xiangyang Li, Zhuwei Rao, Yu Xu, Di Niu

Text Generation

Challenge:	Existing methods for fine-tuning Large Language Models (LLMs) encounter performance limitations, impeding further enhancements in code generation tasks.
Approach:	They propose to combine two distinct prompts through a hybridization process to enhance the evolution of training prompts for code LLMs.
Outcome:	The proposed method significantly improves the performance of Code LLMs across five code generation benchmarks, namely HumanEval, HumanEva+, MBPP, mbap+ and MultiPL-E.

MapCoder: Multi-Agent Code Generation for Competitive Problem Solving (2024.acl-long)

Copied to clipboard

Md. Ashraful Islam, Mohammed Eunus Ali, Md Rizwan Parvez

CodeSim: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging (2025.findings-naacl)

Challenge:	Large language models (LLMs) have impressive proficiency in natural language processing, but performance in code generation tasks remains limited.
Approach:	They propose a framework that emulates the full cycle of program synthesis as observed in humans.
Outcome:	The proposed framework replicates the full cycle of program synthesis as observed in human developers.

Copied to clipboard

Md. Ashraful Islam, Mohammed Eunus Ali, Md Rizwan Parvez

Alignment with Fill-In-the-Middle for Enhancing Code Generation (2025.emnlp-main)

Challenge:	Large Language Models (LLMs) have made significant strides in code generation and problem solving.
Approach:	They propose a multi-agent code generation framework that integrates human-like perception to address the stages of program synthesis.
Outcome:	The proposed framework achieves state-of-the-art (pass@1) results and shows potential for even greater enhancement when cascaded with external debuggers.

Copied to clipboard

Houxing Ren, Zimu Lu, Weikang Shi, Haotian Hou, Yunqiao Yang, Ke Wang, Aojun Zhou, Junting Pan, Mingjie Zhan, Hongsheng Li

Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective (2024.findings-acl)

Challenge:	Existing methods for generating test cases with limited training data are not reliable and may be counterproductive.
Approach:	They propose a method that splits code snippets into smaller, granular blocks, creating more diverse DPO pairs from the same test cases.
Outcome:	The proposed approach shows significant improvements in code generation tasks on benchmark datasets such as HumanEval (+), MBPP (+), and APPS.

Copied to clipboard

Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou

Text Generation

Challenge:	Existing studies decompose complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants.
Approach:	They propose to use code comments as natural logic pivot between natural language and code language to boost the code generation ability of code LLMs.
Outcome:	The proposed method significantly improves the code pass rate on humanEval and MBPP, while the robustness of the logical comment decoding strategy is higher than the Chain-of-thoughts prompting.

Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision (2024.findings-emnlp)

Copied to clipboard

Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, Jingbo Shang

Reasoning

Challenge:	a method for process supervision has shown significant improvements in multi-step problem solving . despite the advances in process supervision, there are still easily observable mistakes in state-of-the-art LLMs.
Approach:	They propose a method for automating data curation by using a trained verifier to evaluate intermediate steps generated by a reasoner.
Outcome:	The proposed method improves the performance of PaLM 2 on math and coding tasks.

NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries (2024.findings-acl)

Copied to clipboard

Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Yuxiao Dong, Jie Tang

Green & Sustainable NLP

Challenge:	Large language models (LLMs) generate code for productive activities, but current benchmarks for code synthesis are oriented towards introductory tasks on algorithm and data science.
Approach:	They propose a code benchmark to mirror the complexity and variety of scenarios in real-world coding tasks.
Outcome:	The proposed benchmark improves on 39 large language models with close HumanEval scores and achieves an efficiency increase of more than 4 times.

ReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generation (2025.acl-long)

Copied to clipboard

Houxing Ren, Mingjie Zhan, Zhongyuan Wu, Aojun Zhou, Junting Pan, Hongsheng Li

LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models (2026.acl-long)

Challenge:	Existing methods to enhance code generation performance include integrating compiler feedback.
Approach:	They propose a method that integrates compiler feedback to improve one-off code generation performance.
Outcome:	The proposed method improves one-off code generation performance on three benchmarks and can be applied to other domains that focus on final results and require long reasoning paths.

Copied to clipboard

Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Zhou, Yankai Lin, Ji-Rong Wen, Chongxuan Li

Lifting Optimized Binaries to Canonical Compiler IR via Structure-Aware Retrieval and Iterative Verification (2026.acl-long)

Challenge:	Masked diffusion language models have achieved significant progress in language modeling . however, the systematic analysis and empirical validation of their alignment on general tasks remains underexplored.
Approach:	They propose a framework that analyzes the bias and variance of preference optimization loss and gradient based on Direct Preference Optimization.
Outcome:	The proposed model outperforms its SFT-only predecessor on general benchmarks . it consistently outperformed other strong language models and ARMs on general tasks .

Copied to clipboard

Xiaoao Zhu, Jie Ren, Zhiqiang Li, Jie Zheng, Zhanyong Tang, Zheng Wang

Information Retrieval

Challenge:	Existing methods for decompiling binary code are brittle due to compiler optimizations that distort control-flow and data-flow structure.
Approach:	They propose a system that lifts optimized binaries to canonical compiler intermediate representation (IR) BRIDGE uses control-flow-aware retrieval-augmented generation with feedback-driven verification .
Outcome:	The proposed system outperforms seven baselines on humanEval-Decompile and MBPP, lifting x86-64 and ARM64 binaries to LLVM IR.

ACECODER: Acing Coder RL via Automated Test-Case Synthesis (2025.acl-long)

Copied to clipboard

Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, Wenhu Chen

Planning-Driven Programming: A Large Language Model Programming Workflow (2025.acl-long)

Challenge:	Recent coder models have been driven by supervised fine-tuning, but the potential of reinforcement learning remains unexplored due to the lack of reliable reward data/model in the code domain.
Approach:	They propose a pipeline that generates extensive test-case pairs from existing code data and constructs preference pairs based on pass rates over sampled programs.
Outcome:	The proposed pipeline generates extensive (question, test-cases) pairs from existing code data and trains them with Bradley-Terry loss.

Copied to clipboard

Chao Lei, Yanchuan Chang, Nir Lipovetzky, Krista A. Ehinger

HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation Task (2025.findings-acl)

Challenge:	Recent research suggests continuous program refinements through visible tests to improve code generation accuracy in large language models (LLMs).
Approach:	They propose an LLM programming workflow to improve both initial code generation and subsequent refinements within a structured two-phase workflow.
Outcome:	The proposed workflow improves both initial code generation and subsequent refinements within a structured two-phase workflow.

Copied to clipboard

Zhaojian Yu, Yilun Zhao, Arman Cohan, Xiao-Ping Zhang

MasRouter: Learning to Route LLMs for Multi-Agent Systems (2025.acl-long)

Challenge:	Existing benchmarks for code generation tasks are inadequate, but performance declines on self-invoking tasks.
Approach:	They propose a general recipe for generating more challenging versions of existing benchmarks . they propose to use instruction-tuned models to evaluate LLMs on self-invoking code generation tasks .
Outcome:	The proposed model improves on humanEval and MBPP but on self-invoking code generation tasks.

Copied to clipboard

Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, Yiyan Qi

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024.findings-acl)

Challenge:	Multi-agent systems (MAS) powered by Large Language Models (LLMs) have been demonstrated to push the boundaries of LLM capabilities, yet they often face significant costs and challenges in dynamic LLM selection.
Approach:	They propose a multi-agent system routing solution that integrates all components of MAS into a unified routing framework.
Outcome:	The proposed solution is high-performing, cost-effective, and efficient . it reduces overhead by up to 52.07 compared to current methods on HumanEval .

Copied to clipboard

Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, Xiang Yue

ReCode: Robustness Evaluation of Code Generation Models (2023.acl-long)

Challenge:	OpenCodeInterpreter-33B provides a high level of performance for code generation, executing, and iterative refinement.
Approach:	They propose a family of open-source code systems for generating, executing, and iteratively refining code.
Outcome:	The OpenCodeInterpreter-33B performs well on humanEval, MBPP, and EvalPlus benchmarks.

Copied to clipboard

Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, Bing Xiang

Responsible & Trustworthy NLP

Challenge:	Existing work on robustness in text or code tasks has focused on classification, while robustness for code generation tasks is an uncharted area.
Approach:	They propose a robustness evaluation benchmark for code generation models that customizes over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format.
Outcome:	The proposed model performs better on human annotators and on SOTA models with human annnotators.

Grammar-Based Code Representation: Is It a Worthy Pursuit for LLMs? (2025.findings-acl)

Copied to clipboard

Qingyuan Liang, Zhao Zhang, Zeyu Sun, Zheng Lin, Qi Luo, Xiao Yueyi, Yizhou Chen, Yuqun Zhang, Haotian Zhang, Lu Zhang, Chenbin Chenbin, Yingfei Xiong

One-to-many testing for code generation from (just) natural language (2024.findings-emnlp)

Challenge:	Existing research demonstrates the effectiveness of grammar-based code representations in small-scale models, showing their ability to reduce syntax errors and enhance performance.
Approach:	They develop a series of billion-scale grammar-based code representations that incorporate grammar rules into the code generation process.
Outcome:	Experiments on HumanEval and MBPP show that grammar-based representations reduce syntax errors and improve performance even in billion-scale models.

Copied to clipboard

Mansi Uniyal, Mukul Singh, Gust Verbruggen, Sumit Gulwani, Vu Le

Code-SPA: Style Preference Alignment to Large Language Models for Effective and Robust Code Debugging (2025.findings-acl)

Challenge:	MBPP relies on test cases to generate the right signature, data contamination is a problem . adapted code generation benchmarks allow for the description to be underspecified with respect to syntactic properties of code.
Approach:	They propose a code generation benchmark that allows for the description to be underspecified with respect to syntactic properties of code.
Outcome:	The proposed model removes ambiguity about the semantics of the task from the descriptions and evaluates generated code on multiple sets of assertions to account for ambiguities in the syntax.

Copied to clipboard

Tengfei Wen, Xuanang Chen, Ben He, Le Sun

BiCSRouter: Bi-Level Cross-System Routing for Utility-Aware LLM Inference (2026.findings-acl)

Challenge:	Large language models (LLMs) have demonstrated impressive capabilities in coding tasks like code generation and debugging.
Approach:	They propose a method which aligns noisy code with the well-structured style familiar to LLMs, mitigating the impact of stylistic inconsistencies.
Outcome:	The proposed method improves debugging performance on poorly styled code across the HumanEval, MBPP and EvalPlus datasets.

Copied to clipboard

Mao Keyu, Eiki Murata, Ukyo Honda

Challenge:	Existing routing frameworks operate within a single computational paradigm . a cross-system routing framework that integrates two orthogonal regimes is proposed .
Approach:	They propose a cross-system routing framework that integrates two orthogonal regimes . they propose MBPP-based model that decomposes routing into intra-regime configuration selection and inter-regem system selection .
Outcome:	The proposed framework outperforms 15 representative baselines on MBPP and MATH benchmarks.

Specialization without Sparsity: Efficient and Expressive Split-Path Experts for LLM Fine-Tuning (2026.findings-acl)

Copied to clipboard

Ganghao Liu, Qin Zhou, Zhe Wang, Xuehan Lu, Haihua Huang, Yunfei Tong, Heng Tian

PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs (2024.findings-emnlp)

Challenge:	Parameter-efficient fine-tuning (PEFT) is a low-cost alternative to full fine-timing due to the massive overhead.
Approach:	They propose a Mixture-of-Experts approach that enhances specialization while maintaining low resource overhead.
Outcome:	The proposed approach outperforms or matches state-of-the-art methods on GLUE, GSM8K, MBPP, and a text rewriting task from SmolTalk.

Copied to clipboard

Ankit Yadav, Himanshu Beniwal, Mayank Singh

EpiCaR: Knowing What You Don’t Know Matters for Better Reasoning in LLMs (2026.acl-long)

Challenge:	HumanEval and MBPP are two popular benchmarks for Python code generation.
Approach:	They propose a large-scale human evaluation of two popular Python benchmarks . they propose 185 hand-crafted prompts in a balanced representation of 38 programming concepts across diverse difficulty levels.
Outcome:	The proposed benchmarks show a critical bias towards a limited set of programming concepts, neglecting most of the other concepts entirely.

Copied to clipboard

Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Taesup Kim

Reasoning

Challenge:	Existing approaches to improving reasoning abilities of large language models incur a significant calibration cost.
Approach:	They propose an epistemic learning problem that integrates reasoning and calibration into an iterative supervised training framework.
Outcome:	The proposed method achieves Pareto-superiority over standard baselines in accuracy and calibration.

Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement (2025.findings-acl)

Copied to clipboard

Xiaoqing Zhang, Yuhan Liu, Flood Sung, Xiuying Chen, Shuo Shang, Rui Yan

Can Language Models Replace Programmers for Coding? REPOCOD Says ‘Not Yet’ (2025.acl-long)

Challenge:	Recent research indicates that large language models (LLMs) have demonstrated remark-able capabilities in various programming-related domains, such as code generation and code refinement.
Approach:	They propose a framework that combines exploration with refinement to reduce test-time computation overhead.
Outcome:	The proposed framework outperforms SOTA and AgentCoder on humanEval and MBPP benchmarks while reducing test-time computation overhead and scalability.

Copied to clipboard

Shanchao Liang, Nan Jiang, Yiran Hu, Lin Tan

SACL: Understanding and Combating Textual Bias in Code Retrieval with Semantic-Augmented Reranking and Localization (2025.findings-emnlp)

Challenge:	Existing benchmarks for code generation use short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks.
Approach:	They propose a Python code-generation benchmark that contains 980 whole-function generation tasks with realistic dependencies from 11 popular projects.
Outcome:	The proposed benchmarks are short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks.

Copied to clipboard

Dhruv Gupta, Gayathri Ganesh Lakshmy, Yiqing Xie

Information Retrieval

Challenge:	a recent study shows that code retrievers exhibit a strong bias towards well-documented code .
Approach:	They propose a framework that augments textual information with semantic information to mask specific features while preserving code functionality.
Outcome:	The proposed framework enhances textual information and reduces bias by augmenting code or structural knowledge with semantic information.

TA-GRPO-d: Trajectory-Aware GRPO for Optimizing Denoising Trajectories in Diffusion LLMs (2026.acl-long)

Copied to clipboard

Gyunyeop Kim, Sangwoo Kang

ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning (2026.findings-acl)

Challenge:	Existing dLLMs rely on fixed denoising schedules and cannot learn efficient unmasking orders.
Approach:	They propose a framework that transforms dLLM decoding into a trajectory-aware policy . it uses a confidence-gated denoising strategy that decides which tokens to unmask .
Outcome:	The proposed model can learn which tokens to unmask and how many to unmak per step . it can learn the output quality and efficiency of the decoding path itself .

Copied to clipboard

Juyong Jiang, Jiasi Shen, Sunghun Kim, Kang Min Yoo, Jeonghoon Kim, Sungju Kim