HUMANEVAL

Think Less, Code Better: Probing When Chain-of-Thought Hurts and How to Route Around It (2026.acl-srw)

Challenge:	Large Language Models excel in stand-alone code tasks but struggle with handling entire code repositories.
Approach:	They propose a system that integrates LLM agents with graph database interfaces extracted from code repositories.
Outcome:	The proposed system integrates LLM agents with graph database interfaces extracted from code repositories.

Copied to clipboard

Rajarshi Ghoshal, Debadri Basak, Salma Emad Mahmoud Abdelhalim, Pratibha Kaur Arora

Tree-of-Evolution: Tree-Structured Instruction Evolution for Code Generation in Large Language Models (2025.acl-long)

Challenge:	Chain-of-Thought (CoT) prompting is the dominant strategy for eliciting step-by-step reasoning in large language models, but its effect on code generation is poorly understood.
Approach:	They develop a chain-of-thought (CoT) prompting router that selects among 12 prompt styles via a single 84 ms forward pass.
Outcome:	The proposed model outperforms CoT in small models with a 84 ms forward pass.

Copied to clipboard

Ziyang Luo, Kaixin Li, Hongzhan Lin, Yuchen Tian, Mohan Kankanhalli, Jing Ma

LETI: Learning to Generate from Textual Interactions (2024.findings-naacl)

Challenge:	Data synthesis is a key research area in large language models (LLMs).
Approach:	They propose a framework that models code instruction synthesis process with a tree structure and optimization-driven evolution to alleviate constraints of unidirectional synthesis and randomness-driven generation.
Outcome:	The proposed framework outperforms open-weight code LLMs on five widely-used benchmarks.

Copied to clipboard

Xingyao Wang, Hao Peng, Reyhaneh Jabbarvand, Heng Ji

Reason-Code: Reliable Code Generation via Test-Driven Monte Carlo Tree Search (2026.acl-industry)

Challenge:	Existing techniques fine-tune on input-output pairs or with numerical rewards that gauge the output quality are not effective.
Approach:	They propose to fine-tune pre-trained language models with binary labels and a Python interpreter to get textual feedback from the inputs.
Outcome:	The proposed model outperforms the base model on unseen problems and achieves comparable or better performance on humanEval.

Copied to clipboard

Zixu Li, Zhiqi Peng

Self-Edit: Fault-Aware Code Editor for Code Generation (2023.acl-long)

Challenge:	Large Language Models (LLMs) are widely used for code generation, but their performance degrades on complex tasks.
Approach:	They propose an inference-time framework that formulates code generation as a search process guided by execution feedback.
Outcome:	The proposed framework improves reliability without paying full cost of additional sampling under strict latency budgets.

Copied to clipboard

Kechi Zhang, Zhuo Li, Jia Li, Ge Li, Zhi Jin

Rethinking Repetition Problems of LLMs in Code Generation (2025.acl-long)

Challenge:	Existing Large language models (LLMs) have low pass rates and accuracy on competitive programming tasks.
Approach:	They propose a generate-and-edit approach that uses execution results of generated code from LLMs to improve code quality on competitive programming tasks.
Outcome:	The proposed method improves pass@1 by 89% on APPS-dev, 31% on apps-test, and 48% on HumanEval over nine popular code generation LLMs with parameter sizes ranging from 110M to 175B.

Copied to clipboard

Yihong Dong, Yuchen Liu, Xue Jiang, Bin Gu, Zhi Jin, Ge Li

Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step (2024.findings-acl)

Challenge:	Recent studies have focused on content repetition, but structural repetition is a more prevalent problem in code generation.
Approach:	They propose a decoding approach that eliminates repetition problems in code generation by identifying grammar rules and strategically decaying the likelihood of critical tokens that contribute to repetitions.
Outcome:	The proposed approach outperforms baselines and humanEval benchmarks on CodeRepetEval dataset and MBPP benchmarks, effectively reducing repetitions and enhancing the quality of generated code.

Copied to clipboard

Li Zhong, Zilong Wang, Jingbo Shang

Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Prompting (2025.naacl-industry)

Challenge:	Large language models (LLMs) are leading progress in code generation, but they are underutilized in the literature.
Approach:	They propose a debugging framework that allows LLMs to refine their generated programs with the runtime execution information.
Outcome:	The proposed framework improves the baseline performance by 9.8% across the HumanEval, MBPP, and TransCoder benchmarks.

Copied to clipboard

Emmanuel Aboah Boateng, Cassiano O Becker, Nabiha Asghar, Kabir Walia, Ashwin Srinivasan, Ehi Nosakhare, Soundararajan Srinivasan, Victor Dibia

TaCIE: Enhancing Instruction Comprehension in Large Language Models through Task-Centred Instruction Evolution (2025.coling-main)

Challenge:	Concept Distillation (CD) is an automated prompt optimization technique for enhancing weaker models on complex tasks.
Approach:	They propose an automatic prompt optimization technique for enhancing weaker models on complex tasks using a base prompt and a strong model to generate reasons for these mistakes.
Outcome:	The proposed technique improves weaker models on NL2Code and mathematical reasoning tasks, while preserving performance.

Copied to clipboard

Jiuding Yang, Shengyao Lu, Weidong Guo, Xiangyang Li, Kaitong Yang, Yu Xu, Di Niu

AMR-Evol: Adaptive Modular Response Evolution Elicits Better Knowledge Distillation for Large Language Models in Code Generation (2024.emnlp-main)

Challenge:	Existing methods for fine-tuning Large Language Models (LLMs) encounter performance limitations, impeding further enhancements in code generation tasks.
Approach:	They propose to combine two distinct prompts through a hybridization process to enhance the evolution of training prompts for code LLMs.
Outcome:	The proposed method significantly improves the performance of Code LLMs across five code generation benchmarks.

Copied to clipboard

Ziyang Luo, Xin Li, Hongzhan Lin, Jing Ma, Lidong Bing

CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023.emnlp-main)

Challenge:	proprietary large language models (LLMs) have demonstrated impressive code generation performance.
Approach:	They propose an adaptive module-based model that refines the direct response distillation process by modular decomposition and adaptive response evolution.
Outcome:	The proposed framework outperforms baseline model and code generation methods on three popular benchmarks.

Copied to clipboard

Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, Steven Hoi

Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024.acl-long)

Challenge:	Existing code LLMs adopt a specific architecture or rely on a unified encoder-decoder network for downstream tasks, lacking flexibility to operate in the optimal architecture for a particular task.
Approach:	They propose to initialize code LLMs with frozen off-the-shelf LLM and explore instruction-tuning to align with natural language instructions.
Outcome:	The proposed model outperforms open-source LLMs on 20 code-related benchmarks.

Copied to clipboard

Baizhou Huang, Shuai Lu, Xiaojun Wan, Nan Duan

Self-Correcting Code Generation Using Small Language Models (2025.findings-emnlp)

Challenge:	Existing work utilizes verification properties to verify and re-rank solutions in a majority voting manner, but this assumption may not hold.
Approach:	They propose a multi-perspective self-consistency framework that incorporates both inter- and intra-consistency across outputs from multiple perspectives.
Outcome:	The proposed framework significantly boosts performance of foundation models on various benchmarks, including HumanEval (+15.91%), MBPP (+6.43%) and CodeContests (+9.37%).

Copied to clipboard

Jeonghun Cho, Deokhyung Kang, Hyounghun Kim, Gary Lee

PairCoder: Pair Programming-Inspired Two-Agent Collaboration for Code Generation (2026.findings-acl)

Challenge:	a recent study has demonstrated that self-correction is a powerful tool for code generation, but whether it is effective for smaller models remains unexplored.
Approach:	They propose a method that trains small language models to maintain correct outputs while progressively correcting incorrect outputs as turns proceed.
Outcome:	The proposed approach improves the ability of small language models for multi-turn code correction.

Copied to clipboard

Junhao Chen, Xiang Li, Yibin Xu, Yuehan Cui, Fangsheng Weng, Hao Zhao, Fei Ma, Qi Tian

CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models (2025.naacl-long)

Challenge:	Existing multi agent frameworks for large language models are brittle on code generation tasks.
Approach:	They propose a framework that brings pair programming to autonomous LLM collaboration.
Outcome:	Using PairCoder, large language models achieve better results on code generation tasks and reduce token usage by 40% to 70% on eight representative backbones.

Copied to clipboard

Jierui Li, Hung Le, Yingbo Zhou, Caiming Xiong, Silvio Savarese, Doyen Sahoo

LLM4Decompile: Decompiling Binary Code with Large Language Models (2024.emnlp-main)

Challenge:	coding tasks require generated code to be fully executable and functionally correct . current agentic approaches struggle with multi-stage planning, generating, and debugging .
Approach:	They propose a framework for LLM agents to efficiently explore the search space in different stages of the code generation process.
Outcome:	The proposed framework achieves top results on 7 code generation benchmarks and a 31.9% solving rate on the SWEBench benchmark.

Copied to clipboard

Hanzhuo Tan, Qi Luo, Jing Li, Yuqun Zhang

Learning to Reason via Self-Iterative Process Feedback for Small Language Models (2025.coling-main)

Challenge:	Decompilation aims to convert binary code to high-level source code, but traditional tools like Ghidra often produce results that are difficult to read and execute.
Approach:	They propose an open-source LLM series trained to decompile binary code . they optimize the LLM training process and introduce the Llm4Decompile-End models .
Outcome:	The proposed models outperform GPT-4o and Ghidra on the HumanEval and ExeBench benchmarks by over 100% in terms of re-executability rate.

Copied to clipboard

Kaiyuan Chen, Jin Wang, Xuejie Zhang

Instruction Fusion: Advancing Prompt Evolution through Hybridization (2024.acl-long)

Challenge:	Existing methods for enhancing SLMs’ reasoning depend on costly external signals, resulting in SLM overly confident with limited supervision signals.
Approach:	They propose to fine-tune and align SLMs using positive and negative feedback signals and introduce process supervision for rewards in preference alignment by sampling-based inference simulation and process reward models.
Outcome:	The proposed method improves Gemma-2B's performance on GSM8K and MBPP, and out-of-domain generalization capabilities on MMLU_Math and HumanEval.

Copied to clipboard

Weidong Guo, Jiuding Yang, Kaitong Yang, Xiangyang Li, Zhuwei Rao, Yu Xu, Di Niu

Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packs (2025.naacl-long)

Challenge:	Existing methods for fine-tuning Large Language Models (LLMs) encounter performance limitations, impeding further enhancements in code generation tasks.
Approach:	They propose to combine two distinct prompts through a hybridization process to enhance the evolution of training prompts for code LLMs.
Outcome:	The proposed method significantly improves the performance of Code LLMs across five code generation benchmarks, namely HumanEval, HumanEva+, MBPP, mbap+ and MultiPL-E.

Copied to clipboard

Jiancheng Dong, Lei Jiang, Wei Jin, Lu Cheng

DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuning (2024.acl-long)

Challenge:	Randomly concatenating data points can lead to cross-contamination due to the significant difference in their subject matter.
Approach:	They propose a method that randomly concatenates data of varying lengths until reaching the designed maximum length to optimize context length and reduce padding.
Outcome:	The proposed method significantly improves performance on GSM8K and HumanEval, and also improves fairness and accuracy by 15%.

Copied to clipboard

Yejie Wang, Keqing He, Guanting Dong, Pei Wang, Weihao Zeng, Muxi Diao, Weiran Xu, Jingang Wang, Mengdi Zhang, Xunliang Cai

Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pre-training (2026.findings-eacl)

Challenge:	Numerous code large language models (LLMs) have been proposed to enhance code generation performance.
Approach:	They propose a diverse instruction model DolphCoder with self-evaluating for code generation that learns diverse instruction targets and combines a code evaluation objective to enhance its code generation ability.
Outcome:	The proposed model achieves superior performance on the HumanEval and MBPP benchmarks, demonstrating new insights for future code instruction tuning work.

Copied to clipboard

Jeffrey Li, Joshua P Gardner, Doug Kang, Fangping Shi, Karanjeet Singh, Chun-Liang Li, Herumb Shandilya, David Leo Wright Hall, Oncel Tuzel, Percy Liang, Ludwig Schmidt, Hadi Pouransari, Fartash Faghri

Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks (2025.acl-long)

Challenge:	Existing open-source datasets predominantly apply a single fixed extractor to all webpages.
Approach:	They propose to take a Union over different extractors to improve model performance . they show that extractor choice can significantly impact downstream task performance based on content type .
Outcome:	The proposed approach can increase the token yield of DCLM-Baseline by 71% while maintaining benchmark performance.

Copied to clipboard

Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Xun Wang, Si-Qing Chen, Michael J. Wooldridge, Janet B. Pierrehumbert, Furu Wei

When KV Cache Reuse Fails in Multi-Agent Systems: Cross-Candidate Interaction is Crucial for LLM Judges (2026.acl-long)

Challenge:	a study aims to assess the fairness and robustness of Large Language Models in dialectal queries . speakers of "non-standard" dialects are known to experience implicit and explicit discrimination .
Approach:	They propose to use a benchmark to assess the fairness of large language models in dialects . they hire speakers with computer science backgrounds to rewrite seven popular benchmarks based on AAVE .
Outcome:	The proposed benchmarks show that most models show significant brittleness and unfairness to queries in AAVE.

Copied to clipboard

Sichu Liang, Zhenglin Wang, null Chujiajia, Pengfei Xia, Hui Zang, Deyu Zhou

Challenge:	Multi-agent LLMs generate multiple candidate responses that are aggregated by an LLM judge.
Approach:	They propose to advocate KV cache reuse across partially shared contexts and report substantial speedups for generation agents.
Outcome:	The proposed reuse strategies weaken cross-candidate attention, especially for later candidate blocks, and highlight judge-centric inference as a distinct regime that requires dedicated, risk-aware system design.

Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking (2025.emnlp-main)

Copied to clipboard

Tianle Gu, Zongqi Wang, Kexin Huang, Yuanqi Yao, Xiangliang Zhang, Yujiu Yang, Xiuying Chen

Dynamic Scaling of Unit Tests for Code Reward Modeling (2025.acl-long)

Challenge:	Existing methods to watermark low-entropy content are expensive and risky . IE reduces parameter size by 99% while achieving performance on par with state-of-the-art methods .
Approach:	They propose a logit-based watermarking paradigm that uses entropy-based features to predict whether the next token is high or low.
Outcome:	The proposed method reduces parameter size by 99% while achieving performance on par with state-of-the-art methods.

Copied to clipboard

Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, Jie Tang

Green & Sustainable NLP

Challenge:	Existing large language models struggle to produce accurate responses on the first attempt for complex reasoning tasks like code generation.
Approach:	They propose a lightweight yet effective unit test generator that scales unit tests based on problem difficulty.
Outcome:	The proposed approach significantly improves performance on three benchmarks.

SolidCoder: Bridging the Mental-Reality Gap in LLM Code Generation through Concrete Execution (2026.findings-acl)

Copied to clipboard

Woojin Lee, Jin-Xia Huang

Automatic Instruction Evolving for Large Language Models (2024.emnlp-main)

Challenge:	State-of-the-art code generation frameworks rely on mental simulations to validate buggy code.
Approach:	They propose a mental-reality gap between mental simulation and actual execution . they propose sandboxed execution with a simple principle: don't imagine—execute .
Outcome:	The proposed framework achieves state-of-the-art pass@1 performance on humanEval, CodeContests and APPS.

Copied to clipboard

Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, Weizhu Chen

Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation (2023.emnlp-main)

Challenge:	Evol-Instruct is an end-to-end framework that evolves instruction datasets without human effort.
Approach:	They propose an end-to-end framework that evolves instruction datasets without human effort by analyzing and analyzing evolutionary strategies for the given instruction data.
Outcome:	The proposed method outperforms human-designed methods on various benchmarks including MT-Bench, AlpacaEval, GSM8K, and HumanEval.

Copied to clipboard

Hailin Chen, Amrita Saha, Steven Hoi, Shafiq Joty

Alignment with Fill-In-the-Middle for Enhancing Code Generation (2025.emnlp-main)

Challenge:	Recent studies have shown that close-sourced LLMs lack the ability to integrate into real-world applications due to their high associated costs and ethical concerns.
Approach:	They propose to use student model to refine its own solution by querying ChatGPT to generate task instruction and solution pairs and querying data to refine model.
Outcome:	The proposed model outperforms standard distillation with only one third of the data.

Copied to clipboard

Houxing Ren, Zimu Lu, Weikang Shi, Haotian Hou, Yunqiao Yang, Ke Wang, Aojun Zhou, Junting Pan, Mingjie Zhan, Hongsheng Li

Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective (2024.findings-acl)

Challenge:	Existing methods for generating test cases with limited training data are not reliable and may be counterproductive.
Approach:	They propose a method that splits code snippets into smaller, granular blocks, creating more diverse DPO pairs from the same test cases.
Outcome:	The proposed approach shows significant improvements in code generation tasks on benchmark datasets such as HumanEval (+), MBPP (+), and APPS.

Copied to clipboard

Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou

Teaching LLMs to Plan, Not Just Solve: Plan Learning Boosts LLMs Generalization in Reasoning Tasks (2025.findings-emnlp)

Challenge:	Existing studies decompose complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants.
Approach:	They propose to use code comments as natural logic pivot between natural language and code language to boost the code generation ability of code LLMs.
Outcome:	The proposed method significantly improves the code pass rate on humanEval and MBPP, while the robustness of the logical comment decoding strategy is higher than the Chain-of-thoughts prompting.

Copied to clipboard

Tianlong Wang, Junzhe Chen, Weibin Liao, Xueting Han, Jing Bai

Reasoning

Challenge:	Existing methods for reinforcement learning (RL) on self-generated data are limited in many domains.
Approach:	a new framework combines plan-based search with Step-level Advantage Preference Optimization to optimize plan learning.
Outcome:	The proposed framework improves in-domain performance and out-of-domain benchmarks.

NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries (2024.findings-acl)

Copied to clipboard

Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Yuxiao Dong, Jie Tang

Green & Sustainable NLP

Challenge:	Large language models (LLMs) generate code for productive activities, but current benchmarks for code synthesis are oriented towards introductory tasks on algorithm and data science.
Approach:	They propose a code benchmark to mirror the complexity and variety of scenarios in real-world coding tasks.
Outcome:	The proposed benchmark improves on 39 large language models with close HumanEval scores and achieves an efficiency increase of more than 4 times.

ReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generation (2025.acl-long)

Copied to clipboard

Houxing Ren, Mingjie Zhan, Zhongyuan Wu, Aojun Zhou, Junting Pan, Hongsheng Li

LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models (2026.acl-long)

Challenge:	Existing methods to enhance code generation performance include integrating compiler feedback.
Approach:	They propose a method that integrates compiler feedback to improve one-off code generation performance.
Outcome:	The proposed method improves one-off code generation performance on three benchmarks and can be applied to other domains that focus on final results and require long reasoning paths.

Copied to clipboard

Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Zhou, Yankai Lin, Ji-Rong Wen, Chongxuan Li

PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation (2025.coling-main)

Challenge:	Masked diffusion language models have achieved significant progress in language modeling . however, the systematic analysis and empirical validation of their alignment on general tasks remains underexplored.
Approach:	They propose a framework that analyzes the bias and variance of preference optimization loss and gradient based on Direct Preference Optimization.
Outcome:	The proposed model outperforms its SFT-only predecessor on general benchmarks . it consistently outperformed other strong language models and ARMs on general tasks .

Copied to clipboard

Jaeseok Yoo, Hojae Han, Youngwon Lee, Jaejin Kim, Seung-won Hwang

ACECODER: Acing Coder RL via Automated Test-Case Synthesis (2025.acl-long)

Challenge:	Using large language models to generate code has shown significant promise, but selecting effective examples to improve generation quality remains a challenging task.
Approach:	They propose a framework that utilizes algorithmic plans to identify and retrieve effective examples.
Outcome:	The proposed framework outperforms the state-of-the-art RAG methods in code generation even when the source and target languages match or differ.

Copied to clipboard

Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, Wenhu Chen

Planning-Driven Programming: A Large Language Model Programming Workflow (2025.acl-long)

Challenge:	Recent coder models have been driven by supervised fine-tuning, but the potential of reinforcement learning remains unexplored due to the lack of reliable reward data/model in the code domain.
Approach:	They propose a pipeline that generates extensive test-case pairs from existing code data and constructs preference pairs based on pass rates over sampled programs.
Outcome:	The proposed pipeline generates extensive (question, test-cases) pairs from existing code data and trains them with Bradley-Terry loss.

Copied to clipboard

Chao Lei, Yanchuan Chang, Nir Lipovetzky, Krista A. Ehinger

Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledge (2025.findings-emnlp)

Challenge:	Recent research suggests continuous program refinements through visible tests to improve code generation accuracy in large language models (LLMs).
Approach:	They propose an LLM programming workflow to improve both initial code generation and subsequent refinements within a structured two-phase workflow.
Outcome:	The proposed workflow improves both initial code generation and subsequent refinements within a structured two-phase workflow.

Copied to clipboard

Yan-Lun Chen, Yi-Ru Wei, Chia-Yi Hsu, Chia-Mu Yu, Chun-Ying Huang, Ying-Dar Lin, Yu-Sung Wu, Wei-Bin Lee

HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation Task (2025.findings-acl)

Challenge:	Large language models (LLMs) demonstrate strong task-specific capabilities through fine-tuning, but merging multiple fine- tuned models often leads to degraded performance due to overlapping instruction-following components.
Approach:	They propose a layer-wise approach that assigns layer-specific weights to task vectors based on their alignment with instruction-following or task-specific components.
Outcome:	The proposed approach outperforms existing methods in learning and forgetting tasks while preserving overall model utility.

Copied to clipboard

Zhaojian Yu, Yilun Zhao, Arman Cohan, Xiao-Ping Zhang

CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024.acl-long)

Challenge:	Existing benchmarks for code generation tasks are inadequate, but performance declines on self-invoking tasks.
Approach:	They propose a general recipe for generating more challenging versions of existing benchmarks . they propose to use instruction-tuned models to evaluate LLMs on self-invoking code generation tasks .
Outcome:	The proposed model improves on humanEval and MBPP but on self-invoking code generation tasks.

Copied to clipboard

Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, Zhi Jin

Efficient Beam Search for Large Language Models Using Trie-Based Decoding (2025.emnlp-main)

Challenge:	Large Language Models excel in simple tasks such as generating standalone code units, but real-world software development often involves complex code repositories with complex dependencies and extensive documentation.
Approach:	They propose a novel LLM-based agent framework that employs external tools for effective repo-level code generation.
Outcome:	The proposed framework outperforms commercial products like Github Copilot in the humanEval benchmark and shows that it is adaptable and efficient across multiple code generation tasks.

Copied to clipboard

Brian J Chan, Mao-xun Huang, Jui-Hung Cheng, Chao-Ting Chen, Hen-Hsen Huang

MasRouter: Learning to Route LLMs for Multi-Agent Systems (2025.acl-long)

Challenge:	Large language models (LLMs) face memorybound performance bottlenecks due to their high memory requirements.
Approach:	They propose a trie-based parallel decoding method that shares a single KV cache across beams with common prefixes to dramatically reduce memory usage and enables efficient decoding.
Outcome:	The proposed method significantly reduces memory usage and enables efficient decoding without compromising generation quality.

Copied to clipboard

Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, Yiyan Qi

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024.findings-acl)

Challenge:	Multi-agent systems (MAS) powered by Large Language Models (LLMs) have been demonstrated to push the boundaries of LLM capabilities, yet they often face significant costs and challenges in dynamic LLM selection.
Approach:	They propose a multi-agent system routing solution that integrates all components of MAS into a unified routing framework.
Outcome:	The proposed solution is high-performing, cost-effective, and efficient . it reduces overhead by up to 52.07 compared to current methods on HumanEval .

Copied to clipboard

Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, Xiang Yue

ReCode: Robustness Evaluation of Code Generation Models (2023.acl-long)

Challenge:	OpenCodeInterpreter-33B provides a high level of performance for code generation, executing, and iterative refinement.
Approach:	They propose a family of open-source code systems for generating, executing, and iteratively refining code.
Outcome:	The OpenCodeInterpreter-33B performs well on humanEval, MBPP, and EvalPlus benchmarks.

Copied to clipboard

Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, Bing Xiang

How Do Your Code LLMs perform? Empowering Code Instruction Tuning with Really Good Data (2024.emnlp-main)

Challenge:	Existing work on robustness in text or code tasks has focused on classification, while robustness for code generation tasks is an uncharted area.
Approach:	They propose a robustness evaluation benchmark for code generation models that customizes over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format.
Outcome:	The proposed model performs better on human annotators and on SOTA models with human annnotators.

Copied to clipboard

Yejie Wang, Keqing He, Dayuan Fu, Zhuoma GongQue, Heyang Xu, Yanxu Chen, Zhexu Wang, Yujia Fu, Guanting Dong, Muxi Diao, Jingang Wang, Mengdi Zhang, Xunliang Cai, Weiran Xu

Grammar-Based Code Representation: Is It a Worthy Pursuit for LLMs? (2025.findings-acl)

Challenge:	Recent research has shown that code pre-trained models improve coding capabilities.
Approach:	They propose a code data pruning strategy to identify which datasets are high-quality code instruction data.
Outcome:	The proposed model achieves state-of-the-art performance using fewer training data.

Copied to clipboard

Qingyuan Liang, Zhao Zhang, Zeyu Sun, Zheng Lin, Qi Luo, Xiao Yueyi, Yizhou Chen, Yuqun Zhang, Haotian Zhang, Lu Zhang, Chenbin Chenbin, Yingfei Xiong

Enhanced Data Synthesis for LLM through Reasoning Structures Generated by Hierarchical GFlowNet (2025.findings-acl)

Challenge:	Existing research demonstrates the effectiveness of grammar-based code representations in small-scale models, showing their ability to reduce syntax errors and enhance performance.
Approach:	They develop a series of billion-scale grammar-based code representations that incorporate grammar rules into the code generation process.
Outcome:	Experiments on HumanEval and MBPP show that grammar-based representations reduce syntax errors and improve performance even in billion-scale models.

Copied to clipboard

Tianpeng Bu, Minying Zhang, Hongtao Duan, Shurui Li, Lulu Hu, Yu Li

Reasoning

Challenge:	Existing methods to optimize instruction-response pairs lack a systematic design for the underlying reasoning structure.
Approach:	They propose a Reasoning Structure driven data Synthesis method that leverages a coarse-to-fine directed acyclic graph to construct reasoning structures efficiently.
Outcome:	The proposed method outperforms existing methods in 48.50%, 84.00%, 79.90% of the synthetic datasets trained on the proposed model.

StoryCoder: Narrative Reformulation for Structured Reasoning in LLM Code Generation (2026.acl-long)

Copied to clipboard

Geonhui Jang, Dongyoon Han, YoungJoon Yoo

Code-SPA: Style Preference Alignment to Large Language Models for Effective and Robust Code Debugging (2025.findings-acl)

Challenge:	Existing approaches augment reasoning steps or inject specific structure into how models think, but leave scattered problem conditions unchanged.
Approach:	They propose a narrative reformulation framework that transforms code generation questions into coherent natural language narratives.
Outcome:	The proposed framework improves the performance of 11 code generation models on HumanEval, LiveCodeBench, and CodeForces.

Copied to clipboard

Tengfei Wen, Xuanang Chen, Ben He, Le Sun

RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding (2026.findings-acl)

Challenge:	Large language models (LLMs) have demonstrated impressive capabilities in coding tasks like code generation and debugging.
Approach:	They propose a method which aligns noisy code with the well-structured style familiar to LLMs, mitigating the impact of stylistic inconsistencies.
Outcome:	The proposed method improves debugging performance on poorly styled code across the HumanEval, MBPP and EvalPlus datasets.

Copied to clipboard

Zihong Zhang, Zuchao Li, Lefei Zhang, Ping Wang, Hai Zhao

Information Retrieval

Challenge:	Existing methods for decoding large language models generate one token per step, causing high inference latency.
Approach:	They propose a method that integrates retrieved exact patterns with logit-driven future cues.
Outcome:	Experiments on Spec-Bench, HumanEval, and MGSM-ZH show that RACER outperforms training-free methods and accelerates inference.

RACC: Regret-Aware Confidence Calibration for Consistent Masked Discrete Diffusion Decoding (2026.findings-acl)

Copied to clipboard

Qinglin Zeng, Jusheng Zhang, Jing Yang, Ningyuan Liu, Keze Wang

CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution (2025.acl-long)

Challenge:	Masked Discrete Diffusion Models (MDMs) enable parallel generation via iterative refinement, but their current decoding paradigms are static and myopic.
Approach:	They propose a Regret-Aware Confidence Calibration framework that aligns decoding decisions with the model’s latent self-correction capabilities.
Outcome:	The proposed framework aligns decoding decisions with model’s latent self-correction capabilities.

Copied to clipboard

Ruiyang Xu, Jialun Cao, Yaojie Lu, Ming Wen, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, Le Sun

Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement (2025.findings-acl)

Challenge:	Existing code benchmarks focus on code generation, while those for code reasoning are insufficient.
Approach:	They propose a multi-lingual code reasoning benchmark that contains 19 programming languages and at least 600 subjects for each language.
Outcome:	The proposed model trains on Python and achieves 34.4% Pass@1 in other languages, revealing the cross-language generalization of LLMs.

Copied to clipboard

Xiaoqing Zhang, Yuhan Liu, Flood Sung, Xiuying Chen, Shuo Shang, Rui Yan

Can Language Models Replace Programmers for Coding? REPOCOD Says ‘Not Yet’ (2025.acl-long)

Challenge:	Recent research indicates that large language models (LLMs) have demonstrated remark-able capabilities in various programming-related domains, such as code generation and code refinement.
Approach:	They propose a framework that combines exploration with refinement to reduce test-time computation overhead.
Outcome:	The proposed framework outperforms SOTA and AgentCoder on humanEval and MBPP benchmarks while reducing test-time computation overhead and scalability.

Copied to clipboard

Shanchao Liang, Nan Jiang, Yiran Hu, Lin Tan

SDC-LoRA: Singular-Subspace Drift Controlled LoRA to Mitigate Knowledge Forgetting (2026.findings-acl)

Challenge:	Existing benchmarks for code generation use short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks.
Approach:	They propose a Python code-generation benchmark that contains 980 whole-function generation tasks with realistic dependencies from 11 popular projects.
Outcome:	The proposed benchmarks are short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks.

Copied to clipboard

Geyuan Zhang, Xiaofei Zhou, Shihao Liu, Jingyuan Tian, Jizheng Ma

FractalLLM: Lossless Self-Speculative Decoding with Layer Embedded Self-Compression (2025.findings-emnlp)

Challenge:	Existing approaches to adapt LLMs to new tasks focus on limiting knowledge forgetting . et al., 2023b) suggest a solution to this problem by limiting update energy in the principal singular subspace of W0 .
Approach:	They propose a low-rank Adaptation (LoRA) that steers early updates away from principal directions and mitigates forgetting by constraining update energy in the principal singular subspace of W0.
Outcome:	The proposed model mitigates forgetting on MMLU, TruthfulQA, and HellaSwag while keeping minor-subspace updates unchanged.

Copied to clipboard

Juhyeong Kim, Sangyeon Yu, Gyunyeop Kim, Sangwoo Kang

CopySpec: Accelerating LLMs with Speculative Copy-and-Paste (2025.emnlp-main)

Challenge:	Autoregressive decoding requires a full forward pass for each generated token, increasing inference latency.
Approach:	They propose a lossless self-speculative decoding method that embeds a compressed model within selected decoder layers of the original model.
Outcome:	The proposed method achieves substantial speed-ups (up to 2.47) over standard autoregressive decoding.

Copied to clipboard

Razvan-Gabriel Dumitru, Minglai Yang, Vikas Yadav, Mihai Surdeanu

Machine Translation

Challenge:	Large Language Models (LLMs) suffer from slower inference as context size grows, but CopySpec leverages larger contexts to accelerate inference.
Approach:	They propose a technique that speculates that the same tokens will follow repeated sequences in the model’s chat history or context and enables seamless copying without compromising output quality.
Outcome:	The proposed technique can generate responses that closely resemble previous outputs or responses that can be verbatim extracted from context without compromising output quality and without requiring additional GPU memory.

SACL: Understanding and Combating Textual Bias in Code Retrieval with Semantic-Augmented Reranking and Localization (2025.findings-emnlp)

Copied to clipboard

Dhruv Gupta, Gayathri Ganesh Lakshmy, Yiqing Xie

Information Retrieval

Challenge:	a recent study shows that code retrievers exhibit a strong bias towards well-documented code .
Approach:	They propose a framework that augments textual information with semantic information to mask specific features while preserving code functionality.
Outcome:	The proposed framework enhances textual information and reduces bias by augmenting code or structural knowledge with semantic information.

CPC-GRPO: Answer-Free Reinforcement Learning with Cross-Prompt Consensus Rewards (2026.findings-acl)

Copied to clipboard

Gyunyeop Kim, Sangwoo Kang

Group-Aware Reinforcement Learning for Output Diversity in Large Language Models (2025.emnlp-main)

Challenge:	Reinforcement learning with verifiable rewards is a popular post-training tool for large language models, but relies on a ground-truth answer or external verifier, which limits applicability and increases cost.
Approach:	They propose an answer-free training objective that derives rewards solely from the model’s own probabilities by exploiting prompt paraphrases as multiple semantic views of the same intent.
Outcome:	The proposed objective derives rewards solely from the model’s own probabilities by exploiting prompt paraphrases as multiple semantic views of the same intent.

Copied to clipboard

Oron Anschel, Alon Shoshan, Adam Botach, Shunit Haviv Hakimi, Asaf Gendler, Emanuel Ben Baruch, Nadav Bhonker, Igor Kviatkovsky, Manoj Aggarwal, Gerard Medioni

IntentCoding: Amplifying User Intent in Code Generation (2026.findings-acl)

Challenge:	Large Language Models suffer from mode collapse, repeatedly generating the same few completions even when many valid answers exist.
Approach:	They propose a group-aware policy optimization extension of GRPO that computes rewards over the group as a whole.
Outcome:	The proposed model improves on standard LLM benchmarks without compromising accuracy.

Copied to clipboard

Zheng Fang, Yihong Dong, Lili Mou, Dongming Jin, Zhi Jin, Ge Li

TA-GRPO-d: Trajectory-Aware GRPO for Optimizing Denoising Trajectories in Diffusion LLMs (2026.acl-long)

Challenge:	IntentCoding captures the influence of user intent by masking out the intent, and integrates seamlessly with existing decoding procedures.
Approach:	They propose a decoding strategy that captures the influence of user intent by masking out the intent and applies a multi-strength ensemble mechanism to amplify the effect of user intention during generation.
Outcome:	The proposed model significantly improves both constraint satisfaction and functional correctness compared to greedy decoding approaches.

Copied to clipboard

Gyunyeop Kim, Sangwoo Kang

ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning (2026.findings-acl)

Challenge:	Existing dLLMs rely on fixed denoising schedules and cannot learn efficient unmasking orders.
Approach:	They propose a framework that transforms dLLM decoding into a trajectory-aware policy . it uses a confidence-gated denoising strategy that decides which tokens to unmask .
Outcome:	The proposed model can learn which tokens to unmask and how many to unmak per step . it can learn the output quality and efficiency of the decoding path itself .

Copied to clipboard

Juyong Jiang, Jiasi Shen, Sunghun Kim, Kang Min Yoo, Jeonghoon Kim, Sungju Kim

OmniCode: A Benchmark for Evaluating Software Development Agents (2026.findings-acl)

Challenge:	Existing iterative refinement strategies that generate solutions in a single forward pass often hit a performance ceiling on complex algorithmic tasks.
Approach:	They propose a reinforcement learning framework that internalizes the structured reasoning trajectory directly into the model’s weights.
Outcome:	The proposed framework achieves 94.51% (87.20%) on HumanEval, 81.80% (78.57%) on MBPP, 35.00% on BigCodeBench, 52.21% on LiveCodeBech, and 37.34% on CodeForces in a single-attempt setting.

Copied to clipboard

Atharv Sonwane, Eng-Shen Tu, Wei-Chung Lu, Claas Beger, Carter Larsen, Debjit Dhar, Simon Alford, Rachel Chen, Ronit Pattanayak, Tuan Anh Dang, Guohao Chen, Gloria Geng, Kevin Ellis, Saikat Dutta