COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis (2025.findings-naacl)
Copied to clipboard
Weiqing Yang, Hanbin Wang, Zhenghao Liu, Xinze Li, Yukun Yan, Shuo Wang, Yu Gu, Minghe Yu, Zhiyuan Liu, Ge Yu
| Challenge: | Existing code debugging benchmarks focus on the Code Repair stage of the code generation process. |
| Approach: | They propose a framework to evaluate the debugging abilities of large language models by emulating the human debug process. |
| Outcome: | The proposed framework outperforms human-curated and GPT-4-generated training data, enabling 7B-scale LLMs to achieve comparable debugging performance to GPT-3.5. |
Similar Papers
DebugBench: Evaluating Debugging Capability of Large Language Models (2024.findings-acl)
Copied to clipboard
Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, Maosong Sun
| Challenge: | Large language models (LLMs) have demonstrated exceptional coding capabilities, but their debugging capabilities remain relatively unexplored. |
| Approach: | They propose a debugging benchmark consisting of 4,253 LLMs with four major bug categories and 18 minor types in C++, Java, and Python. |
| Outcome: | The proposed benchmark covers four major bug categories and 18 minor types in C++, Java, and Python. |
Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are leading progress in code generation, but they are underutilized in the literature. |
| Approach: | They propose a debugging framework that allows LLMs to refine their generated programs with the runtime execution information. |
| Outcome: | The proposed framework improves the baseline performance by 9.8% across the HumanEval, MBPP, and TransCoder benchmarks. |
MapCoder: Multi-Agent Code Generation for Competitive Problem Solving (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive proficiency in natural language processing, but performance in code generation tasks remains limited. |
| Approach: | They propose a framework that emulates the full cycle of program synthesis as observed in humans. |
| Outcome: | The proposed framework replicates the full cycle of program synthesis as observed in human developers. |
CodeSim: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant strides in code generation and problem solving. |
| Approach: | They propose a multi-agent code generation framework that integrates human-like perception to address the stages of program synthesis. |
| Outcome: | The proposed framework achieves state-of-the-art (pass@1) results and shows potential for even greater enhancement when cascaded with external debuggers. |
MapCoder-Lite: Distilling Multi-Agent Coding into a Single Small LLM (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing large-scale (> 30 B) models are costly and collapse when downsized to small open-source models. |
| Approach: | They propose a framework for distilling large, multi-agent coding systems into a single 7B model. |
| Outcome: | The proposed framework doubles xCodeEval accuracy and reduces GPU memory and token generation time by 4 compared to a 32B model. |
UniDebugger: Hierarchical Multi-Agent Framework for Unified Software Debugging (2025.emnlp-main)
Copied to clipboard
Cheryl Lee, Chunqiu Steven Xia, Longji Yang, Jen-tse Huang, Zhouruixing Zhu, Lingming Zhang, Michael R. Lyu
| Challenge: | Existing LLMs focus on isolated steps and struggle with complex bugs. |
| Approach: | They propose a framework for unified debugging through multi-agent synergy . it mimics the entire cognitive processes of developers with each agent specialized as a particular component of this process . |
| Outcome: | The proposed framework outperforms state-of-the-art methods on repo-level benchmarks. |
MdEval: Massively Multilingual Code Debugging (2026.findings-acl)
Copied to clipboard
Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Jin Ke, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, Ge Zhang, Zhoujun Li
| Challenge: | Existing benchmarks primarily focus on Python and are limited in terms of language diversity. |
| Approach: | They propose a multilingual debugging benchmark that includes 3.9K test samples of 20 programming languages and introduces the debug instruction corpora MdEval-Instruct by injecting bugs into the correct multilingual queries and solutions. |
| Outcome: | The proposed benchmark includes 3.9K test samples of 20 programming languages and covers the automated program repair task, bug localization task, and bug identification task. |
AgentTuning: Enabling Generalized Agent Abilities for LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Open large language models (LLMs) with great performance in various tasks are far inferior to commercial models such as ChatGPT and GPT-4 when acting as agents to tackle complex tasks in the real world. |
| Approach: | They propose a method to enhance the agent capabilities of LLMs while maintaining their general abilities. |
| Outcome: | The AgentLM-70B is comparable to GPT-3.5-turbo on unseen agent tasks, demonstrating generalized agent capabilities. |
CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | coding tasks require generated code to be fully executable and functionally correct . current agentic approaches struggle with multi-stage planning, generating, and debugging . |
| Approach: | They propose a framework for LLM agents to efficiently explore the search space in different stages of the code generation process. |
| Outcome: | The proposed framework achieves top results on 7 code generation benchmarks and a 31.9% solving rate on the SWEBench benchmark. |
Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement (2025.acl-long)
Copied to clipboard
Maosongcao Maosongcao, Taolin Zhang, Mo Li, Chuyu Zhang, Yunxin Liu, Conghui He, Haodong Duan, Songyang Zhang, Kai Chen
| Challenge: | Existing high-quality human-annotated SFT data is a bottleneck for Large Language Models (LLMs). |
| Approach: | They propose a two-stage synthetic data generation framework that incorporates World Knowledge Trees and Self-Reflection Refinement to produce high-quality SFT data at scale. |
| Outcome: | The proposed model fine-tuned on 20K condor-generated samples achieves superior performance compared to instruct model trained with RLHF. |