HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation Task (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for code generation tasks are inadequate, but performance declines on self-invoking tasks. |
| Approach: | They propose a general recipe for generating more challenging versions of existing benchmarks . they propose to use instruction-tuned models to evaluate LLMs on self-invoking code generation tasks . |
| Outcome: | The proposed model improves on humanEval and MBPP but on self-invoking code generation tasks. |
Similar Papers
HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). |
| Approach: | They propose a benchmark to evaluate multilingual LLMs using multiple natural languages. |
| Outcome: | The proposed benchmarks focus on translating English prompts to multilingual code or have been constrained to very limited natural languages (NLs). |
The Program Testing Ability of Large Language Models for Code (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent development of large language models (LLMs) for code shows promise in achieving code intelligence. |
| Approach: | They explore the ability of large language models to generate automated test cases . they show +11.77% and +4.22% higher code pass rates on HumanEval+ . |
| Outcome: | The proposed models show higher pass rates on humanEval+ compared with the current state-of-the-art models. |
mHumanEval - A Multilingual Benchmark to Evaluate Large Language Models for Code Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Current evaluations focus on English-to-Python conversion tasks with limited test cases . code generation from low-resource language prompts remains largely unexplored . |
| Approach: | They propose a benchmark that supports prompts in over 200 natural languages . they provide expert human translations for 15 diverse natural languages (NLs) |
| Outcome: | The HumanEval Benchmark is the most widely used code generation benchmark . it provides expert human translations for 15 diverse natural languages . |
PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | *HumanEval* and *MBPP* are two popular benchmarks for Python code generation. |
| Approach: | They propose a large-scale human evaluation of two popular Python benchmarks . they propose 185 hand-crafted prompts in a balanced representation of 38 programming concepts across diverse difficulty levels. |
| Outcome: | The proposed benchmarks show a critical bias towards a limited set of programming concepts, neglecting most of the other concepts entirely. |
CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding? (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have showcased impressive code generation capabilities, primarily evaluated through language-to-code benchmarks. |
| Approach: | They propose a benchmark to assess LLMs’ code understanding abilities from the perspective of code judging rather than code generation. |
| Outcome: | The proposed benchmark evaluates 12 well-known large language models to determine the correctness of provided code solutions. |
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
SEK: Self-Explained Keywords Empower Large Language Models for Code Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved impressive performance in code generation. |
| Approach: | They propose a technique that extracts and explicates the key terms in the problem description with the LLM itself. |
| Outcome: | The proposed technique improves the Pass@1 of DeepSeek-Coder-V2-Instruct from 85.4% to 93.3% on the humaneval benchmark. |
Grammar-Based Code Representation: Is It a Worthy Pursuit for LLMs? (2025.findings-acl)
Copied to clipboard
Qingyuan Liang, Zhao Zhang, Zeyu Sun, Zheng Lin, Qi Luo, Xiao Yueyi, Yizhou Chen, Yuqun Zhang, Haotian Zhang, Lu Zhang, Chenbin Chenbin, Yingfei Xiong
| Challenge: | Existing research demonstrates the effectiveness of grammar-based code representations in small-scale models, showing their ability to reduce syntax errors and enhance performance. |
| Approach: | They develop a series of billion-scale grammar-based code representations that incorporate grammar rules into the code generation process. |
| Outcome: | Experiments on HumanEval and MBPP show that grammar-based representations reduce syntax errors and improve performance even in billion-scale models. |
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood. |
| Approach: | They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models . |
| Outcome: | The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks. |
CodeJudge: Evaluating Code Generation with Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising performance in code generation, but how to reliably evaluate code generated by LLMs remains a challenging problem. |
| Approach: | They propose a framework that leverages Large Language Models to evaluate the semantic correctness of generated code without the need for test cases. |
| Outcome: | The proposed framework outperforms existing methods on four code generation datasets and five programming languages. |