DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale

Challenge:	Existing studies highlight that dependency-related issues cause over 40% of observed runtime errors on the generated repository.
Approach:	They propose a large-scale benchmark and evaluation framework specifically designed to assess LLMs’ capability on dependency inference.
Outcome:	The proposed model achieves only a 48% execution pass rate on Python, indicating room for improvement.

Similar Papers

Overview

Challenge

Approach

Outcome

DependEval: Benchmarking LLMs for Repository Dependency Understanding (2025.findings-acl)

Copied to clipboard

Junjia Du, Yadi Liu, Hongcheng Guo, Jiawei Wang, Haojian Huang, Yunyi Ni, Zhoujun Li

Challenge:	a benchmark is designed to evaluate the repository-level dependency understanding of large language models (LLMs) based on 2683 repositories from real-world websites.
Approach:	They propose a benchmark to evaluate repository dependency understanding for large language models . DEPENDEVAL evaluates models on three core tasks across 8 programming languages .
Outcome:	The benchmark evaluates models on three core tasks across 8 programming languages from real-world repositories.

Turning the Tide: Repository-based Code Reflection (2025.findings-emnlp)

Copied to clipboard

Wei Zhang, Jian Yang, Jiaxi Yang, Ya Wang, Zhoujun Li, Zeyu Cui, Binyuan Hui, Junyang Lin

Programming Languages in NLP

Challenge:	Code large language models (LLMs) enhance programming by understanding and generating code across languages.
Approach:	a new benchmark evaluates code understanding and generation in repositories using code large language models.
Outcome:	The proposed model improves code understanding and generation in repositories by evaluating 1,888 test cases across 6 programming languages.

How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench (2023.findings-emnlp)

Copied to clipboard

Qinyuan Ye, Harvey Fu, Xiang Ren, Robin Jia

Language Models

Challenge:	a recent study shows that large language models can be used to predict performance on new configurations.
Approach:	They investigate the predictability of large language model capabilities by using BIG-bench . they find a subset of BIG-Bench tasks as informative as BIG-bnch Hard .
Outcome:	The proposed model achieves an R2 score greater than 95% on BIG-bench . the model is 3 smaller than BIG-Bench Hard, and the model performs better on the full set.

Benchmarking for Domain-Specific LLMs: A Case Study on Academia and Beyond (2025.findings-emnlp)

Copied to clipboard

Rubing Chen, Jiaxin Wu, Jian Wang, Xulu Zhang, Wenqi Fan, Chenghua Lin, Xiaoyong Wei, Li Qing

Question Answering

Challenge:	Comp-Comp is an iterative benchmarking framework grounded in the principles of comprehensiveness and compactness.
Approach:	They propose a benchmark framework that incorporates the principle of comprehensiveness and compactness.
Outcome:	The proposed framework is domain-agnostic and adaptable to a wide range of specialized fields.

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories (2024.findings-acl)

Copied to clipboard

Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, Mengfei Yang

Language Models

Challenge:	Existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of Large Language Models (LLMs).
Approach:	They propose a repository-level benchmark named DevEval to evaluate LLMs' coding abilities in real-world code repositories.
Outcome:	The proposed benchmarks show that the LLMs perform better in real-world code repositories than existing benchmarks.

FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation (2025.acl-long)

Copied to clipboard

Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, Scarlett Li

Programming Languages in NLP

Challenge:	Existing benchmarks focus on standalone programming problems, such as HumanEval, MBPP, and LiveCodeBench.
Approach:	They propose to use large language models to evaluate their ability to perform incremental development within code repositories by collecting pull requests from 83 GitHub repositorias and using rule-based and intent-based filtering to construct task instances focused on new feature development.
Outcome:	The proposed benchmarks show that large language models perform significantly worse in the FEA-Bench, highlighting considerable challenges in repository-level incremental code development.

CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Tasks (2026.acl-long)

Copied to clipboard

Lingyue Fu, Hao Guan, Bolun Zhang, Haowei Yuan, Yaoming Zhu, Lin Qiu, ZongYu Wang, Xuezhi Cao, Xunliang Cai, Weiwen Liu, Weinan Zhang, Yong Yu

Challenge:	Existing large language models for software engineering rely on coarse-grained pass rates obscuring specific cognitive bottlenecks.
Approach:	They propose a repository-level benchmark that dissects coding capabilities through atomized tasks.
Outcome:	The proposed framework achieves a 78.55% validity yield, surpassing the 31.7% retention rate of SWE-bench-Verified.

CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)

Copied to clipboard

Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, Robby T. Tan

Robustness in NLP

Challenge:	Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood.
Approach:	They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models .
Outcome:	The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks.

CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding? (2025.coling-main)

Copied to clipboard

Yuwei Zhao, Ziyang Luo, Yuchen Tian, Hongzhan Lin, Weixiang Yan, Annan Li, Jing Ma

Semantic Text Processing

Challenge:	Recent advances in large language models (LLMs) have showcased impressive code generation capabilities, primarily evaluated through language-to-code benchmarks.
Approach:	They propose a benchmark to assess LLMs’ code understanding abilities from the perspective of code judging rather than code generation.
Outcome:	The proposed benchmark evaluates 12 well-known large language models to determine the correctness of provided code solutions.

Evaluating Large Language Models with Enterprise Benchmarks (2025.naacl-industry)

Copied to clipboard

Bing Zhang, Mikio Takeuchi, Ryo Kawahara, Shubhi Asthana, Maruf Hossain, Guang-Jie Ren, Kate Soule, Yifan Mai, Yada Zhu

Language Models

Challenge:	Existing benchmarks lack domain-specific datasets for evaluating large language models . existing benchmarks often lack domain specific datasets, which can be difficult to convert to standardized metrics or regulatory issues.
Approach:	They propose to use 25 publicly available domain-specific English benchmarks from diverse domains . they propose to combine a wide range of natural language processing tasks for holistic evaluation .
Outcome:	The proposed framework includes 25 publicly available domain-specific English benchmarks from diverse enterprise domains like financial services, legal, climate, cyber security, and 2 public Japanese finance benchmarks.