CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art large language models (LLMs) have demonstrated impressive code generation capabilities but struggle with real-world software engineering tasks such as revising source code to address code reviews. |
| Approach: | They propose a benchmark to evaluate large language models' ability to bridge both technical and conversational contexts by decomposing the generation task of code refinement into three essential reasoning steps. |
| Outcome: | The proposed benchmark exposes specific model weaknesses in code review comprehension disentangled from their generative automated code refinement results. |
Similar Papers
CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding? (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have showcased impressive code generation capabilities, primarily evaluated through language-to-code benchmarks. |
| Approach: | They propose a benchmark to assess LLMs’ code understanding abilities from the perspective of code judging rather than code generation. |
| Outcome: | The proposed benchmark evaluates 12 well-known large language models to determine the correctness of provided code solutions. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models (2025.naacl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | a recent study evaluated language models using abstract evaluation criteria that lack the flexibility and granularity of human assessment. |
| Approach: | They propose a benchmark to evaluate nine distinct language models' capabilities . they use instance-specific evaluation criteria to mirror human evaluation . |
| Outcome: | The proposed benchmark evaluates nine distinct capabilities of language models across 77 tasks. |
Towards Intrinsic Interpretability of Large Language Models: A Survey of Design Principles and Architectures (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies on explainable AI focus on post-hoc explanation methods that interpret trained models through external approximations. |
| Approach: | They propose to categorize existing approaches into five design paradigms: functional transparency, concept alignment, representational decomposability, explicit modularization, and latent sparsity induction. |
| Outcome: | The proposed approaches are categorized into five design paradigms: functional transparency, concept alignment, representational decomposability, explicit modularization, and latent sparsity induction. |
Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape . established automatic evaluation metrics are poor surrogates, correlating weakly with human judgement. |
| Approach: | They propose to use both automatic and human evaluation to evaluate generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction. |
| Outcome: | The proposed model outperforms many popular models according to human reviewers on the majority of metrics, while scoring much worse when using classic automatic evaluation metrics. |
Turning the Tide: Repository-based Code Reflection (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Code large language models (LLMs) enhance programming by understanding and generating code across languages. |
| Approach: | a new benchmark evaluates code understanding and generation in repositories using code large language models. |
| Outcome: | The proposed model improves code understanding and generation in repositories by evaluating 1,888 test cases across 6 programming languages. |
ICE-Score: Instructing Large Language Models to Evaluate Code (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in the field of natural language generation have facilitated the use of large language models to assess the quality of generated text. |
| Approach: | They propose a new evaluation metric by instructing large language models for code assessments using a set of programming languages. |
| Outcome: | The proposed evaluation metric surpasses state-of-the-art metrics for code generation, delivering high levels of accuracy and consistency across programming languages and tasks. |
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models are contaminated with data from pretraining and finetuning tasks. |
| Approach: | They perform extensive analysis on the factors that affect model memorization and generalization, such as model size, problem difficulty, and question length. |
| Outcome: | The results show that models perform better on the subset of the benchmarks where similar solutions are seen during training. |
CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation (2025.acl-industry)
Copied to clipboard
| Challenge: | CodeIF assesses the ability of large language models to adhere to task-oriented instructions in code generation tasks. |
| Approach: | They introduce a benchmark designed to assess LLMs' ability to adhere to task-oriented instructions within diverse code generation scenarios. |
| Outcome: | The proposed benchmark assesses LLMs' ability to adhere to task-oriented instructions in code generation tasks across a wide range of complexity levels and programming domains. |
CodeJudge: Evaluating Code Generation with Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising performance in code generation, but how to reliably evaluate code generated by LLMs remains a challenging problem. |
| Approach: | They propose a framework that leverages Large Language Models to evaluate the semantic correctness of generated code without the need for test cases. |
| Outcome: | The proposed framework outperforms existing methods on four code generation datasets and five programming languages. |