ICE-Score: Instructing Large Language Models to Evaluate Code

Challenge:	Recent advances in the field of natural language generation have facilitated the use of large language models to assess the quality of generated text.
Approach:	They propose a new evaluation metric by instructing large language models for code assessments using a set of programming languages.
Outcome:	The proposed evaluation metric surpasses state-of-the-art metrics for code generation, delivering high levels of accuracy and consistency across programming languages and tasks.

Similar Papers

Overview

Challenge

Approach

Outcome

Co-Eval: Augmenting LLM-based Evaluation with Machine Metrics (2025.emnlp-main)

Copied to clipboard

Ling-I Wu, Weijie Wu, Minyu Chen, Jianxin Xue, Guoqiang Li

Challenge:	Existing LLMs suffer from biases and misalignment due to limited functional understanding and knowledge gaps.
Approach:	They introduce a framework that leverages a criteria planner model and optimized machine metrics to enhance the scalability and fairness of LLM-based evaluation.
Outcome:	The proposed framework reduces biases and improves alignment with human preferences, with gains of up to 0.324 in Spearman correlation.

Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation? (2024.findings-eacl)

Copied to clipboard

Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, Sunayana Sitaram

Semantic Text Processing

Challenge:	Large Language Models (LLMs) excel in various tasks, but their evaluation, especially in languages beyond the top 20, remains inadequate due to existing benchmarks and metrics limitations.
Approach:	They propose to use Large Language Models as evaluators to rank or score other models’ outputs by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages.
Outcome:	The proposed evaluation methods can be used to improve multilingual evaluation by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages.

Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks (2025.coling-main)

Copied to clipboard

Qintong Li, Leyang Cui, Lingpeng Kong, Wei Bi

Language Models

Challenge:	Existing work uses large language models (LLMs) to evaluate natural language process tasks, but there are shortcomings in current LLMs.
Approach:	They examine the alignment between LLM evaluators and human annotators by comparing conventional and alignment tasks with different evaluation criteria.
Outcome:	The proposed models excel in general criteria, such as fluency, but face challenges with complex criteria, including numerical reasoning.

CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding? (2025.coling-main)

Copied to clipboard

Yuwei Zhao, Ziyang Luo, Yuchen Tian, Hongzhan Lin, Weixiang Yan, Annan Li, Jing Ma

Semantic Text Processing

Challenge:	Recent advances in large language models (LLMs) have showcased impressive code generation capabilities, primarily evaluated through language-to-code benchmarks.
Approach:	They propose a benchmark to assess LLMs’ code understanding abilities from the perspective of code judging rather than code generation.
Outcome:	The proposed benchmark evaluates 12 well-known large language models to determine the correctness of provided code solutions.

Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)

Copied to clipboard

Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao, Shuai Ma

Language Models

Challenge:	introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance.
Approach:	They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them.
Outcome:	The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods.

XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2024.acl-long)

Copied to clipboard

Mohammad Abdullah Matin Khan, M Saiful Bari, Do Long, Weishi Wang, Md Rizwan Parvez, Shafiq Joty

Multilinguality

Challenge:	Recent advances in large language models have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments.
Approach:	They propose to use a multilingual multitask benchmark to evaluate large language models that can generate codes from natural language descriptions, repair buggy codes, and translate between languages.
Outcome:	The proposed model performs 7 tasks covering up to 11 languages with execution-level parallelism and 25 M document-level coding examples (16.5 B tokens)

Characterizing the Confidence of Large Language Model-Based Automatic Evaluation Metrics (2024.eacl-short)

Copied to clipboard

Rickard Stureborg, Dimitris Alikaniotis, Yoshi Suhara

Language Models

Challenge:	Recent studies have focused on using Large Language Models (LLMs) to evaluate NLP tasks automatically.
Approach:	They characterize LLM evaluators’ confidence in ranking candidate NLP models and develop a configurable Monte Carlo simulation method to compensate for loss of correlation.
Outcome:	The proposed method can reach 95% confidence rankings of candidate models with reasonable evaluation set sizes.

Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026.findings-eacl)

Copied to clipboard

Clemencia Siro, Pourya Aliannejadi, Mohammad Aliannejadi

Explainability & Interpretability in NLP

Challenge:	Large language models are increasingly used as evaluators for natural language generation . human rubrics are often static and misaligned with how models internally represent language quality.
Approach:	They propose to use large language models to generate interpretable and task-aware evaluation dimensions and apply them within models.
Outcome:	The proposed model improves the semantic coherence and scoring reliability of LLM-defined criteria and their alignment with human criteria.

CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models (2025.findings-acl)

Copied to clipboard

Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, Christoph Treude

Semantic Text Processing

Challenge:	State-of-the-art large language models (LLMs) have demonstrated impressive code generation capabilities but struggle with real-world software engineering tasks such as revising source code to address code reviews.
Approach:	They propose a benchmark to evaluate large language models' ability to bridge both technical and conversational contexts by decomposing the generation task of code refinement into three essential reasoning steps.
Outcome:	The proposed benchmark exposes specific model weaknesses in code review comprehension disentangled from their generative automated code refinement results.

Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks (2023.emnlp-main)

Copied to clipboard

Andrea Sottana, Bin Liang, Kai Zou, Zheng Yuan

Language Models

Challenge:	Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape . established automatic evaluation metrics are poor surrogates, correlating weakly with human judgement.
Approach:	They propose to use both automatic and human evaluation to evaluate generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction.
Outcome:	The proposed model outperforms many popular models according to human reviewers on the majority of metrics, while scoring much worse when using classic automatic evaluation metrics.