CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells (2025.naacl-long)
Copied to clipboard
| Challenge: | Current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). |
| Approach: | They propose to use a reference-free metric to measure review quality like conciseness, comprehensiveness, and relevance to compare the quality of code changes with human-written references. |
| Outcome: | The proposed metric can produce fine-grained scores that have the greatest alignment with human judgment and are more sensitive than reference-based metrics. |
Similar Papers
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing learning metrics are limited to tasks where large human ratings are available. |
| Approach: | They propose a model-based natural language generation (NLG) evaluation metric that is highly correlated with human judgements without requiring human annotation. |
| Outcome: | The proposed metric outperforms all prior unsupervised metrics on multiple NLG tasks including translation, image captioning, and WebNLG text generation. |
SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes (2023.acl-long)
Copied to clipboard
| Challenge: | Existing learned metrics perform unsatisfactory across text generation tasks or require human annotations for training on specific tasks. |
| Approach: | They propose a self-supervised approach to train a model-based metric for text generation evaluation using sentences retrieved from a corpus. |
| Outcome: | The proposed model outperforms all prior unsupervised metrics on four text generation evaluation benchmarks, with an average Kendall improvement of 0.158. |
CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systems (2025.coling-main)
Copied to clipboard
| Challenge: | Existing metrics for dialogue quality evaluation show low correlation with human judgements . current metrics do not accurately evaluate dialogue responses based on dialogue history . |
| Approach: | They propose a new metric measuring causal strength between dialogue histories and responses . they collect a dialogue dataset with human-annotated causal relations and pairwise human judgements . |
| Outcome: | The proposed metric outperforms existing state-of-the-art metrics in human judgements . it is based on a dialogue dataset with human-annotated causal relations and human judgement sets . |
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code (2023.emnlp-main)
Copied to clipboard
| Challenge: | NLCode generates long expressions and statements rather than a single next-token . evaluating and comparing different models has remained a challenge . |
| Approach: | They propose a code-generating evaluation metric built on BERTScore . they use five language-specific pretrained models to evaluate their code . |
| Outcome: | The proposed evaluation metric achieves higher correlation with human preference and functional correctness than existing metrics across four programming languages. |
ReviewEval: An Evaluation Framework for AI-Generated Reviews (2025.findings-emnlp)
Copied to clipboard
| Challenge: | escalating volume of academic research necessitates innovative approaches to peer review . authors propose reviewEval, ReviewAgent and ReviewEval to improve on existing reviews . |
| Approach: | They propose a framework for AI-generated reviews that measures alignment with human assessments . they propose 'reviewAgent' that iteratively optimizes its intermediate outputs and external improvement loops . |
| Outcome: | The proposed framework improves actionable insights and analytical depth by 6.78% and 47.62% over baselines and expert reviews. |
T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for evaluating text quality are discriminative and generative . current methods use manual annotation of human judgements to train them . |
| Approach: | They propose a framework that combines the best of both worlds by using supervised and unsupervised signals from whatever data we have available. |
| Outcome: | The proposed method outperforms existing metrics on 5 datasets, 19 languages and 280 systems. |
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (D19-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for natural language generation are inadequate . distinguishing machine-generated text is challenging even for human evaluators . |
| Approach: | They compare human-based evaluators with automated evaluation procedures . they find human evaluers do not correlate well with discriminative evalators . |
| Outcome: | The proposed evaluation methods are compared with a dozen state-of-the-art generators for online product reviews. |
The Good, the Bad and the Constructive: Automatically Measuring Peer Review’s Utility for Authors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Providing constructive feedback to authors is a core component of peer review . authors lack guidance on how to improve their review, a problem that is often overlooked . |
| Approach: | They use a RevUtil dataset to benchmark fine-tuned models for assessing review comments . they find that machine-generated reviews generally underperform human reviews on these aspects . |
| Outcome: | The proposed model outperforms closed models on four aspects of review comments . the proposed model achieves agreement levels comparable to and exceeding those of human models . |
QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics for assessing question generation fail to take into account the input context of generation. |
| Approach: | They propose a context-aware Relevance evaluation metric for Question Generation that takes into account the context of question generation into account. |
| Outcome: | The proposed metric achieves higher correlation with human judgments while being much more robust to adversarial samples. |
DATScore: Evaluating Translation with Data Augmented Translations (2023.findings-eacl)
Copied to clipboard
| Challenge: | Experimental results show that DATScore correlates better with human meta-evaluations than the other recent state-of-the-art metrics. |
| Approach: | They propose to use data augmented translations to improve the evaluation of machine translations by using two new scoring strategies. |
| Outcome: | The proposed metric improves on 3 NLG tasks other than translation. |