Papers with SummEval
Fusion-Eval: Integrating Assistant Evaluators with LLMs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent studies have employed large language models (LLMs) as reference-free metrics for NLG evaluation, enhancing adaptability to new tasks tasks. |
| Approach: | They propose a method that leverages large language models to integrate insights from various assistant evaluators. |
| Outcome: | The proposed approach achieves a 0.962 system-level Kendall-Tau correlation with humans on SummEval and a 0.7444 turn-level Spearman correlation on TopicalChat, which is significantly higher than baseline methods. |
SAJA: A Simple Approach to Judge Alignment for LLM-as-a-Judge (2026.acl-industry)
Copied to clipboard
| Challenge: | Current approaches to evaluate text at scale require multiple calls and per-dataset prompt tuning. |
| Approach: | They propose a model-agnostic approach to evaluate judge alignment that uses a lightweight calibration head. |
| Outcome: | a new model with SAJA matches more complex systems across four evaluation paradigms . it outperforms uncalibrated models on MT-Bench pairwise preference and competitive performance on five classification benchmarks compared to uncalibred models . |
HaRiM+: Evaluating Summary Quality with Hallucination Risk (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing summarization models are limited in measuring the factual inconsistency of generated summaries. |
| Approach: | They propose a decoder overconfidence-regularizing objective as a hallucination risk measurement to better estimate the quality of generated summaries. |
| Outcome: | The proposed metric is reference-free and requires no training or modules . it records state-of-the-art correlation to human judgment on three sets of summary-quality annotations. |
DocAsRef: An Empirical Study on Repurposing Reference-based Summary Quality Metrics as Reference-free Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing reference-based metrics are limited by their reliance on human input. |
| Approach: | They propose to adapt some reference-based metrics to assess system summary against human-written references. |
| Outcome: | The proposed model outperforms reference-based metrics on two datasets and is comparable to reference-free metrics. |
DEBATE: Devil’s Advocate-Based Assessment and Text Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for evaluating the quality of machine-generated texts have a relatively low correlation with human performance. |
| Approach: | They propose an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil’s Advocate. |
| Outcome: | The proposed evaluation framework outperforms the previous state-of-the-art methods in two meta-evaluation benchmarks in NLG evaluation, SummEval and TopicalChat. |
BMX: Boosting Natural Language Generation Metrics with Explainability (2024.findings-eacl)
Copied to clipboard
| Challenge: | Modern language model (LM) based natural language generation evaluation metrics achieve astonishing results in grading machine generated sentences like humans would. |
| Approach: | They use word-level explanations to convert a word- level score into a segment-level score and combine this segment- level with the original metric to obtain a better metric. |
| Outcome: | The proposed metrics improve across MT and summarization datasets while achieving small improvements on machine translation. |
CourtEval: A Courtroom-Based Multi-Agent Evaluation Framework (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing automated evaluation metrics like ROUGE and BLEU show low correlation with human judgments. |
| Approach: | They propose a multi-agent evaluation framework that integrates multiple agents . they use ROUGE and BLEU to evaluate natural language models . |
| Outcome: | The proposed evaluation framework outperforms the current state-of-the-art methods in two meta-evaluation benchmarks. |
Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) offer new opportunities to enhance the annotation process, particularly for detecting label errors in existing datasets. |
| Approach: | They propose to use an ensemble of large language models to flag mislabeled examples by using an LLM-as-a-judge approach to detect label errors in existing datasets. |
| Outcome: | The proposed method improves label accuracy and consistency in large language models. |