| Challenge: | Existing studies on summary quality measure have shown that it correlates well with quality scores produced by human annotators. |
| Approach: | They propose to use a criterion that does not rely on human scores to judge summary quality . they propose to develop a method that can be used to determine the best measure from a family of measures . |
| Outcome: | The proposed measure could be used to determine the best summary quality measure from a family of measures. |
Similar Papers
Studying Summarization Evaluation Metrics in the Appropriate Scoring Range (P19-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics are compared based on their ability to correlate with humans, but they disagree in the higher-scoring range in which current systems operate. |
| Approach: | They show that evaluation metrics which behave similarly on these datasets strongly disagree in the higher-scoring range in which current systems operate. |
| Outcome: | The evaluation metrics which behave similarly on these datasets strongly disagree in the higher-scoring range in which current systems operate. |
How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to evaluate summary coherence are often evaluated using disparate datasets and metrics. |
| Approach: | They propose to use automatic evaluation to evaluate coherence of summaries by selecting high-scoring candidates. |
| Outcome: | The proposed methods show that they can perform better on an even playing field. |
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing definitions of system-level correlations are inconsistent with how they are used to evaluate systems. |
| Approach: | They propose to calculate correlations only on pairs of systems separated by small differences in automatic scores . they propose to use the full test set instead of the subset of summaries judged by humans . |
| Outcome: | The proposed changes improve the accuracy of the estimated correlations on pairs of systems separated by small differences in automatic scores. |
Re-Examining Summarization Evaluation across Multiple Quality Criteria (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a number of automated evaluation metrics are evaluated by multiple quality criteria, such as relevance, consistency, fluency and coherence. |
| Approach: | They propose a method that removes the confounding variable and detects unreliable correlations. |
| Outcome: | The proposed method detects unreliable correlations between QCs and human scores . it is based on a multi-QC setup, but it fails to detect summary corruptions . |
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023.acl-long)
Copied to clipboard
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Existing studies for summarization evaluation exhibit low inter-annotator agreement or lack scale. |
| Approach: | They propose a modified summarization salience protocol based on fine-grained semantic units and a robust summarizing evaluation benchmark. |
| Outcome: | The proposed protocol is based on fine-grained semantic units and allows for high inter-annotator agreement. |
Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics for summarization are reference-based and correlate poorly with relevance . fluency, faithfulness, coherence and relevance are all measures of human evaluation . |
| Approach: | They propose a reference-free metric that correlates well with human evaluated relevance . n-gram importance weighting is used to weight a summary's importance . |
| Outcome: | The proposed metric can be used along reference-based metrics to improve their robustness in low quality reference settings. |
How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation (2021.eacl-main)
Copied to clipboard
| Challenge: | Current manual evaluation methods for text summarization have low correlation with human judgements on summary quality. |
| Approach: | They conduct two evaluation experiments on two aspects of summaries’ linguistic quality (coherence and repetitiveness) they find that study parameters such as the total number of annotators and distribution of annotes to annotation items are often not fully reported. |
| Outcome: | The proposed methods can inflate type I errors up to eight-fold and the overall number of annotators can have a strong impact on study power. |
Evaluating and Characterizing Human Rationales (2020.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines how human rationales perform on automatic metrics . human-generated rationale evaluation is difficult because of its ambiguity . |
| Approach: | They propose to use model-dependent baseline performance to evaluate rationale quality . they propose to also use "fidelity curves" to reveal properties such as irrelevance and redundancy . |
| Outcome: | The proposed methods characterize rationale quality based on model retraining and using "fidelity curves" the proposed methods lead to actionable suggestions for evaluating and characterizing rationales . |
Does Summary Evaluation Survive Translation to Other Languages? (2022.naacl-main)
Copied to clipboard
| Challenge: | a quality summarization dataset requires the production and evaluation of summaries by trained humans and machines. |
| Approach: | They translate a summarization dataset in English and compare its performance to seven languages . they explore equivalence testing as an appropriate statistical paradigm for evaluating correlations between human and automated scoring of summaries . |
| Outcome: | The proposed method could be used in seven languages and compares performance across measures. |
PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies on summarization evaluation without a human-written reference summary have shown high correlations with human ratings. |
| Approach: | They propose to judge summary quality by learning preference rank from corrupted summaries . they use Bradley-Terry power ranking model to learn preference rank . |
| Outcome: | Experiments on several datasets show that the proposed model can produce scores highly correlated with human ratings. |