Papers by Hebi Li
PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies on summarization evaluation without a human-written reference summary have shown high correlations with human ratings. |
| Approach: | They propose to judge summary quality by learning preference rank from corrupted summaries . they use Bradley-Terry power ranking model to learn preference rank . |
| Outcome: | Experiments on several datasets show that the proposed model can produce scores highly correlated with human ratings. |
SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on automatic summary evaluation metrics focus on lexical similarity and require a reference summary which is expensive to obtain. |
| Approach: | They propose to use a weakly supervised summary evaluation approach without the presence of reference summaries to transform existing summarization datasets into corrupted reference summarizers. |
| Outcome: | The proposed method outperforms baselines and shows that it improves linguistic quality over all metrics. |
DocAsRef: An Empirical Study on Repurposing Reference-based Summary Quality Metrics as Reference-free Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing reference-based metrics are limited by their reliance on human input. |
| Approach: | They propose to adapt some reference-based metrics to assess system summary against human-written references. |
| Outcome: | The proposed model outperforms reference-based metrics on two datasets and is comparable to reference-free metrics. |