CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing reference-free metrics have obvious limitations for evaluating controlled text generation models. |
| Approach: | They propose an unsupervised reference-free metric which evaluates controlled text generation from different aspects by formulating each aspect into multiple text infilling tasks. |
| Outcome: | The proposed metric has higher correlations with human judgments while obtaining better generalization of evaluating generated texts from different models and with different qualities. |
Similar Papers
On the Limitations of Reference-Free Evaluations of Generated Text (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has shown that evaluation metrics which accurately estimate the quality of generated text are limited in their ability to evaluate generated text. |
| Approach: | They argue that reference-free metrics are limited in their ability to evaluate generated text . they recommend that they be used as diagnostic tools for analyzing and understanding model behavior . |
| Outcome: | The proposed evaluation metrics are limited in their ability to evaluate generated text . they can be optimized at test time, can be biased against models with similar outputs . |
SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes (2023.acl-long)
Copied to clipboard
| Challenge: | Existing learned metrics perform unsatisfactory across text generation tasks or require human annotations for training on specific tasks. |
| Approach: | They propose a self-supervised approach to train a model-based metric for text generation evaluation using sentences retrieved from a corpus. |
| Outcome: | The proposed model outperforms all prior unsupervised metrics on four text generation evaluation benchmarks, with an average Kendall improvement of 0.158. |
INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate the quality of language generation do not provide explicit explanation of their verdicts. |
| Approach: | They propose a fine-grained explainable evaluation metric for text generation that harnesses human instruction and implicit knowledge of GPT-4 to fine-tune it. |
| Outcome: | The proposed model outperforms all other unsupervised metrics on translation, captioning, data-to-text, and commonsense generation tasks. |
NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist (2023.acl-long)
Copied to clipboard
| Challenge: | a systematic review of automatic evaluation metrics for Natural Language Generation (NLG) shows that task-agnostic metrics have a weak correlation with human . |
| Approach: | They propose a framework to assess the effectiveness of automatic metrics in three NLG tasks . they propose task-agnostic and human-aligned metrics to be used for evaluation . |
| Outcome: | The proposed framework provides access to the evaluation tools for three NLG tasks. |
Spurious Correlations in Reference-Free Evaluation of Text Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work suggests that reference-free evaluation metrics may rely on spurious correlations with human judgments. |
| Approach: | They propose to use model-based, reference-free evaluation metrics to evaluate natural language generation systems. |
| Outcome: | The proposed metrics achieve high correlations with human judgments, but they may not be robust enough to evaluate their efficacy and robustness. |
REFeREE: A REference-FREE Model-Based Metric for Text Simplification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for text simplification lack a universal standard of quality and require a small number of human annotations. |
| Approach: | They propose to introduce a reference-free model-based metric with a 3-stage curriculum that can be applied to any quality standard with fewer annotations. |
| Outcome: | The proposed metric outperforms existing reference-based metrics in predicting ratings while requiring no reference simplifications at inference time. |
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing learning metrics are limited to tasks where large human ratings are available. |
| Approach: | They propose a model-based natural language generation (NLG) evaluation metric that is highly correlated with human judgements without requiring human annotation. |
| Outcome: | The proposed metric outperforms all prior unsupervised metrics on multiple NLG tasks including translation, image captioning, and WebNLG text generation. |
HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks for LLM-as-a-judge use zero-shot setting without consulting any human input, which leads to low alignment, or fine-tune LLMs on labeled data, which requires a non-trivial number of samples. |
| Approach: | They propose a hypothesis-guided evaluation framework that uses a small corpus of human evaluations to generate more detailed rubrics for human judgments and incorporates a checklist-like approach to combine LLM’s assigned scores on each decomposed dimension to acquire overall scores. |
| Outcome: | The proposed framework outperforms existing frameworks in both human rankings and human scores with 30 human evaluations and fine-tunes LLMs on labeled data with 3 times more human evaluation by 11.95%. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for evaluating text quality are discriminative and generative . current methods use manual annotation of human judgements to train them . |
| Approach: | They propose a framework that combines the best of both worlds by using supervised and unsupervised signals from whatever data we have available. |
| Outcome: | The proposed method outperforms existing metrics on 5 datasets, 19 languages and 280 systems. |