SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes (2023.acl-long)
Copied to clipboard
| Challenge: | Existing learned metrics perform unsatisfactory across text generation tasks or require human annotations for training on specific tasks. |
| Approach: | They propose a self-supervised approach to train a model-based metric for text generation evaluation using sentences retrieved from a corpus. |
| Outcome: | The proposed model outperforms all prior unsupervised metrics on four text generation evaluation benchmarks, with an average Kendall improvement of 0.158. |
Similar Papers
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing learning metrics are limited to tasks where large human ratings are available. |
| Approach: | They propose a model-based natural language generation (NLG) evaluation metric that is highly correlated with human judgements without requiring human annotation. |
| Outcome: | The proposed metric outperforms all prior unsupervised metrics on multiple NLG tasks including translation, image captioning, and WebNLG text generation. |
On the Blind Spots of Model-Based Evaluation Metrics for Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text generation evaluation metrics are lacking in robustness analysis. |
| Approach: | They propose to use stress tests to test for errors in text generation evaluation metrics . they find that BERTScore is confused by truncation errors in summarization . |
| Outcome: | The proposed stress tests show that they are insensitive to errors in open-ended generation, translation, and summarization. |
Toward Human-Like Evaluation for Natural Language Generation with Error Analysis (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have been used to evaluate language generation tasks . pretrained error analysis can be used to refine the generated sentence toward higher confidence . |
| Approach: | They propose to combine pretrained language model based metrics with human-like error analysis to improve sentence confidence. |
| Outcome: | The proposed method outperforms top-scoring metrics in 19/25 settings. |
INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate the quality of language generation do not provide explicit explanation of their verdicts. |
| Approach: | They propose a fine-grained explainable evaluation metric for text generation that harnesses human instruction and implicit knowledge of GPT-4 to fine-tune it. |
| Outcome: | The proposed model outperforms all other unsupervised metrics on translation, captioning, data-to-text, and commonsense generation tasks. |
DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability. |
| Approach: | They propose a metric that evaluates natural language generation tasks as an instruction-style question answering task and utilizes instruction-tuned pre-trained language models without training on evaluation datasets. |
| Outcome: | The proposed metric achieves state-of-the-art performance in untrained metrics for evaluating text summarization and dialogue generation, which exhibits strong dimension-level / task-level generalization ability and interpretability. |
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale (2020.coling-main)
Copied to clipboard
| Challenge: | a few popular metrics are still used to evaluate language generation systems despite their known limitations. |
| Approach: | They propose to use automatic metrics to evaluate language generation systems . they show that they prefer system outputs to human-authored texts . |
| Outcome: | The proposed metrics are insensitive to correct translations of rare words and can yield high scores when given a single sentence as system output for the entire test set. |
Correction of Errors in Preference Ratings from Automated Metrics for Text Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods are over-confident in assigning significant differences between systems . Currently, the most reliable evaluation methods for text generation are human-based evaluations. |
| Approach: | They propose to combine human ratings with automated ratings to reduce the amount of human ratings needed to arrive at robust results. |
| Outcome: | The proposed evaluation protocol reduces the amount of human ratings by 50% while yielding the same evaluation outcome as the pure human evaluation in 95% of cases. |
BLEURT: Learning Robust Metrics for Text Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Text generation has made significant advances, but evaluation metrics have lagged behind. |
| Approach: | They propose a learning evaluation metric for English based on BERT . BLEURT can model human judgment with a few thousand possibly biased training examples . |
| Outcome: | The proposed model can model human judgment with a few thousand potentially biased training examples. |
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing reference-free metrics have obvious limitations for evaluating controlled text generation models. |
| Approach: | They propose an unsupervised reference-free metric which evaluates controlled text generation from different aspects by formulating each aspect into multiple text infilling tasks. |
| Outcome: | The proposed metric has higher correlations with human judgments while obtaining better generalization of evaluating generated texts from different models and with different qualities. |
How Do Seq2Seq Models Perform on End-to-End Data-to-Text Generation? (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models for data-to-text generation are based on pipelines and end-to end architectures. |
| Approach: | They use multidimensional quality metrics to evaluate models on end-to-end data-totext generation and compare their performance against pipeline models. |
| Outcome: | The proposed model improves in Omission and Inaccuracy Extrinsic errors but increases errors such as Addition. |