| Challenge: | a recent human evaluation of AMR generation systems is compared to automated metrics. |
| Approach: | They propose a human evaluation which collects fluency and adequacy scores and categorization of error types for AMR generation systems. |
| Outcome: | The results show that human evaluations are more nuanced than automated metrics. |
Similar Papers
A Partially Rule-Based Approach to AMR Generation (N19-3)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a representation of a sentence as a labeled graph . because of these abstractions, it can be difficult to generate from AMR back to a fluent English sentence . |
| Approach: | They propose a new approach to generating English text from Abstract Meaning Representation (AMR) it is largely rule-based, supplemented by a language model and simple statistical linearization models . they also address difficulties of automatically evaluating AMR generation systems . |
| Outcome: | The proposed approach produces a fluent English sentence with a high quality . it is difficult to generate from an AMR back to a sentence which preserves original meaning . |
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale (2020.coling-main)
Copied to clipboard
| Challenge: | a few popular metrics are still used to evaluate language generation systems despite their known limitations. |
| Approach: | They propose to use automatic metrics to evaluate language generation systems . they show that they prefer system outputs to human-authored texts . |
| Outcome: | The proposed metrics are insensitive to correct translations of rare words and can yield high scores when given a single sentence as system output for the entire test set. |
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for judging metrics are sensitive to the translations used for evaluation, leading to falsely confident conclusions about a metric’s efficacy. |
| Approach: | They propose a method for thresholding performance improvement under an automatic metric against human judgements by using a pairwise system ranking method. |
| Outcome: | The proposed method allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. |
Factorising AMR generation through syntax (N19-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a semantic annotation framework which abstracts away from the surface form of text to capture the core 'who did what to whom' structure. |
| Approach: | They propose to decompose the generation process into two steps: first generate a syntactic structure, and then generate the surface form. |
| Outcome: | The proposed approach generates meaning-preserving syntactic paraphrases of the same graph, as judged by humans. |
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2021.eacl-main)
Copied to clipboard
| Challenge: | Abstract meaning representations are typically evaluated using surface matching metrics . however, there are problems with these metrics, since they allow multiple surface realizations . |
| Approach: | They propose a decomposable metric that measures the distance between the original and reconstructed AMRs. |
| Outcome: | The proposed metric reduces the complexity of the evaluation process by allowing for multiple surface realizations. |
Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies on automatic story generation (ASG) rely on human criteria, but there is little research on how well they correlate with human criteria. |
| Approach: | They propose to use human criteria to evaluate automatic story generation (ASG) their paper proposes to use HANNA to quantitatively evaluate correlations between 72 automatic metrics and human criteria. |
| Outcome: | The proposed model compared human criteria with automatic criteria and found that they were significantly better than human criteria. |
Multilingual AMR-to-Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on generating text from structured data into English has focused on bridging the gap between structure and natural language (NL) and semantically underspecified input and fully specified output. |
| Approach: | They propose a multilingual approach that can decode into 21 different languages . they leverage advances in cross-lingual embeddings and pretraining to generate multilingual models . |
| Outcome: | The proposed model surpasses baselines that generate into one language in eighteen languages. |
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing learning metrics are limited to tasks where large human ratings are available. |
| Approach: | They propose a model-based natural language generation (NLG) evaluation metric that is highly correlated with human judgements without requiring human annotation. |
| Outcome: | The proposed metric outperforms all prior unsupervised metrics on multiple NLG tasks including translation, image captioning, and WebNLG text generation. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
Toward Human-Like Evaluation for Natural Language Generation with Error Analysis (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have been used to evaluate language generation tasks . pretrained error analysis can be used to refine the generated sentence toward higher confidence . |
| Approach: | They propose to combine pretrained language model based metrics with human-like error analysis to improve sentence confidence. |
| Outcome: | The proposed method outperforms top-scoring metrics in 19/25 settings. |