A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021.eacl-main)
Copied to clipboard
| Challenge: | a lack of transparency is a key issue for robotics and AI. |
| Approach: | They propose to map existing automatic evaluation methods for natural language generation onto explanations. |
| Outcome: | The proposed model shows that embedding-based evaluation methods have higher correlations with human ratings than word-overlap metrics. |
Similar Papers
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
NLG-Metricverse: An End-to-End Library for Evaluating Natural Language Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Natural language generation models are a key component of deep learning, says aaron eliott . he says it is crucial to develop and apply better metrics for NLG evaluation . |
| Approach: | a new open-source library for NLG evaluation is created to facilitate researchers to judge the effectiveness of their models. the framework provides a living collection of NLG metrics in a unified and easy-to-use environment. |
| Outcome: | a new open-source library for NLG evaluation aims to improve performance of models . the framework provides tools to apply, analyze, compare, and visualize the metrics . |
Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that generate NL explanations for tasks have been evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations. |
| Approach: | They propose to use a model as a proxy for a human observer to evaluate NL explanations from the model simulatability perspective. |
| Outcome: | The proposed model-generated explanations are evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations. |
NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist (2023.acl-long)
Copied to clipboard
| Challenge: | a systematic review of automatic evaluation metrics for Natural Language Generation (NLG) shows that task-agnostic metrics have a weak correlation with human . |
| Approach: | They propose a framework to assess the effectiveness of automatic metrics in three NLG tasks . they propose task-agnostic and human-aligned metrics to be used for evaluation . |
| Outcome: | The proposed framework provides access to the evaluation tools for three NLG tasks. |
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing evaluation methods for natural language generation rely on token-level or embedding-level comparisons with text references. |
| Approach: | They propose to use text-to-image generator to generate an image as the embodied imagination for the text snippet and compute the imagination similarity using contextual embeddings. |
| Outcome: | The proposed metric improves existing evaluation metrics’ correlations with human similarity judgments in both reference-based and reference-free scenarios. |
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021.emnlp-main)
Copied to clipboard
| Challenge: | Evaluation metrics are a key ingredient for progress of text generation systems . a class of novel evaluation metrics based on BERT and its variants has been explored . |
| Approach: | They propose to disentangle BERT-based evaluation metrics along linguistic factors . they show they are sensitive to lexical overlap, just like BLEU and ROUGE . |
| Outcome: | The proposed metrics capture all aspects but are sensitive to lexical overlap, just like BLEU and ROUGE, the authors show . |
RoMe: A Robust Metric for Evaluating Natural Language Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Empirical results suggest that RoMe has a stronger correlation to human judgment over state-of-the-art metrics in evaluating system-generated sentences across several NLG tasks. |
| Approach: | They propose an automatic evaluation metric incorporating several core aspects of natural language understanding (language competence, syntactic and semantic variation). |
| Outcome: | The proposed evaluation metric is trained on language features such as semantic similarity combined with tree edit distance and grammatical acceptability, using a self-supervised neural network. |
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications (2022.naacl-main)
Copied to clipboard
| Challenge: | Evaluating natural language generation systems is difficult, as there are many ways to express similar things in text. |
| Approach: | They combine interviews with NLG practitioners to examine ethical considerations and their implications for NLG evaluation. |
| Outcome: | The findings of the study surface goals, community practices, assumptions, and constraints that shape NLG evaluations, and examine their implications and how they embody ethical considerations. |
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification (N18-1)
Copied to clipboard
| Challenge: | Text classification models are becoming increasingly complex and opaque, however for many applications it is essential that the models are interpretable. |
| Approach: | They propose to use automatic word deletion to generate local explanations for a text classification model by crowdsourcing the evaluation using a crowdsourced experiment. |
| Outcome: | The proposed evaluations of local explanations using crowdsourcing and automatic measures correlate with the results. |
Automatic Evaluation of Language Generation Technology Based on Structure Alignment (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for automatic evaluation ignore syntax of sentences despite its importance in determining meaning. |
| Approach: | They propose an automatic evaluation metric that considers both the words in sentences and their syntactic structures. |
| Outcome: | The proposed method is comparable to baselines from two NLP tasks. |