Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, Sebastian Gehrmann
| Challenge: | Existing evaluation metrics that are not robust to dialect variation are difficult to measure for many groups of users and can penalize systems for producing text in lower-resource dialects. |
| Approach: | They propose a dialect-robust evaluation metric that produces the same score for system outputs that share the same semantics but are expressed in different dialects. |
| Outcome: | The proposed method significantly improves dialect robustness while preserving the correlation between automated metrics and human ratings. |
Similar Papers
A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models, we evaluated their robustness on multiple datasets. |
| Approach: | They propose a new metric for assessing model robustness by empirical evaluation of several models on multiple datasets. |
| Outcome: | The proposed metric is based on a set of datasets that are constructed by introducing naturally-occurring, non-malicious perturbations or by generating semantically equivalent paraphrases of input questions or statements. |
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale (2020.coling-main)
Copied to clipboard
| Challenge: | a few popular metrics are still used to evaluate language generation systems despite their known limitations. |
| Approach: | They propose to use automatic metrics to evaluate language generation systems . they show that they prefer system outputs to human-authored texts . |
| Outcome: | The proposed metrics are insensitive to correct translations of rare words and can yield high scores when given a single sentence as system output for the entire test set. |
RoMe: A Robust Metric for Evaluating Natural Language Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Empirical results suggest that RoMe has a stronger correlation to human judgment over state-of-the-art metrics in evaluating system-generated sentences across several NLG tasks. |
| Approach: | They propose an automatic evaluation metric incorporating several core aspects of natural language understanding (language competence, syntactic and semantic variation). |
| Outcome: | The proposed evaluation metric is trained on language features such as semantic similarity combined with tree edit distance and grammatical acceptability, using a self-supervised neural network. |
On the Effectiveness of Automated Metrics for Text Generation Systems (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods lack a sound theoretical foundation for evaluation campaigns . imperfect automated metrics and insufficiently sized test sets are some of the factors that cause uncertainty. |
| Approach: | They propose a theoretical framework that incorporates different sources of uncertainty, such as imperfect automated metrics and insufficiently sized test sets. |
| Outcome: | The proposed model can be leveraged to improve evaluation protocols regarding reliability, robustness, and significance of the evaluation outcome. |
GRUEN for Evaluating Linguistic Quality of Generated Text (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics focus on content selection, not linguistic quality . proposed GRUEN measures Grammaticality, non-redundancy, focUs, structure and coherence of generated text. |
| Approach: | They propose to use a BERT-based model and a class of syntactic, semantic, and contextual features to examine the system output. |
| Outcome: | Experiments show that the proposed metric correlates highly with human judgments. |
On the Blind Spots of Model-Based Evaluation Metrics for Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text generation evaluation metrics are lacking in robustness analysis. |
| Approach: | They propose to use stress tests to test for errors in text generation evaluation metrics . they find that BERTScore is confused by truncation errors in summarization . |
| Outcome: | The proposed stress tests show that they are insensitive to errors in open-ended generation, translation, and summarization. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
Exploring the Limitations of Detecting Machine-Generated Text (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in the quality of the generation of text by large language models have spurred research into identifying machine-generated text. |
| Approach: | They audit classification performance for detecting machine-generated text by evaluating on texts with varying writing styles. |
| Outcome: | The proposed methods are highly sensitive to stylistic changes and complexity, and in some cases degrade entirely to random classifiers. |
Dialetto, ma Quanto Dialetto? Transcribing and Evaluating Dialects on a Continuum (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent work on dialect variation in NLP treats dialects as discrete categories . dialect variation is a focus of increasing interest in the field . |
| Approach: | They examine performance differences between Italian dialects by incorporating performance data from different regions of the world. |
| Outcome: | The results show that performance disparities are due to dialects that are more similar to the standard variety. |
MENLI: Robust Evaluation Metrics from Natural Language Inference (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent proposed BERT-based evaluation metrics for text generation are vulnerable to adversarial attacks, e.g., relating to information correctness. |
| Approach: | They propose to use BERT-based evaluation metrics for text generation to evaluate text for semantic similarity but are vulnerable to adversarial attacks using Natural Language Inference. |
| Outcome: | The proposed metrics outperform existing summarization metrics but perform below SOTA MT metrics on standard benchmarks. |