Reliability Crisis of Reference-free Metrics for Grammatical Error Correction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Reference-free evaluation metrics for grammatical error correction have high correlation with human judgments, but they are not designed to evaluate adversarial systems that aim to obtain unjustifiably high scores. |
| Approach: | They propose adversarial attack strategies for four reference-free metrics . they propose SOME, Scribendi, IMPARA, and LLM-based metrics based on these metrics a . |
| Outcome: | The proposed attacks outperform the current state-of-the-art for four reference-free metrics . |
Similar Papers
Is this the end of the gold standard? A straightforward reference-less grammatical error correction metric (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations of grammatical error correction systems use reference-based metrics, but they are limited because of multiple correct outputs. |
| Approach: | They propose a system that uses commonly available tools to evaluate grammatical error correction (GEC) systems. |
| Outcome: | The proposed system solves the issues related to the use of a reference and does not need another annotated dataset for fine-tuning. |
IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing reference-free automatic grammatical error correction methods do not correlate with human evaluation. |
| Approach: | They propose a reference-free automatic grammatical error correction evaluation method with enhanced gramma-ed capabilities. |
| Outcome: | The proposed method achieves highest correlation with human evaluations on a meta-evaluation dataset. |
Grammatical Error Correction Systems for Automated Assessment: Are They Susceptible to Universal Adversarial Attacks? (2022.aacl-main)
Copied to clipboard
| Challenge: | With advances in deep learning, GEC systems are susceptible to adversarial attacks, in which a small change at the input can cause large undesired changes at the output. |
| Approach: | They propose to use a concatenative universal attack to deceive the system into not correcting grammatical errors to create the perception of higher language ability. |
| Outcome: | The proposed attack can deceive the system into not correcting (concealing) grammatical errors to create the perception of higher language ability. |
A Reassessment of Reference-Based Grammatical Error Correction Metrics (C18-1)
Copied to clipboard
| Challenge: | Existing studies on the correlation of GEC metrics with human judgments were inconclusive . a recent study found that GLEU produces counter-intuitive scores in common test examples . |
| Approach: | They propose to use GLEU to evaluate grammatical error correction (GEC) systems . they also use statistical significance tests to assess their agreement with human judgments . |
| Outcome: | The proposed metrics show no significant advantage over MaxMatch (GLEU) the results contradict previous studies that claim GLEU superior . |
SOME: Reference-less Sub-Metrics Optimized for Manual Evaluations of Grammatical Error Correction (2020.coling-main)
Copied to clipboard
| Challenge: | Existing reference-less metrics are not optimized for manual evaluations of system outputs because no dataset exists for manual analysis. |
| Approach: | They propose a reference-less metric trained on manual evaluations of system outputs for grammatical error correction. |
| Outcome: | The proposed metric improves correlation with manual evaluation in system- and sentence-level meta-evaluation. |
Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human? (2025.acl-short)
Copied to clipboard
| Challenge: | Existing automatic evaluation metrics are based on procedures that diverge from human evaluation. |
| Approach: | They propose to aggregate automatic evaluation metrics to bridge this gap . they propose to use edit-based metrics, -gram based metrics and sentence-level metrics to find the best ranking system. |
| Outcome: | The proposed method outperforms existing metrics on the SEEDA benchmark and improves edit-based metrics, -gram based metrics and sentence-level metrics. |
Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for Grammatical error correction lack explainability . lack of explainability hinders researchers from analyzing strengths and weaknesses of models . |
| Approach: | They propose to assign sentence-level scores to individual edits to improve GEC performance . they use Shapley values, from cooperative game theory, to compute contribution of each edit . |
| Outcome: | The proposed method shows that the evaluation metrics are consistent across edits and human evaluations. |
gec-metrics: A Unified Library for Grammatical Error Correction Evaluation (2025.acl-demo)
Copied to clipboard
| Challenge: | a library for using and developing grammatical error correction (GEC) evaluation metrics is released under the MIT license . |
| Approach: | They propose a library for using and developing grammatical error correction (GEC) evaluation metrics through a unified interface. |
| Outcome: | The proposed method is based on a unified evaluation framework with a strong focus on API usage and extensible. |
How Good (really) are Grammatical Error Correction Systems? (2021.eacl-main)
Copied to clipboard
| Challenge: | Standard evaluations of Grammatical Error Correction systems use a fixed reference text generated relative to the original text. |
| Approach: | They propose to use a gold reference text to evaluate Grammatical Error Correction systems that is generated relative to the original text and is independent of the system output. |
| Outcome: | The proposed evaluations show that the system performs 20-40 points better than standard evaluations. |
CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction (2025.acl-long)
Copied to clipboard
Jingheng Ye, Zishan Xu, Yinghui Li, Linlin Song, Qingyu Zhou, Hai-Tao Zheng, Ying Shen, Wenhao Jiang, Hong-Gee Kim, Ruitong Liu, Xin Su, Zifei Shan
| Challenge: | Existing studies have focused on the interpretability of Grammatical Error Correction (GEC) evaluation metrics, but the interpretabilty of these metrics has been neglected. |
| Approach: | They propose a reference-based metric that describes four aspects of GEC systems: hit-correction, wrong-corrections, under-correcties, and over-corrects. |
| Outcome: | The proposed metric reveals critical qualities and locates drawbacks of GEC systems. |