| Challenge: | Existing methods for automatic evaluation of grammatical error correction require multiple reference sentences or manual scores. |
| Approach: | They propose an Impact-based Metric for GEC using PARAllel data, IMPARA . IMPRA computes correction impacts computed by parallel data comprising pairs of grammatical/ungrammatically-spaced sentences. |
| Outcome: | The proposed method can perform evaluations that fit different domains and correction styles. |
Similar Papers
IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing reference-free automatic grammatical error correction methods do not correlate with human evaluation. |
| Approach: | They propose a reference-free automatic grammatical error correction evaluation method with enhanced gramma-ed capabilities. |
| Outcome: | The proposed method achieves highest correlation with human evaluations on a meta-evaluation dataset. |
gec-metrics: A Unified Library for Grammatical Error Correction Evaluation (2025.acl-demo)
Copied to clipboard
| Challenge: | a library for using and developing grammatical error correction (GEC) evaluation metrics is released under the MIT license . |
| Approach: | They propose a library for using and developing grammatical error correction (GEC) evaluation metrics through a unified interface. |
| Outcome: | The proposed method is based on a unified evaluation framework with a strong focus on API usage and extensible. |
Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human? (2025.acl-short)
Copied to clipboard
| Challenge: | Existing automatic evaluation metrics are based on procedures that diverge from human evaluation. |
| Approach: | They propose to aggregate automatic evaluation metrics to bridge this gap . they propose to use edit-based metrics, -gram based metrics and sentence-level metrics to find the best ranking system. |
| Outcome: | The proposed method outperforms existing metrics on the SEEDA benchmark and improves edit-based metrics, -gram based metrics and sentence-level metrics. |
Is this the end of the gold standard? A straightforward reference-less grammatical error correction metric (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations of grammatical error correction systems use reference-based metrics, but they are limited because of multiple correct outputs. |
| Approach: | They propose a system that uses commonly available tools to evaluate grammatical error correction (GEC) systems. |
| Outcome: | The proposed system solves the issues related to the use of a reference and does not need another annotated dataset for fine-tuning. |
SOME: Reference-less Sub-Metrics Optimized for Manual Evaluations of Grammatical Error Correction (2020.coling-main)
Copied to clipboard
| Challenge: | Existing reference-less metrics are not optimized for manual evaluations of system outputs because no dataset exists for manual analysis. |
| Approach: | They propose a reference-less metric trained on manual evaluations of system outputs for grammatical error correction. |
| Outcome: | The proposed metric improves correlation with manual evaluation in system- and sentence-level meta-evaluation. |
Reliability Crisis of Reference-free Metrics for Grammatical Error Correction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Reference-free evaluation metrics for grammatical error correction have high correlation with human judgments, but they are not designed to evaluate adversarial systems that aim to obtain unjustifiably high scores. |
| Approach: | They propose adversarial attack strategies for four reference-free metrics . they propose SOME, Scribendi, IMPARA, and LLM-based metrics based on these metrics a . |
| Outcome: | The proposed attacks outperform the current state-of-the-art for four reference-free metrics . |
CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction (2025.acl-long)
Copied to clipboard
Jingheng Ye, Zishan Xu, Yinghui Li, Linlin Song, Qingyu Zhou, Hai-Tao Zheng, Ying Shen, Wenhao Jiang, Hong-Gee Kim, Ruitong Liu, Xin Su, Zifei Shan
| Challenge: | Existing studies have focused on the interpretability of Grammatical Error Correction (GEC) evaluation metrics, but the interpretabilty of these metrics has been neglected. |
| Approach: | They propose a reference-based metric that describes four aspects of GEC systems: hit-correction, wrong-corrections, under-correcties, and over-corrects. |
| Outcome: | The proposed metric reveals critical qualities and locates drawbacks of GEC systems. |
Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for Grammatical error correction lack explainability . lack of explainability hinders researchers from analyzing strengths and weaknesses of models . |
| Approach: | They propose to assign sentence-level scores to individual edits to improve GEC performance . they use Shapley values, from cooperative game theory, to compute contribution of each edit . |
| Outcome: | The proposed method shows that the evaluation metrics are consistent across edits and human evaluations. |
Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses (2020.emnlp-main)
Copied to clipboard
| Challenge: | CWEB is a new benchmark for grammatical error correction (GEC) systems . website data contains far fewer grammamatical errors than learner essays . |
| Approach: | They propose to broaden the target domain of grammatical error correction (GEC) systems . website data contains far fewer grammamatical errors than learner essays . |
| Outcome: | The proposed model can't rely on a strong internal language model in low error density domains. |
A Reassessment of Reference-Based Grammatical Error Correction Metrics (C18-1)
Copied to clipboard
| Challenge: | Existing studies on the correlation of GEC metrics with human judgments were inconclusive . a recent study found that GLEU produces counter-intuitive scores in common test examples . |
| Approach: | They propose to use GLEU to evaluate grammatical error correction (GEC) systems . they also use statistical significance tests to assess their agreement with human judgments . |
| Outcome: | The proposed metrics show no significant advantage over MaxMatch (GLEU) the results contradict previous studies that claim GLEU superior . |