Inherent Biases in Reference-based Evaluation for Grammatical Error Correction (P18-1)
Copied to clipboard
| Challenge: | Existing evaluation systems obtain comparable or superior performance compared to humans by making few but targeted changes to the input. |
| Approach: | They propose to re-scale M 2 by the inter-annotator agreement and increase the number of references in any feasible range to overcome low coverage bias in GEC evaluation. |
| Outcome: | The proposed measure overcomes low coverage bias in GEC evaluation by re-scaling or increasing the number of references in any feasible range. |
Similar Papers
How Good (really) are Grammatical Error Correction Systems? (2021.eacl-main)
Copied to clipboard
| Challenge: | Standard evaluations of Grammatical Error Correction systems use a fixed reference text generated relative to the original text. |
| Approach: | They propose to use a gold reference text to evaluate Grammatical Error Correction systems that is generated relative to the original text and is independent of the system output. |
| Outcome: | The proposed evaluations show that the system performs 20-40 points better than standard evaluations. |
Grammatical Error Correction: Are We There Yet? (2022.coling-1)
Copied to clipboard
| Challenge: | grammatical error correction (GEC) systems outperform humans on the CoNLL-2014 test set, but there are still classes of errors that they fail to correct. |
| Approach: | They found that state-of-the-art GEC systems outperform humans by a wide margin on the CoNLL-2014 test set . however, they found that there are still classes of errors that they fail to correct . |
| Outcome: | The F0.5 evaluation metric outperforms the CoNLL-2014 test set, but there are still classes of errors that they fail to correct. |
Is this the end of the gold standard? A straightforward reference-less grammatical error correction metric (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations of grammatical error correction systems use reference-based metrics, but they are limited because of multiple correct outputs. |
| Approach: | They propose a system that uses commonly available tools to evaluate grammatical error correction (GEC) systems. |
| Outcome: | The proposed system solves the issues related to the use of a reference and does not need another annotated dataset for fine-tuning. |
A Reassessment of Reference-Based Grammatical Error Correction Metrics (C18-1)
Copied to clipboard
| Challenge: | Existing studies on the correlation of GEC metrics with human judgments were inconclusive . a recent study found that GLEU produces counter-intuitive scores in common test examples . |
| Approach: | They propose to use GLEU to evaluate grammatical error correction (GEC) systems . they also use statistical significance tests to assess their agreement with human judgments . |
| Outcome: | The proposed metrics show no significant advantage over MaxMatch (GLEU) the results contradict previous studies that claim GLEU superior . |
Evaluation of Really Good Grammatical Error Correction (2024.lrec-main)
Copied to clipboard
| Challenge: | emergence of large language models has highlighted the shortcomings of evaluation methods . evaluators often use grammatical error correction (GEC) to correct language errors at multiple levels . |
| Approach: | They perform a comprehensive evaluation of various GEC systems using Swedish learner texts . they suggest using human post-editing to analyze amount of change required to reach native-level human performance . |
| Outcome: | The proposed evaluations outperform existing methods for grammatical error correction in Swedish . the results highlight the shortcomings of existing evaluation methods . |
Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses (2020.emnlp-main)
Copied to clipboard
| Challenge: | CWEB is a new benchmark for grammatical error correction (GEC) systems . website data contains far fewer grammamatical errors than learner essays . |
| Approach: | They propose to broaden the target domain of grammatical error correction (GEC) systems . website data contains far fewer grammamatical errors than learner essays . |
| Outcome: | The proposed model can't rely on a strong internal language model in low error density domains. |
CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating the performance of Grammatical Error Correction systems is a challenging task due to its subjectivity. |
| Approach: | They propose a method to evaluate GEC systems in multi-reference evaluation setting . they use consistent edit boundaries to eliminate bias caused by inconsistent edit boundaries . |
| Outcome: | The proposed evaluation metric eliminates bias caused by inconsistent edit boundaries on six English reference sets. |
Reliability Crisis of Reference-free Metrics for Grammatical Error Correction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Reference-free evaluation metrics for grammatical error correction have high correlation with human judgments, but they are not designed to evaluate adversarial systems that aim to obtain unjustifiably high scores. |
| Approach: | They propose adversarial attack strategies for four reference-free metrics . they propose SOME, Scribendi, IMPARA, and LLM-based metrics based on these metrics a . |
| Outcome: | The proposed attacks outperform the current state-of-the-art for four reference-free metrics . |
Mitigating Exposure Bias in Grammatical Error Correction with Data Augmentation and Reweighting (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing approaches to grammatical error correction (GEC) use sequence-to-sequence models, but there is an exposure bias problem. |
| Approach: | They propose a data manipulation approach to overcome the exposure bias problem in seq2seq GEC . they propose augmentation methods to mimic decoder input and reweighting methods to automatically balance the importance of each kind of augmented samples. |
| Outcome: | The proposed method improves on benchmark GEC datasets. |
GPT-3.5 for Grammatical Error Correction (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent work shows that GPT-3.5 struggles with several error types, including punctuation mistakes, tense errors, syntactic dependencies between words, and lexical compatibility at the sentence level. |
| Approach: | They evaluate GPT-3.5 for grammatical error correction in multiple languages . they use it to re-rank correction hypotheses generated by other GEC models . |
| Outcome: | The proposed model performs well in English and Russian, but struggles with errors in other languages. |