| Challenge: | Existing evaluation datasets based on learner-produced texts are insufficient for evaluating models . Currently, sequence-to-sequence models and sequence tagging models perform well on beginner-level grammar items . |
| Approach: | They propose a new evaluation paradigm that assesses GEC models using minimal pairs of ungrammatical and grammatically paired sentences for each grammar item. |
| Outcome: | The proposed evaluation paradigm assesses models using minimal pairs of ungrammatical and grammatically-spaced sentences for each grammar item. |
Similar Papers
Cross-Corpora Evaluation and Analysis of Grammatical Error Correction Models — Is Single-Corpus Evaluation Enough? (N19-1)
Copied to clipboard
| Challenge: | Existing studies have evaluated grammatical error correction models on a single corpus, but the evaluation is incomplete because the task difficulty varies depending on the corpus and conditions such as proficiency levels of the writers and essay topics. |
| Approach: | They evaluate the performance of several GEC models against various learner corpora and compare their rankings against the corpus. |
| Outcome: | The evaluation of several models against learner corpora shows that the models’ rankings vary depending on the corpus, indicating that single-corpus evaluation is insufficient for GEC models. |
Evaluation of Really Good Grammatical Error Correction (2024.lrec-main)
Copied to clipboard
| Challenge: | emergence of large language models has highlighted the shortcomings of evaluation methods . evaluators often use grammatical error correction (GEC) to correct language errors at multiple levels . |
| Approach: | They perform a comprehensive evaluation of various GEC systems using Swedish learner texts . they suggest using human post-editing to analyze amount of change required to reach native-level human performance . |
| Outcome: | The proposed evaluations outperform existing methods for grammatical error correction in Swedish . the results highlight the shortcomings of existing evaluation methods . |
Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses (2020.emnlp-main)
Copied to clipboard
| Challenge: | CWEB is a new benchmark for grammatical error correction (GEC) systems . website data contains far fewer grammamatical errors than learner essays . |
| Approach: | They propose to broaden the target domain of grammatical error correction (GEC) systems . website data contains far fewer grammamatical errors than learner essays . |
| Outcome: | The proposed model can't rely on a strong internal language model in low error density domains. |
Grammatical Error Correction: Are We There Yet? (2022.coling-1)
Copied to clipboard
| Challenge: | grammatical error correction (GEC) systems outperform humans on the CoNLL-2014 test set, but there are still classes of errors that they fail to correct. |
| Approach: | They found that state-of-the-art GEC systems outperform humans by a wide margin on the CoNLL-2014 test set . however, they found that there are still classes of errors that they fail to correct . |
| Outcome: | The F0.5 evaluation metric outperforms the CoNLL-2014 test set, but there are still classes of errors that they fail to correct. |
GPT-3.5 for Grammatical Error Correction (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent work shows that GPT-3.5 struggles with several error types, including punctuation mistakes, tense errors, syntactic dependencies between words, and lexical compatibility at the sentence level. |
| Approach: | They evaluate GPT-3.5 for grammatical error correction in multiple languages . they use it to re-rank correction hypotheses generated by other GEC models . |
| Outcome: | The proposed model performs well in English and Russian, but struggles with errors in other languages. |
An Extended Sequence Tagging Vocabulary for Grammatical Error Correction (2023.findings-eacl)
Copied to clipboard
| Challenge: | Current sequence-to-sequence and sequence-tagging approaches treat GEC as a machine-translation problem. |
| Approach: | They propose to introduce specialised tags for spelling correction and morphological inflection using the SymSpell and LemmInflect algorithms. |
| Outcome: | The proposed approach outperforms existing methods on the BEA benchmark. |
IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing reference-free automatic grammatical error correction methods do not correlate with human evaluation. |
| Approach: | They propose a reference-free automatic grammatical error correction evaluation method with enhanced gramma-ed capabilities. |
| Outcome: | The proposed method achieves highest correlation with human evaluations on a meta-evaluation dataset. |
Prompting open-source and commercial language models for grammatical error correction of English learner text (2024.findings-acl)
Copied to clipboard
Christopher Davis, Andrew Caines, O Andersen, Shiva Taslimipoor, Helen Yannakoudakis, Zheng Yuan, Christopher Bryant, Marek Rei, Paula Buttery
| Challenge: | Recent advances in generative AI have enabled us to prompt large language models (LLMs) to produce texts which are fluent and grammatical. |
| Approach: | They evaluate model performance by measuring their performance on established benchmarks. |
| Outcome: | The proposed models outperform supervised English GEC models on fluency correction benchmarks and commercial LLMs on edit benchmarks. |
A Simple Recipe for Multilingual Grammatical Error Correction (2021.acl-short)
Copied to clipboard
| Challenge: | Modern approaches view the task of Grammatical Error Correction (GEC) as monolingual text-to-text rewriting and employ encoderdecoder neural architectures. |
| Approach: | They propose a language-agnostic method to generate a large number of synthetic examples and use large-scale multilingual language models to train state-of-the-art GEC models. |
| Outcome: | The proposed method surpasses state-of-the-art results on GEC benchmarks in English, Czech, German and Russian. |
Revisiting Grammatical Error Correction Evaluation and Beyond (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining-based (PT) evaluation metrics are not effective for training grammatical error correction systems. |
| Approach: | They propose a pretraining-based GEC evaluation metric which only uses PT-based metrics to score the corrected parts of the system. |
| Outcome: | The proposed evaluation metric outperforms existing methods on a CoNLL14 evaluation task. |