Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for incorporating a masked language model into an EncDec model have potential drawbacks when applied to GEC. |
| Approach: | They propose to incorporate a pre-trained masked language model (MLM) into an encoder-decoder model for grammatical error correction. |
| Outcome: | The proposed method achieves state-of-the-art on BEA-2019 and CoNLL-2014 benchmarks. |
Similar Papers
Chinese Grammatical Correction Using BERT-based Pre-trained Model (2020.aacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that pre-trained models improve performance on downstream tasks. |
| Approach: | They propose to incorporate a pre-trained model into an encoder-decoder model to improve the performance of Chinese grammatical error correction tasks. |
| Outcome: | The proposed method improves the performance of Chinese grammatical error correction tasks. |
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that pretrained Masked Language Models are not effective as universal lexical and sentence encoders off-the-shelf, i.e., without further task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks using annotated task data. |
| Approach: | They propose a contrastive learning technique which turns pretrained MLMs into effective universal lexical and sentence encoders without additional data. |
| Outcome: | The proposed technique can turn MLMs into effective universal lexical and sentence encoders even without additional data. |
Revisiting Pre-Trained Models for Chinese Natural Language Processing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained language models have shown tremendous improvements across various NLP tasks. |
| Approach: | They propose to revisit Chinese pre-trained language models to examine their effectiveness in a non-English language and release the Chinese pretrained model series to the community. |
| Outcome: | The proposed model improves on RoBERTa in several ways, especially the masking strategy that adopts MLM as correction (Mac). |
Stronger Baselines for Grammatical Error Correction Using a Pretrained Encoder-Decoder Model (2020.aacl-main)
Copied to clipboard
| Challenge: | Strong pretraining approaches for grammatical error correction require extensive use of a pseudo-parallel corpus. |
| Approach: | They propose to use bidirectional and auto-regressive transformers as a generic pretrained encoder-decoder model for grammatical error correction (GEC) they find that monolingual and multilingual BART models achieve high performance in GEC, with one of the results being comparable to the current strong results in English GEC. |
| Outcome: | The proposed model achieves comparable results to the current strong results in English GEC. |
Spelling Error Correction with Soft-Masked BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Experimental results show that the proposed method is significantly better than the baselines including the one solely based on BERT. |
| Approach: | They propose a neural architecture which uses a network for error detection and a system for error correction based on BERT, with the latter connected to the other using what they call soft-masking technique. |
| Outcome: | The proposed method performs better than baselines including the one solely based on BERT, and is general and may be employed in other language detection-correction problems. |
ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to model coarse-grained linguistic information do not integrate coarse-gram information into pre-training. |
| Approach: | They propose an explicitly n-gram masking method to enhance integration of coarse-grained linguistic information into pre-training. |
| Outcome: | The proposed method outperforms existing models on English and Chinese text corpora and fine-tunes on 19 downstream tasks. |
Rethinking Masked Language Modeling for Chinese Spelling Correction (2023.acl-long)
Copied to clipboard
| Challenge: | Existing CSC models over-fit the error model while under-fitting the language model, resulting in poor generalization to out-of-distribution error patterns. |
| Approach: | They propose to use a multi-domain benchmark LEMON to assess the open domain generalization of Chinese Spelling Correction models. |
| Outcome: | The proposed method achieves state-of-the-art results on SIGHAN, ECSpell, and LEMON. |
Masked Language Model Scoring (2020.acl-main)
Copied to clipboard
| Challenge: | Pretrained masked language models require finetuning for most tasks. |
| Approach: | They evaluate pretrained masked language models out of the box via their pseudo-log-likelihood scores (PLLs) they attribute this success to PLL’s unsupervised expression of linguistic acceptability without a left-to-right bias, greatly improving on scores from GPT-2 . |
| Outcome: | The proposed model outperforms autoregressive language models in a variety of tasks. |
Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models excel in natural language understanding (NLU) tasks. |
| Approach: | They propose to apply layer-dependent removal of the causal mask (CM) during LLM fine-tuning to improve SL performance. |
| Outcome: | The proposed approach outperforms state-of-the-art SL models on IE tasks, while achieving state- of-the art results is unclear. |
Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations. |
| Approach: | They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective. |
| Outcome: | Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture. |