Papers by Zachary Ankner
Dynamic Masking Rate Schedules for MLM Pretraining (2024.eacl-short)
Copied to clipboard
| Challenge: | Most transformers trained with the Masked Language Modeling objective use a fixed masking rate of 15%, but wettig et al. ( 2022) found that the standard 15% masking rates are suboptimal for a variety of model settings and recommended a higher rate. |
| Approach: | They propose to dynamically schedule the masking rate throughout training to improve GLUE accuracy by 0.46% and 0.25% in BERT-base and BERT large, respectively. |
| Outcome: | The proposed method improves GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT large compared to baselines. |