Papers by Megi Dervishi
Training compute-optimal transformer encoder models (2025.emnlp-main)
Copied to clipboard
| Challenge: | OptiBERT is a family of compute-optimal BERT-style models that matches or surpasses leading baselines while training with dramatically less FLOPS. |
| Approach: | They propose to train OptiBERT models with a Masked Language Model objective . they train a family of compute-optimal BERT-style models that matches or surpasses leading baselines . |
| Outcome: | The proposed model matches or surpasses leading baselines on GLUE and MTEB while training with dramatically less FLOPS. |