Papers by Inigo Urteaga
Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking (2023.findings-acl)
Copied to clipboard
| Challenge: | Using a Bayesian optimization framework, we pre-train Transformer-based language models (TLMs) using a multi-armed bandit framework requires high computational resources and introduces many unresolved design choices. |
| Approach: | They propose a Bayesian optimization framework for resource efficient pre-training of Transformer-based language models. |
| Outcome: | The proposed framework achieves lower MLM loss in fewer epochs, across settings, while avoiding expensive hyperparameter grid search. |