Papers with Nadam
Should I try multiple optimizers when fine-tuning a pre-trained Transformer for NLP tasks? Should I tune their hyperparameters? (2024.eacl-long)
Copied to clipboard
| Challenge: | NLP research has explored different neural model architectures and sizes, datasets, training objectives, and transfer-learning techniques. |
| Approach: | They propose to use a variant of Stochastic Gradient Descent (SGD) to select among numerous variants, often with minimal or no tuning of the optimizer’s hyperparameters. |
| Outcome: | Experiments with five GLUE datasets, two models and seven popular optimizers show that tuning just the learning rate is as good as tuning all the hyperparameters. |