Papers by Shreyas Subramanian
Hop, skip, jump to Convergence: Dynamics of Learning Rate Transitions for Improved Training of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Modern deep neural networks have achieved stateof-the-art performance across a wide range of machine learning tasks. |
| Approach: | They propose to switch the learning rate at a predetermined time during training to improve the performance of large language models. |
| Outcome: | The proposed model shows that switching the learning rate causes the loss curves to contract towards each other. |