Papers by Vivek Ramanujan
Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent (2021.emnlp-main)
Copied to clipboard
| Challenge: | Evidence is emerging that neural networks learn due to inductive bias in the training routine, typically a variant of gradient descent (GD). |
| Approach: | They propose to characterize GD as an inductive bias in transformer training . they document norm growth in transformer language models and show they are saturated . |
| Outcome: | Empirically, we document norm growth in the training of transformer language models . the results suggest saturation is a new characterization of an inductive bias implicit in GD . |