Papers by Shaked Brody
On the Expressivity Role of LayerNorm in Transformers’ Attention (2023.findings-acl)
Copied to clipboard
| Challenge: | Layer Normalization (LayerNorm) is an inherent component in all Transformer-based models. |
| Approach: | They propose to use LayerNorm to normalize the activations during the forward pass and their gradients during the backward pass. |
| Outcome: | The proposed model is able to express the multi-head attention layer that follows it in a d-1 space and scales to the same norm of d. |