Papers by Anemily Machina
Anisotropy is Not Inherent to Transformers (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work has shown that Transformer embeddings are anisotropic, which is called the representation degradation problem. |
| Approach: | They identify a set of Transformer models with isotropic embedding spaces, the large Pythia models. |
| Outcome: | The proposed model sets show that isotropic models do not develop as previously theorized. |