Papers by Maksim Podkorytov
Too Much in Common: Shifting of Embeddings in Transformer Language Models and its Implications (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing studies have shown that word embeddings do not occupy a narrow cone, but rather drift in common directions. |
| Approach: | They show that anisotropy can be restored using a simple transformation of word embeddings. |
| Outcome: | The proposed model can restore anisotropy using a simple transformation. |