Papers by Tobias Stollenwerk
Better Embeddings with Coupled Adam (2025.acl-long)
Copied to clipboard
| Challenge: | Anisotropic Embeddings Large Language Models exhibit undesirable yet poorly understood feature of anisotropy. |
| Approach: | They propose an algorithm that uses the second moment in Adam to mitigate anisotropic embeddings . they propose an embeddable matrix and unembedding matrix to map the input and output tokens based on weight tying . |
| Outcome: | The proposed model improves quality and performance on large datasets. |