Papers by Antonio Lopardo
Weight Tying Biases Token Embeddings Towards the Output Space (2026.findings-acl)
Copied to clipboard
| Challenge: | Weight tying is a common practice in language model design, but its impact on learning embedding space remains unclear. |
| Approach: | They show that weight tying optimizes the embedding matrix for output prediction . they also show that tied embeddable matrices align more closely with output embedders . |
| Outcome: | The proposed weight tying approach harms performance at scale and has implications for training smaller LLMs. |