Papers by Michael Guerzhoy
Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings (2025.coling-main)
Copied to clipboard
| Challenge: | Recent results suggest that positional encodings are not necessary when training decoder-only Transformer language models. |
| Approach: | They propose a causal attention mechanism that allows Transformers to store positional information without positional encodings. |
| Outcome: | The proposed model can reconstruct the positions of tokens without positional encodings. |