Papers by Valeria Ruscio
Beyond Position: the emergence of wavelet-like properties in Transformers (2025.acl-long)
Copied to clipboard
| Challenge: | Despite its widespread adoption, theoretical limitations in positional encodings are resolved by developing emergent, wavelet-like processing strategies. |
| Approach: | They propose to use Rotary Position Embeddings to develop emergent, wavelet-like properties that compensate for the positional encoding’s theoretical limitations. |
| Outcome: | The attention heads evolve to implement multi-resolution processing analogous to wavelet transforms. |