Papers by Takuya Matsuzaki
Absolute Position Embedding Learns Sinusoid-like Waves for Attention Based on Relative Position (2023.emnlp-main)
Copied to clipboard
| Challenge: | Attention weight is a clue to interpret how a Transformer-based model makes an inference. |
| Approach: | They analyze the mechanism behind the concentration of attention on nearby tokens . they find that attention in some heads is largely determined by relative positions . |
| Outcome: | The attention weights of the self-attention in a Transformer-based model are analyzed . the model can learn relationships between tokens while allowing parallelization, they show . |