Papers with APEs
The Curious Case of Absolute Position Embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | In natural language, it is not absolute position that matters, but relative position . et al., 2017) language models incorporate positional encodings that encode absolute (linear) word order. |
| Approach: | They find that Transformer language models encode word order using positional information . they also find that models that use absolute position embeddings over-rely on positional data . |
| Outcome: | The results raise questions about the efficacy of APEs to model the relativity of position information. |