Papers by Peter Cholak
Overcoming a Theoretical Limitation of Self-Attention (2022.acl-long)
Copied to clipboard
| Challenge: | Hahn shows that for languages where acceptance depends on a single input symbol, a transformer’s classification decisions get closer and closer to random guessing as input strings get longer and longer. |
| Approach: | They propose a transformer that recognizes PARITY with perfect accuracy and a model that uses layer normalization to bring the cross-entropy of both models arbitrarily close to zero. |
| Outcome: | The proposed model can accept and reject strings with perfect accuracy and bring cross-entropy close to zero when they need to focus on a single position. |