Papers by Karahan Sarıtaş
A Reproduction Study: The Kernel PCA Interpretation of Self-Attention Fails Under Scrutiny (2025.acl-srw)
Copied to clipboard
| Challenge: | Recent studies suggest that self-attention implements kernel principal component analysis (KPCA) Across 10 transformer architectures, we conclude that the KPCA interpretation of self- attention lacks empirical support. |
| Approach: | They revisit claims that self-attention implements kernel principal component analysis . they argue that self attention projects queries onto principal component axes of key matrix K . |
| Outcome: | The proposed kernel principal component analysis does not match the proposed kernel . the proposed method is not able to detect the eigenvalues of the gram matrix . |