Papers by Çağatay Yıldız
A Reproduction Study: The Kernel PCA Interpretation of Self-Attention Fails Under Scrutiny (2025.acl-srw)
Copied to clipboard
| Challenge: | Recent studies suggest that self-attention implements kernel principal component analysis (KPCA) Across 10 transformer architectures, we conclude that the KPCA interpretation of self- attention lacks empirical support. |
| Approach: | They revisit claims that self-attention implements kernel principal component analysis . they argue that self attention projects queries onto principal component axes of key matrix K . |
| Outcome: | The proposed kernel principal component analysis does not match the proposed kernel . the proposed method is not able to detect the eigenvalues of the gram matrix . |
Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve? (2024.emnlp-main)
Copied to clipboard
| Challenge: | In the last decade, the generalization and adaptation abilities of deep learning models were evaluated on fixed training and test distributions. |
| Approach: | They propose to train large language models on unlabeled text corpora and train them online. |
| Outcome: | The proposed model training on a text domain could degrade its perplexity on the test portion of the same domain. |