Papers by Yaroslav Aksenov
Linear Transformers with Learnable Kernel Functions are Better In-Context Models (2024.acl-long)
Copied to clipboard
Yaroslav Aksenov, Nikita Balagansky, Sofia Lo Cicero Vaina, Boris Shaposhnikov, Alexey Gorbatovski, Daniil Gavrilov
| Challenge: | Current Language Models (LMs) lack essential In-Context Learning capabilities, a domain where the Transformer excels. |
| Approach: | They propose a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions, augmented by convolutional networks. |
| Outcome: | The proposed model amplifies its In-Context Learning abilities on the Pile dataset. |
Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy (2025.emnlp-main)
Copied to clipboard
Nikita Balagansky, Yaroslav Aksenov, Daniil Laptev, Vadim Kurochkin, Gleb Gerasimov, Nikita Koriagin, Daniil Gavrilov
| Challenge: | Sparse Autoencoders (SAEs) are powerful tools for interpreting neural networks . conventional SAEs are constrained by the fixed sparsity level chosen during training . |
| Approach: | They propose a training objective that trains a single SAE to optimise reconstructions across multiple sparsity levels simultaneously. |
| Outcome: | The proposed objective achieves Pareto-optimal trade-offs between sparsity and explained variance, outperforming traditional SAEs trained at individual sparsities. |