Papers by Andrey Kuznetsov
Your Transformer is Secretly Linear (2024.acl-long)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | a novel linear characteristic exclusive to transformer decoders is revealed: embedding transformations between sequential layers exhibit almost perfect linearity. |
| Approach: | They propose a cosine-similarity-based regularization to reduce layer linearity in transformer decoders. |
| Outcome: | The proposed method improves performance metrics on Tiny Stories and SuperGLUE but also decreases the linearity of the models. |
Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework (2024.emnlp-demo)
Copied to clipboard
Arkhipkin Vladimir, Viacheslav Vasilev, Andrei Filatov, Igor Pavlov, Julia Agafonova, Nikolai Gerasimenko, Anna Averchenkova, Evelina Mironova, Bukashkin Anton, Konstantin Kulikov, Andrey Kuznetsov, Denis Dimitrov
| Challenge: | Text-to-image (T2I) diffusion models are popular for image manipulation, but also for video generation. |
| Approach: | They propose a novel T2I diffusion model based on latent diffusion that extends the base model for various applications. |
| Outcome: | The proposed model achieves high quality and photorealism and is 3 times faster than the base model. |
The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models (2024.findings-eacl)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | Embeddings in transformers encode vast amounts of linguistic nuances and patterns. |
| Approach: | They investigate the anisotropy dynamics and intrinsic dimension of embeddings in transformers . they found that transformer decoders exhibit a bell-shaped anisotropie profile . |
| Outcome: | The investigated embeddings exhibit a bell-shaped curve with the highest anisotropy concentrations in the middle layers . the intrinsic dimension increases in the initial phases of training, indicating an expansion into higher-dimensional space. |
LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers (2025.findings-naacl)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, Andrey Kuznetsov
| Challenge: | Large Language Models (LLMs) encode and store contextual information, but internal mechanisms are opaque. |
| Approach: | They propose a toolkit that assesses token-level nonlinearity, evaluates contextual memory, visualizes intermediate layer contributions and measures intrinsic dimensionality of representations. |
| Outcome: | The proposed framework assesses token-level nonlinearity, evaluates contextual memory, visualizes intermediate layer contributions, and measures the intrinsic dimensionality of representations. |
SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space (2026.eacl-long)
Copied to clipboard
Viktoriia Zinkovich, Anton Antonov, Andrei Spiridonov, Denis Shepelev, Andrey Moskalenko, Daria Pugacheva, Elena Tutubalina, Andrey Kuznetsov, Vlad Shakhuro
| Challenge: | Existing work on semantically equivalent textual paraphrases has focused on perturbing image inputs. |
| Approach: | They propose a novel adversarial paraphrasing task that generates grammatically correct paraphrases that sighed the original query meaning while degrading segmentation performance. |
| Outcome: | The proposed task outperforms previous methods by up to 2x on ReasonSeg and LLMSeg-40k datasets. |
Kandinsky: An Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion (2023.emnlp-demo)
Copied to clipboard
Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, Ilya Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, Denis Dimitrov
| Challenge: | Experimental evaluations demonstrate FID score of 8.03 on the COCO-30K dataset, marking our model as the top open source performer in terms of measurable image generation quality. |
| Approach: | They propose a latent diffusion-based model that combines image prior and latent diffusive techniques to create a text-to-image architecture. |
| Outcome: | The proposed model achieves the highest FID score among open-source models . it is compared with the state-of-the-art models on the COCO-30K dataset . |
Fast and Accurate Fisher-Guided Quantization via Efficient Kronecker Factorization (2026.acl-long)
Copied to clipboard
| Challenge: | Quantization has shown strong results in preserving model quality under compression, but under aggressive bit-width reductions, even quantization may require additional information to prevent performance degradation. |
| Approach: | They propose a Kronecker-factored approximation that captures second-order curvature information, captured by the Hessian, to achieve a 10 speedup over prior approaches. |
| Outcome: | The proposed method significantly accelerates the most expensive component in second-order quantization – Hessian parameterization . it achieves up to a 10 speedup over prior approaches. |
Bring the Apple, Not the Sofa: Impact of Irrelevant Context in Embodied AI Commands on VLA Models (2026.eacl-srw)
Copied to clipboard
Andrey Moskalenko, Daria Pugacheva, Denis Shepelev, Andrey Kuznetsov, Vlad Shakhuro, Elena Tutubalina
| Challenge: | Embodied AI is undergoing rapid development, with robots increasingly exhibiting practical utility in everyday environments. |
| Approach: | They evaluate the robustness of vision language action models under linguistic perturbations . they categorize irrelevant contexts into two groups according to their length and proximity to robot commands . |
| Outcome: | The proposed model can exhibit relative robustness to random context, with a performance drop within 10%, the authors show . human paraphrases of instructions lead to a drop of nearly 20%, the study shows . |