Papers by Ivan Oseledets
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers (2024.emnlp-main)
Copied to clipboard
Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Aleksandr Mikhalev, Alexander Panchenko, Ivan Oseledets
| Challenge: | High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and overlook dense MLP blocks, which contain about half of the model parameters. |
| Approach: | They propose a selective PEFT method that performs well on MLP blocks by converting layer gradients into a sparse structure and reducing the number of updated parameters. |
| Outcome: | The proposed method outperforms LoRA and MeProp, robust state-of-the-art PEFT approaches. |
Layerwise universal adversarial attack on NLP models (2023.findings-acl)
Copied to clipboard
| Challenge: | a new technique for layerwise UATs searches hidden layers of a network for universal adversarial triggers . a previous study showed that adversarials can fool models by perturbing samples that leave the ground truth label unchanged but can modify model prediction drastically. |
| Approach: | They propose a new approach to construct layerwise UATs by perturbing hidden layers of a network and propose LUATs that are more efficient than vanilla UAT methods. |
| Outcome: | The proposed method provides better transferability in a model-to-model setting with an average gain of 9.3% in fooling rate over baseline. |
One Task Vector is not Enough: A Large-Scale Study for In-Context Learning (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing studies limit comprehensive analysis of large language models based on task vectors . recent work points to "task vectors" as mechanism for encoding task rules . |
| Approach: | They propose a novel task vector with 30 input-output pairs for in-context learning . they use a few prompt-based examples to adapt to new tasks without weight updates . |
| Outcome: | Experiments with Llama-3-8B on QAF show task vector performance peaks at intermediate layer . complex tasks rely on multiple, subtask-specific vectors rather than a single vector . |
Your Transformer is Secretly Linear (2024.acl-long)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | a novel linear characteristic exclusive to transformer decoders is revealed: embedding transformations between sequential layers exhibit almost perfect linearity. |
| Approach: | They propose a cosine-similarity-based regularization to reduce layer linearity in transformer decoders. |
| Outcome: | The proposed method improves performance metrics on Tiny Stories and SuperGLUE but also decreases the linearity of the models. |
Confidence Leaps in LLM Reasoning: Early Stopping and Cross-Model Transfer (2026.eacl-short)
Copied to clipboard
| Challenge: | Large Language Models build confidence gradually during reasoning, but internal dynamics of how confidence evolves during this reasoning process remain poorly understood. |
| Approach: | They propose a model-agnostic early-stopping heuristic that halts generation upon detecting a "confidence leap" they argue that conviction is often reached in a discrete "moment of insight" they propose to train models without sacrificing accuracy . |
| Outcome: | The proposed model-agnostic heuristic reduces generation time without sacrificing accuracy and significantly reduces the generation time. |
CLEAR: Character Unlearning in Textual and Visual Modalities (2025.findings-acl)
Copied to clipboard
Alexey Dontsov, Dmitrii Korzh, Alexey Zhavoronkin, Boris Mikheev, Denis Bobkov, Aibek Alanov, Oleg Rogov, Ivan Oseledets, Elena Tutubalina
| Challenge: | Existing methods for removing private or hazardous data from deep learning models are focused on single-modality models. |
| Approach: | They propose CLEAR, the first open-source benchmark specifically for MMU. CLEAR contains 200 fictitious individuals and 3,700 images linked with corresponding question-answer pairs. |
| Outcome: | The proposed benchmarks show that unlearning both modalities outperform single-modality approaches. |
The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models (2024.findings-eacl)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | Embeddings in transformers encode vast amounts of linguistic nuances and patterns. |
| Approach: | They investigate the anisotropy dynamics and intrinsic dimension of embeddings in transformers . they found that transformer decoders exhibit a bell-shaped anisotropie profile . |
| Outcome: | The investigated embeddings exhibit a bell-shaped curve with the highest anisotropy concentrations in the middle layers . the intrinsic dimension increases in the initial phases of training, indicating an expansion into higher-dimensional space. |
Exploring the Hidden Capacity of LLMs for One-Step Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can reconstruct surprisingly long texts via autoregressive generation from just one trained input embedding. |
| Approach: | They show that large language models can reconstruct surprisingly long texts via autoregressive generation from just one trained input embedding. |
| Outcome: | The proposed model can generate hundreds of accurate tokens in one token-parallel forward pass, when provided with only two learned embeddings. |
Tensorized Embedding Layers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Using the Tensor Train decomposition, embeddings layers occupy large portion of model weights, preventing their deployment in limited resource settings. |
| Approach: | They propose a method for parameterizing embedding layers based on the Tensor Train decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. |
| Outcome: | The proposed method can be plugged into any model and trained end-to-end. |
LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers (2025.findings-naacl)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, Andrey Kuznetsov
| Challenge: | Large Language Models (LLMs) encode and store contextual information, but internal mechanisms are opaque. |
| Approach: | They propose a toolkit that assesses token-level nonlinearity, evaluates contextual memory, visualizes intermediate layer contributions and measures intrinsic dimensionality of representations. |
| Outcome: | The proposed framework assesses token-level nonlinearity, evaluates contextual memory, visualizes intermediate layer contributions, and measures the intrinsic dimensionality of representations. |
Feature Drift: How Fine-Tuning Repurposes Representations in LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Sparse autoencoders (SAEs) are a powerful tool for interpreting neural networks by extracting concepts (features) represented in their activations. |
| Approach: | They propose to use Sparse Autoencoders to extract concepts from their activations to explain how fine-tuning changes model capabilities. |
| Outcome: | The proposed model recombines existing concepts rather than learning new ones, and shows that it is a better explanation for how fine-tuning changes model capabilities. |
CLARITY: Clinical Assistant for Routing, Inference, and Triage (2025.emnlp-industry)
Copied to clipboard
Vladimir Shaposhnikov, Alexandr Nesterov, Ilia Kopanichuk, Ivan Bakulin, Zhelvakov Egor, Ruslan Abramov, Tsapieva Ekaterina Olegovna, Iaroslav Radionovich Bespalov, Dmitry V. Dylov, Ivan Oseledets
| Challenge: | Medical dialogue systems are still flawed for real-world adoption in healthcare. |
| Approach: | They propose to integrate CLARITY (Clinical Assistant for Routing, Inference and Triage) it combines a Finite State Machine (FSM) and collaborative agents that employ Large Language Model (LLM) they report that it surpasses human-level performance in terms of first-attempt routing precision . |
| Outcome: | The proposed platform surpasses human-level performance in terms of first-attempt routing precision. |
Run LoRA Run: Faster and Lighter LoRA Implementations (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing studies on low-rank adapter training use the default chain of operations while calculating the output. |
| Approach: | They propose a framework that allows for efficient LoRA implementations by introducing low-rank adapters to linear layers and selecting the best forward and backward graphs based on FLOPs and time estimations. |
| Outcome: | The proposed framework significantly improves the speed of neural network training and fine-tuning with low-rank adapters. |