Papers by Aleksandr Mikhalev
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers (2024.emnlp-main)
Copied to clipboard
Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Aleksandr Mikhalev, Alexander Panchenko, Ivan Oseledets
| Challenge: | High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and overlook dense MLP blocks, which contain about half of the model parameters. |
| Approach: | They propose a selective PEFT method that performs well on MLP blocks by converting layer gradients into a sparse structure and reducing the number of updated parameters. |
| Outcome: | The proposed method outperforms LoRA and MeProp, robust state-of-the-art PEFT approaches. |
Run LoRA Run: Faster and Lighter LoRA Implementations (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing studies on low-rank adapter training use the default chain of operations while calculating the output. |
| Approach: | They propose a framework that allows for efficient LoRA implementations by introducing low-rank adapters to linear layers and selecting the best forward and backward graphs based on FLOPs and time estimations. |
| Outcome: | The proposed framework significantly improves the speed of neural network training and fine-tuning with low-rank adapters. |