Papers by Hossein Rajabzadeh
FLOP-Efficient Training: Early Stopping Based on Test-Time Compute Awareness (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior work shows that increasing test-time compute (TTC) can improve accuracy of large language models. |
| Approach: | They propose a TTC-aware training algorithm that jointly selects a checkpoint and a corresponding TTC configuration to minimize training compute without sacrificing accuracy. |
| Outcome: | The proposed method reduces training compute by 92% while maintaining accuracy. |
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning (2024.emnlp-industry)
Copied to clipboard
Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Kwon, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
| Challenge: | Existing methods to fine tune large language models require huge memory, limiting the choice to acquire Larger models. |
| Approach: | They propose an efficient quantization approach for dynamic low-rank adaptation that can efficiently fine tune large language models on a set of pre-defined LoRA ranks. |
| Outcome: | The proposed method outperforms QLoRA and is competitive to QLouRA and outperformed when employing its optimal rank. |
ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training (2025.emnlp-industry)
Copied to clipboard
Maryam Dialameh, Rezaul Karim, Hossein Rajabzadeh, Omar Mohamed Awad, Boxing Chen, Hyock Ju Kwon, Walid Ahmed, Yang Liu
| Challenge: | ECHO-LLaMA transforms LLa MA models into shared KV caching across certain layers, significantly reducing KV computational complexity while maintaining or improving language performance. |
| Approach: | They propose an efficient LLaMA architecture that transforms LLama models into shared KV caching across certain layers, reducing computational complexity while maintaining or improving language performance. |
| Outcome: | ECHO-LLaMA achieves up to 77% higher token-per-second throughput during training, up to 16% higher Model FLOPs Utilization (MFU) and up to 14% lower loss when trained on an equal number of tokens. |
Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models (2025.emnlp-main)
Copied to clipboard
Benyamin Jamialahmadi, Parsa Kavehzadeh, Mehdi Rezagholizadeh, Parsa Farinneya, Hossein Rajabzadeh, Aref Jafari, Boxing Chen, Marzieh S. Tahaei
| Challenge: | Existing methods for dynamic inference are limited by hardware inefficiencies or performance degradation. |
| Approach: | They propose a framework for depth-based dynamic inference that freezes the pre-trained model and inserts additional transformer layers at selected exit points. |
| Outcome: | The proposed framework outperforms state-of-the-art methods such as Flextron and Layerskip on multiple models at various scales, as well as other leading compression techniques across a variety of benchmarks. |