Papers by Marzieh Tahaei
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning (2024.emnlp-industry)
Copied to clipboard
Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Kwon, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
| Challenge: | Existing methods to fine tune large language models require huge memory, limiting the choice to acquire Larger models. |
| Approach: | They propose an efficient quantization approach for dynamic low-rank adaptation that can efficiently fine tune large language models on a set of pre-defined LoRA ranks. |
| Outcome: | The proposed method outperforms QLoRA and is competitive to QLouRA and outperformed when employing its optimal rank. |
Efficient Citer: Tuning Large Language Models for Enhanced Answer Quality and Verification (2024.findings-naacl)
Copied to clipboard
Marzieh Tahaei, Aref Jafari, Ahmad Rashid, David Alfonso-Hermelo, Khalil Bibi, Yimeng Wu, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
| Challenge: | Existing models with explicit citations lack the ability to verify information generated by these models. |
| Approach: | They construct a citation training dataset and fine-tune two models to address the challenge of explicit citations efficiently. |
| Outcome: | The proposed models surpass ChatGPT and exhibit exceptional out-of-domain generalization in both human and automatic evaluation. |
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large language models excel at understanding and generating human-like text, but their widespread deployment can be prohibitively expensive. |
| Approach: | They propose a method that makes large language models dynamic without Pre-Training . they use modularity in networks and sort sub-models based on computation/accuracy in a nested manner. |
| Outcome: | The proposed method can make large language models dynamic without pre-training and replace standard fine-tuning with sorted fine- tuning. |
Kronecker Decomposition for GPT Compression (2022.acl-short)
Copied to clipboard
| Challenge: | GPT is an auto-regressive Transformer-based pre-trained language model . but its huge size can be prohibitive for deploying on low capacity devices . |
| Approach: | They use a Kronecker decomposition technique to compress GPT models . they use ILKD to refine the model on downstream tasks . |
| Outcome: | The proposed model outperforms the existing DistilGPT2 model on language modeling and general language understanding evaluation benchmark tasks. |
KroneckerBERT: Significant Compression of Pre-trained Language Models Through Kronecker Decomposition and Knowledge Distillation (2022.naacl-main)
Copied to clipboard
| Challenge: | a recent study shows that over-parameterized pre-trained language models are unsuitable for low-capacity devices. |
| Approach: | They propose a transformer-based pre-trained language model that is overparameterized . they use a two-stage knowledge distillation scheme to train the model . |
| Outcome: | The proposed model outperforms state-of-the-art models on well-known NLP benchmarks. |