Papers by Ali Ghodsi
Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMax (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing kNN-based augmentation techniques blindly incorporate all samples, but MiniMax-kNN uses a subset of augmented samples to maximize KL-divergence between teacher and student models. |
| Approach: | They propose a semi-supervised approach to augmented data augmentation using kNN. |
| Outcome: | The proposed method outperforms existing kNN-based augmentation techniques on several classification tasks and requires fewer augmented examples and less computation to achieve superior performance. |
RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) is used to compress the pre-training and task-specific fine-tuning phases of large neural language models. |
| Approach: | They propose a sample-wise loss weighting method that re-weights the two losses for each sample. |
| Outcome: | The proposed method outperforms existing methods on 7 datasets of the GLUE benchmark. |
When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing DA methods naively add a certain number of augmented samples without considering the quality and the added computational cost of these samples. |
| Approach: | They propose a data-augmented DA technique that generates or reweights augmented samples . they say it is faster to train and can be plugged into any DA method . |
| Outcome: | The proposed technique is faster to train and more efficient than existing methods. |
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that the flatness of the local minimum correlates well with better generalization. |
| Approach: | They propose to use a method encouraging convergence to a flatter minimum to fine-tune PLMs. |
| Outcome: | The proposed method outperforms state-of-the-art methods on NLP tasks without extra computation cost. |
How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) is a model compression algorithm that helps transfer knowledge in a large neural network into a smaller one. |
| Approach: | They propose a framework to assess adversarial robustness of multiple KD algorithms. |
| Outcome: | The proposed algorithm achieves state-of-the-art on the GLUE benchmark and out-of domain generalization and adversarial robustness compared to competitive methods. |
Annealing Knowledge Distillation (2021.eacl-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a powerful model compression technique for deep neural networks. |
| Approach: | They propose a method to feed the rich information provided by teacher’s soft-targets incrementally and more efficiently by annealing the teacher output incrementally. |
| Outcome: | The proposed method can be used on image classification and NLP language inference tasks with BERT-based models on the GLUE benchmark. |
Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for intermediate layer matching are limited due to huge over-parameterization . |
| Approach: | They propose to match intermediate layers of teacher and student in output space via attention-based layer projection. |
| Outcome: | The proposed method outperforms existing methods on GLUE tasks. |
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning (2024.emnlp-industry)
Copied to clipboard
Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Kwon, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
| Challenge: | Existing methods to fine tune large language models require huge memory, limiting the choice to acquire Larger models. |
| Approach: | They propose an efficient quantization approach for dynamic low-rank adaptation that can efficiently fine tune large language models on a set of pre-defined LoRA ranks. |
| Outcome: | The proposed method outperforms QLoRA and is competitive to QLouRA and outperformed when employing its optimal rank. |
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation (KD) do not mitigate the noise in the teacher’s output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. |
| Approach: | They propose a method that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. |
| Outcome: | The proposed method achieves state-of-the-art performance on NLU and computer vision tasks. |
Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher (2022.coling-1)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a powerful tool for deep learning applications. |
| Approach: | They propose a method which defines a smoother training path for the student by following the training footprints of the teacher rather than solely relying on distilling from a single mature fully-trained teacher. |
| Outcome: | The proposed technique is quite effective in mitigating the capacity-gap problem and the checkpoint search problem. |
Efficient Citer: Tuning Large Language Models for Enhanced Answer Quality and Verification (2024.findings-naacl)
Copied to clipboard
Marzieh Tahaei, Aref Jafari, Ahmad Rashid, David Alfonso-Hermelo, Khalil Bibi, Yimeng Wu, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
| Challenge: | Existing models with explicit citations lack the ability to verify information generated by these models. |
| Approach: | They construct a citation training dataset and fine-tune two models to address the challenge of explicit citations efficiently. |
| Outcome: | The proposed models surpass ChatGPT and exhibit exceptional out-of-domain generalization in both human and automatic evaluation. |
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large language models excel at understanding and generating human-like text, but their widespread deployment can be prohibitively expensive. |
| Approach: | They propose a method that makes large language models dynamic without Pre-Training . they use modularity in networks and sort sub-models based on computation/accuracy in a nested manner. |
| Outcome: | The proposed method can make large language models dynamic without pre-training and replace standard fine-tuning with sorted fine- tuning. |
Do we need Label Regularization to Fine-tune Pre-trained Language Models? (2023.eacl-main)
Copied to clipboard
Ivan Kobyzev, Aref Jafari, Mehdi Rezagholizadeh, Tianda Li, Alan Do-Omri, Peng Lu, Pascal Poupart, Ali Ghodsi
| Challenge: | Knowledge Distillation (KD) is a label regularization technique that can be replaced with lighter teacher-free variants such as the label-smoothing technique. |
| Approach: | They propose to use knowledge distillation to train student models by deploying the teacher network during training. |
| Outcome: | The proposed method can be replaced with lighter teacher-free variants on PLMs with more than 600 distinct trials and ran each configuration five times. |
DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2023.eacl-main)
Copied to clipboard
| Challenge: | Pre-training/fine-tuning of pre-training models has become more expensive and resource-hungry. |
| Approach: | They propose a low-rank adaptation technique that trains LoRA blocks for a range of ranks instead of a single rank. |
| Outcome: | The proposed method trains LoRA blocks for a range of ranks instead of a single rank . it can train dynamic search-free models with DyLoRA at least 4 to 7 times faster than LoRA . |
KroneckerBERT: Significant Compression of Pre-trained Language Models Through Kronecker Decomposition and Knowledge Distillation (2022.naacl-main)
Copied to clipboard
| Challenge: | a recent study shows that over-parameterized pre-trained language models are unsuitable for low-capacity devices. |
| Approach: | They propose a transformer-based pre-trained language model that is overparameterized . they use a two-stage knowledge distillation scheme to train the model . |
| Outcome: | The proposed model outperforms state-of-the-art models on well-known NLP benchmarks. |