Papers by Ahmad Rashid
Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition (2021.tacl-1)
Copied to clipboard
| Challenge: | Name Regularity Bias is a problem in NER models that use contextual information to predict the type of an ambiguous entity. |
| Approach: | They propose a model-agnostic training method that adds learnable adversarial noise to some entity mentions to improve their accuracy. |
| Outcome: | The proposed method outperforms feature-based models on name regularity bias . it adds learnable adversarial noise to some entity mentions, leading to gains . |
End-to-End Self-Debiasing Framework for Robust NLU Training (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing models incorporate dataset biases leading to strong performance on in-distribution test sets but poor performance on out-of-distortion (OOD) tests. |
| Approach: | They propose a debiasing framework where the shallow representations of the main model are used to derive a bias model and both models are trained simultaneously. |
| Outcome: | The proposed framework outperforms existing approaches on three well-studied NLU tasks while still delivering high in-distribution performance. |
RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) is used to compress the pre-training and task-specific fine-tuning phases of large neural language models. |
| Approach: | They propose a sample-wise loss weighting method that re-weights the two losses for each sample. |
| Outcome: | The proposed method outperforms existing methods on 7 datasets of the GLUE benchmark. |
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that the flatness of the local minimum correlates well with better generalization. |
| Approach: | They propose to use a method encouraging convergence to a flatter minimum to fine-tune PLMs. |
| Outcome: | The proposed method outperforms state-of-the-art methods on NLP tasks without extra computation cost. |
Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Word-embeddings are vital components of natural language processing (NLP) but they consume a lot of memory which poses a challenge for edge deployment. |
| Approach: | They propose an embedding compression method based on matrix decomposition and knowledge distillation that initializes weights of pre-trained word-embeddings and fine-tunes end-to-end. |
| Outcome: | The proposed method has higher BLEU score on translation and lower perplexity on language modeling compared to complex, difficult to tune methods. |
How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) is a model compression algorithm that helps transfer knowledge in a large neural network into a smaller one. |
| Approach: | They propose a framework to assess adversarial robustness of multiple KD algorithms. |
| Outcome: | The proposed algorithm achieves state-of-the-art on the GLUE benchmark and out-of domain generalization and adversarial robustness compared to competitive methods. |
Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing (2022.emnlp-main)
Copied to clipboard
Abbas Ghaddar, Yimeng Wu, Sunyam Bagga, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Xinyu Duan, Zhefeng Wang, Baoxing Huai, Xin Jiang, Qun Liu, Phillippe Langlais
| Challenge: | Existing pre-trained language models are not well-explored and are not reproducible in the literature. |
| Approach: | They propose to improve existing Arabic language pre-trained language models using a more methodical approach. |
| Outcome: | The proposed models outperform existing models on ALUE, a leaderboard-powered benchmark for Arabic NLU and NLG tasks. |
Latent Code and Text-based Generative Adversarial Networks for Soft-text Generation (N19-1)
Copied to clipboard
| Challenge: | Text generation with generative adversarial networks (GANs) can be divided into text-based and code-based categories depending on the type of signals used for discrimination. |
| Approach: | They propose a text-based approach to exploit generative adversarial networks (GANs) by using autoencoders to provide a continuous representation of sentences, which they will refer to as soft-text, and hybrid latent code and text-oriented approaches with one or more discriminators. |
| Outcome: | The proposed approach outperforms the traditional GAN-based methods on two well-known datasets. |
LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for regularizing deep neural networks rely on weight decay, dropout, batch/layer normalization to converge faster and generalize. |
| Approach: | They propose a framework for training with label regularization which includes conventional LS but can also model instance-specific variants. |
| Outcome: | The proposed approach consistently yields better results than conventional regularization on seven machine translation and three image classification tasks while maintaining training efficiency. |
Efficient Citer: Tuning Large Language Models for Enhanced Answer Quality and Verification (2024.findings-naacl)
Copied to clipboard
Marzieh Tahaei, Aref Jafari, Ahmad Rashid, David Alfonso-Hermelo, Khalil Bibi, Yimeng Wu, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
| Challenge: | Existing models with explicit citations lack the ability to verify information generated by these models. |
| Approach: | They construct a citation training dataset and fine-tune two models to address the challenge of explicit citations efficiently. |
| Outcome: | The proposed models surpass ChatGPT and exhibit exceptional out-of-domain generalization in both human and automatic evaluation. |
Towards Zero-Shot Knowledge Distillation for Natural Language Processing (2021.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions. |
| Approach: | They propose to use teacher training data for model compression . they investigate six tasks and find they can achieve between 75% and 92% of the teacher’s classification score while compressing the model 30 times. |
| Outcome: | The proposed solution achieves between 75% and 92% of the teacher’s classification score while compressing the model 30 times. |
Kronecker Decomposition for GPT Compression (2022.acl-short)
Copied to clipboard
| Challenge: | GPT is an auto-regressive Transformer-based pre-trained language model . but its huge size can be prohibitive for deploying on low capacity devices . |
| Approach: | They use a Kronecker decomposition technique to compress GPT models . they use ILKD to refine the model on downstream tasks . |
| Outcome: | The proposed model outperforms the existing DistilGPT2 model on language modeling and general language understanding evaluation benchmark tasks. |
Attribute Controlled Dialogue Prompting (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt-tuning is an increasingly popular parameter-efficient method for adapting large pretrained language models to downstream tasks. |
| Approach: | They propose an instance-specific prompt-tuning algorithm for dialog generation that generates prompts based on instance-level control code rather than the conversation history. |
| Outcome: | The proposed prompt-tuning module is a fraction of the size of the pretrained language model and saves memory and expensive storage space. |
MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that the trillion parameter mark for pre-trained language models is not achievable without a change in training paradigm. |
| Approach: | They propose a text-based adversarial training algorithm which enhances the performance of knowledge distillation by maximizing the divergence between teacher and student logits. |
| Outcome: | The proposed algorithm outperforms competing adversarial learning and data augmentation baselines on the GLUE benchmark. |