Papers by Mehdi Rezagholizadeh

Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition (2021.tacl-1)

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated their potential across a wide spectrum of natural language processing tasks.
Approach:	They propose a novel approach to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions.
Outcome:	The proposed approach improves performance without additional online computational costs on train-short-test-long scenarios.

Copied to clipboard

Abbas Ghaddar, Philippe Langlais, Ahmad Rashid, Mehdi Rezagholizadeh

Named Entity Recognition

Challenge:	Name Regularity Bias is a problem in NER models that use contextual information to predict the type of an ambiguous entity.
Approach:	They propose a model-agnostic training method that adds learnable adversarial noise to some entity mentions to improve their accuracy.
Outcome:	The proposed method outperforms feature-based models on name regularity bias . it adds learnable adversarial noise to some entity mentions, leading to gains .

End-to-End Self-Debiasing Framework for Robust NLU Training (2021.findings-acl)

Copied to clipboard

Abbas Ghaddar, Phillippe Langlais, Mehdi Rezagholizadeh, Ahmad Rashid

Robustness in NLP

Challenge:	Existing models incorporate dataset biases leading to strong performance on in-distribution test sets but poor performance on out-of-distortion (OOD) tests.
Approach:	They propose a debiasing framework where the shallow representations of the main model are used to derive a bias model and both models are trained simultaneously.
Outcome:	The proposed framework outperforms existing approaches on three well-studied NLU tasks while still delivering high in-distribution performance.

RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation (2021.findings-emnlp)

Copied to clipboard

Peng Lu, Abbas Ghaddar, Ahmad Rashid, Mehdi Rezagholizadeh, Ali Ghodsi, Philippe Langlais

OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection (2024.findings-acl)

Challenge:	Knowledge Distillation (KD) is used to compress the pre-training and task-specific fine-tuning phases of large neural language models.
Approach:	They propose a sample-wise loss weighting method that re-weights the two losses for each sample.
Outcome:	The proposed method outperforms existing methods on 7 datasets of the GLUE benchmark.

Copied to clipboard

Chenyang Huang, Abbas Ghaddar, Ivan Kobyzev, Mehdi Rezagholizadeh, Osmar Zaiane, Boxing Chen

Machine Translation

Challenge:	Existing methods for detecting hallucinations and omissions in Machine Translation systems focus on analyzing the model’s internal states or relying on external tools.
Approach:	They propose an Optimal Transport-based word aligner specifically designed to enhance the detection of hallucinations and omissions in Machine Translation systems.
Outcome:	The proposed method is competitive with state-of-the-art methods across 18 language pairs on the HalOmi benchmark and shows promising features.

CHARP: Conversation History AwaReness Probing for Knowledge-grounded Dialogue Systems (2024.findings-acl)

Copied to clipboard

Abbas Ghaddar, David Alfonso-Hermelo, Philippe Langlais, Mehdi Rezagholizadeh, Boxing Chen, Prasanna Parthasarathi

Dialogue Systems & Conversational Agents

Challenge:	CHARP is a testbed for knowledge-grounded dialogue evaluation of models trained on FaithDial data.
Approach:	They propose a testbed for evaluating models trained on FaithDial with annotation artifacts that may bias models towards completely ignoring the conversation history.
Outcome:	The proposed model fails to accurately evaluate the conversational history and lacks hallucination detection.

When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation (2022.findings-acl)

Copied to clipboard

Ehsan Kamalloo, Mehdi Rezagholizadeh, Ali Ghodsi

Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination (2025.findings-acl)

Challenge:	Existing DA methods naively add a certain number of augmented samples without considering the quality and the added computational cost of these samples.
Approach:	They propose a data-augmented DA technique that generates or reweights augmented samples . they say it is faster to train and can be plugged into any DA method .
Outcome:	The proposed technique is faster to train and more efficient than existing methods.

Copied to clipboard

Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh, Boxing Chen, Sarath Chandar

Reasoning

Challenge:	Large language models (LLMs) have a tendency to hallucinate false or misleading information, limiting their reliability.
Approach:	They examine how architecture-based inductive biases affect the propensity to hallucinate . they find that the models are more reliable and more reliable than traditional models .
Outcome:	The proposed models can be used to train and train large language models that are factual or able to explain themselves through their knowledge.

Efficient Classification of Long Documents via State-Space Models (2023.emnlp-main)

Copied to clipboard

Peng Lu, Suyuchen Wang, Mehdi Rezagholizadeh, Bang Liu, Ivan Kobyzev

Information Retrieval

Challenge:	Existing transformer-based models can only process long documents with limited computational resources due to their quadratic computation time and space.
Approach:	They propose to use state-space models for long document classification tasks instead of using sparse or hierarchical structures to solve this problem.
Outcome:	The proposed model performs comparable to self-attention models while being 36% more efficient.

Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers (2020.emnlp-main)

Copied to clipboard

Yimeng Wu, Peyman Passban, Mehdi Rezagholizadeh, Qun Liu

Dynamic Position Encoding for Transformers (2022.coling-1)

Challenge:	Existing knowledge distillation techniques are not suitable for deep learning tasks due to memory constraints.
Approach:	They propose to combine knowledge from a large teacher network into a student network (S) they propose to use a combinatorial mechanism to inject layer-level supervision from T to S .
Outcome:	The proposed model outperforms existing models in PortugueseEnglish, TurkishEnglish and EnglishGerman directions and students trained using it have 50% fewer parameters and can deliver comparable results to 12-layer teachers.

Copied to clipboard

Joyce Zheng, Mehdi Rezagholizadeh, Peyman Passban

Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging (2022.findings-emnlp)

Challenge:	In neural machine translation, the general task of translating is to reduce the input sentence into smaller units (also known as statistical phrases), select an optimal translation for each unit, and place them in the correct order.
Approach:	They propose a novel architecture that relies on a feed-forward backbone and self-attention mechanism to encode sequential/positional information.
Outcome:	The proposed architecture improves on multiple datasets in French, Italian, and German and shows that it is more efficient than the current model.

Copied to clipboard

Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh, Ahmad Rashid, Ali Ghodsi, Phillippe Langlais

“Knowing When You Don’t Know”: A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation (2024.findings-emnlp)

Challenge:	Recent studies show that the flatness of the local minimum correlates well with better generalization.
Approach:	They propose to use a method encouraging convergence to a flatter minimum to fine-tune PLMs.
Outcome:	The proposed method outperforms state-of-the-art methods on NLP tasks without extra computation cost.

Copied to clipboard

Nandan Thakur, Luiz Bonifacio, Crystina Zhang, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Boxing Chen, Mehdi Rezagholizadeh, Jimmy Lin

Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models (2023.findings-emnlp)

Challenge:	Prior work on RAG grounds Large Language Models to reduce factual hallucinations lacks a comprehensive evaluation of different language families.
Approach:	They propose a human-annotated dataset for evaluating LLM robustness in RAG . they find that most models struggle to balance the two capacities .
Outcome:	The proposed dataset includes both a non-relevant and a relevant subset.

Copied to clipboard

Amirhossein Kazemnejad, Mehdi Rezagholizadeh, Prasanna Parthasarathi, Sarath Chandar

Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition (2020.findings-emnlp)

Challenge:	Recent research has demonstrated that pre-trained language models acquire a broad range of knowledge about linguistic structures, encyclopedic relations, levels of commonsense, and even coding and reasoning rules.
Approach:	They propose a systematic framework to measure parametric knowledge utilization in pre-trained language models by extracting parametric information from a PLM and constructing a downstream task around this extracted knowledge.
Outcome:	The proposed framework extracts parametric knowledge from a PLM and constructs a downstream task around this extracted knowledge.

Copied to clipboard

Vasileios Lioutas, Ahmad Rashid, Krtin Kumar, Md. Akmal Haidar, Mehdi Rezagholizadeh

Context-Aware Assistant Selection for Improved Inference Acceleration with Large Language Models (2024.emnlp-main)

Challenge:	Word-embeddings are vital components of natural language processing (NLP) but they consume a lot of memory which poses a challenge for edge deployment.
Approach:	They propose an embedding compression method based on matrix decomposition and knowledge distillation that initializes weights of pre-trained word-embeddings and fine-tunes end-to-end.
Outcome:	The proposed method has higher BLEU score on translation and lower perplexity on language modeling compared to complex, difficult to tune methods.

Copied to clipboard

Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh, Sarath Chandar

How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding (2021.findings-emnlp)

Challenge:	Large language models (LLMs) are prohibitive to use under resource constraints due to their high latency and high latex.
Approach:	They propose to use a contextual bandit to help choose a model based on a context to improve performance.
Outcome:	The proposed model can be used to improve performance on multiple domains even without prior knowledge of the model.

Copied to clipboard

Tianda Li, Ahmad Rashid, Aref Jafari, Pranav Sharma, Ali Ghodsi, Mehdi Rezagholizadeh

Annealing Knowledge Distillation (2021.eacl-main)

Challenge:	Knowledge Distillation (KD) is a model compression algorithm that helps transfer knowledge in a large neural network into a smaller one.
Approach:	They propose a framework to assess adversarial robustness of multiple KD algorithms.
Outcome:	The proposed algorithm achieves state-of-the-art on the GLUE benchmark and out-of domain generalization and adversarial robustness compared to competitive methods.

Copied to clipboard

Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, Ali Ghodsi

MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages (2023.tacl-1)

Challenge:	Knowledge distillation (KD) is a powerful model compression technique for deep neural networks.
Approach:	They propose a method to feed the rich information provided by teacher’s soft-targets incrementally and more efficiently by annealing the teacher output incrementally.
Outcome:	The proposed method can be used on image classification and NLP language inference tasks with BERT-based models on the GLUE benchmark.

Copied to clipboard

Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, Jimmy Lin

Multilinguality

Challenge:	MIRACL is a multilingual dataset for ad hoc retrieval across 18 languages that collectively encompass over three billion native speakers around the world.
Approach:	They have gathered over 726k high-quality relevance judgments for 78k queries over Wikipedia in these languages, where all annotations have been performed by native speakers hired by their team.
Outcome:	MIRACL covers languages that are typologically close as well as distant from 10 language families and 13 sub-families, associated with varying amounts of publicly available resources.

Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation (2021.emnlp-main)

Copied to clipboard

Yimeng Wu, Mehdi Rezagholizadeh, Abbas Ghaddar, Md Akmal Haidar, Ali Ghodsi

ReGLA: Refining Gated Linear Attention (2025.naacl-long)

Challenge:	Existing methods for intermediate layer matching are limited due to huge over-parameterization .
Approach:	They propose to match intermediate layers of teacher and student in output space via attention-based layer projection.
Outcome:	The proposed method outperforms existing methods on GLUE tasks.

Copied to clipboard

Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh, Boxing Chen, Philippe Langlais

Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing (2022.emnlp-main)

Challenge:	Recent advances in Large Language Models (LLMs) are known for their computational and storage requirements due to the quadratic computation complexity of softmax attention.
Approach:	They propose to reduce the quadratic computation complexity of softmax attention by using feature maps, normalization and the gating mechanism to improve performance.
Outcome:	The proposed model outperforms existing gated linear attention models in extensive tasks including training from scratch and post-linearization with continual pre-training.

Copied to clipboard

Abbas Ghaddar, Yimeng Wu, Sunyam Bagga, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Xinyu Duan, Zhefeng Wang, Baoxing Huai, Xin Jiang, Qun Liu, Phillippe Langlais

Evaluating Embedding APIs for Information Retrieval (2023.acl-industry)

Challenge:	Existing pre-trained language models are not well-explored and are not reproducible in the literature.
Approach:	They propose to improve existing Arabic language pre-trained language models using a more methodical approach.
Outcome:	The proposed models outperform existing models on ALUE, a leaderboard-powered benchmark for Arabic NLU and NLG tasks.

Copied to clipboard

Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso-hermelo, Mehdi Rezagholizadeh, Jimmy Lin

Towards Fine-tuning Pre-trained Language Models with Integer Forward and Backward Propagation (2023.findings-eacl)

Challenge:	a growing number of language models are limiting their access to the community . we evaluate existing APIs for domain generalization and multilingual retrieval .
Approach:	They evaluate semantic embedding APIs in retrieval scenarios to assess their capabilities . they use BEIR and MIRACL to re-rank BM25 results using the APIs .
Outcome:	The proposed model is based on semantic embedding APIs that build vector representations of a given text.

Copied to clipboard

Mohammadreza Tayaranian Hosseini, Alireza Ghaffari, Marzieh S. Tahaei, Mehdi Rezagholizadeh, Masoud Asgharian, Vahid Partovi Nia

Latent Code and Text-based Generative Adversarial Networks for Soft-text Generation (N19-1)

Challenge:	Previously, researchers focused on lower bit-width integer data types for forward propagation and backward propagation of language models to save memory and computation.
Approach:	They use integer arithmetic for both forward and back propagation in the fine-tuning of BERT.
Outcome:	The proposed method improves on the GLUE and SQUAD benchmarks.

Copied to clipboard

Md. Akmal Haidar, Mehdi Rezagholizadeh, Alan Do Omri, Ahmad Rashid

Robustness in NLP

Challenge:	Text generation with generative adversarial networks (GANs) can be divided into text-based and code-based categories depending on the type of signals used for discrimination.
Approach:	They propose a text-based approach to exploit generative adversarial networks (GANs) by using autoencoders to provide a continuous representation of sentences, which they will refer to as soft-text, and hybrid latent code and text-oriented approaches with one or more discriminators.
Outcome:	The proposed approach outperforms the traditional GAN-based methods on two well-known datasets.

QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning (2024.emnlp-industry)

Copied to clipboard

Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Kwon, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh

Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization (2022.findings-emnlp)

Challenge:	Existing methods to fine tune large language models require huge memory, limiting the choice to acquire Larger models.
Approach:	They propose an efficient quantization approach for dynamic low-rank adaptation that can efficiently fine tune large language models on a set of pre-defined LoRA ranks.
Outcome:	The proposed method outperforms QLoRA and is competitive to QLouRA and outperformed when employing its optimal rank.

Copied to clipboard

Aref Jafari, Ivan Kobyzev, Mehdi Rezagholizadeh, Pascal Poupart, Ali Ghodsi

CILDA: Contrastive Data Augmentation Using Intermediate Layer Knowledge Distillation (2022.coling-1)

Challenge:	Existing methods for knowledge distillation (KD) do not mitigate the noise in the teacher’s output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features.
Approach:	They propose a method that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds.
Outcome:	The proposed method achieves state-of-the-art performance on NLU and computer vision tasks.

Copied to clipboard

Md Akmal Haidar, Mehdi Rezagholizadeh, Abbas Ghaddar, Khalil Bibi, Phillippe Langlais, Pascal Poupart

EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing (P19-1)

Challenge:	Knowledge distillation (KD) is an efficient framework for compressing large-scale pre-trained language models.
Approach:	They propose a data augmentation technique tailored for knowledge distillation based on contrastive loss to improve masked adversarial data augmented by intermediate layer matching.
Outcome:	The proposed technique outperforms state-of-the-art methods on the GLUE benchmark and in an out-of domain evaluation.

Copied to clipboard

Yue Dong, Zichao Li, Mehdi Rezagholizadeh, Jackie Chi Kit Cheung

Text Generation

Challenge:	Current sentence simplification systems are variants of sequence-to-sequence models adopted from machine translation.
Approach:	They propose a sentence simplification model that learns explicit edit operations via a neural programmer-interpreter approach.
Outcome:	The proposed model outperforms state-of-the-art models on three benchmark text simplification corpora in terms of SARI (+0.95 WikiLarge, +1.89 WikiSmall, -1.41 Newsela)

LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization (2023.findings-acl)

Copied to clipboard

Peng Lu, Ahmad Rashid, Ivan Kobyzev, Mehdi Rezagholizadeh, Phillippe Langlais

Information Retrieval

Challenge:	Existing methods for regularizing deep neural networks rely on weight decay, dropout, batch/layer normalization to converge faster and generalize.
Approach:	They propose a framework for training with label regularization which includes conventional LS but can also model instance-specific variants.
Outcome:	The proposed approach consistently yields better results than conventional regularization on seven machine translation and three image classification tasks while maintaining training efficiency.

RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation (2022.findings-naacl)

Copied to clipboard

Md Akmal Haidar, Nithin Anchuri, Mehdi Rezagholizadeh, Abbas Ghaddar, Philippe Langlais, Pascal Poupart

Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher (2022.coling-1)

Challenge:	Existing methods for intermediate layer knowledge distillation suffer from computational burdens and engineering efforts for setting up a proper layer mapping.
Approach:	They propose a method where intermediate layers from teacher and student models are randomly selected to be distilled into intermediate layers of student models.
Outcome:	The proposed method outperforms state-of-the-art intermediate layer knowledge distillation methods on GLUE tasks and out-of domain test sets.

Copied to clipboard

Mehdi Rezagholizadeh, Aref Jafari, Puneeth S.M. Saladi, Pranav Sharma, Ali Saheb Pasand, Ali Ghodsi

Fully Quantized Transformer for Machine Translation (2020.findings-emnlp)

Challenge:	Knowledge distillation (KD) is a powerful tool for deep learning applications.
Approach:	They propose a method which defines a smoother training path for the student by following the training footprints of the teacher rather than solely relying on distilling from a single mature fully-trained teacher.
Outcome:	The proposed technique is quite effective in mitigating the capacity-gap problem and the checkpoint search problem.

Copied to clipboard

Gabriele Prato, Ella Charlaix, Mehdi Rezagholizadeh

Efficient Citer: Tuning Large Language Models for Enhanced Answer Quality and Verification (2024.findings-naacl)

Challenge:	State-of-the-art neural machine translation methods use huge amounts of parameters.
Approach:	They propose an all-inclusive quantization strategy for the Transformer to reduce computational costs and improve translation quality.
Outcome:	The proposed method achieves state-of-the-art results on most tasks compared to previous methods .

Copied to clipboard

Marzieh Tahaei, Aref Jafari, Ahmad Rashid, David Alfonso-Hermelo, Khalil Bibi, Yimeng Wu, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh

Towards Zero-Shot Knowledge Distillation for Natural Language Processing (2021.emnlp-main)

Challenge:	Existing models with explicit citations lack the ability to verify information generated by these models.
Approach:	They construct a citation training dataset and fine-tune two models to address the challenge of explicit citations efficiently.
Outcome:	The proposed models surpass ChatGPT and exhibit exceptional out-of-domain generalization in both human and automatic evaluation.

Copied to clipboard

Ahmad Rashid, Vasileios Lioutas, Abbas Ghaddar, Mehdi Rezagholizadeh

CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search (2024.emnlp-main)

Challenge:	Knowledge distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions.
Approach:	They propose to use teacher training data for model compression . they investigate six tasks and find they can achieve between 75% and 92% of the teacher’s classification score while compressing the model 30 times.
Outcome:	The proposed solution achieves between 75% and 92% of the teacher’s classification score while compressing the model 30 times.

Copied to clipboard

Fengran Mo, Abbas Ghaddar, Kelong Mao, Mehdi Rezagholizadeh, Boxing Chen, Qun Liu, Jian-Yun Nie

Paraphrasing

Challenge:	Recent advances in task-solving capabilities of Large Language Models (LLMs) have motivated researchers to integrate these models into existing conversational search systems.
Approach:	They propose a method that leverages the capabilities of large language models to resolve ambiguities in conversation history before query rewriting.
Outcome:	The proposed method leads to state-of-the-art results across most settings compared with closed-source LLMs.

Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models (2025.emnlp-main)

Copied to clipboard

Benyamin Jamialahmadi, Parsa Kavehzadeh, Mehdi Rezagholizadeh, Parsa Farinneya, Hossein Rajabzadeh, Aref Jafari, Boxing Chen, Marzieh S. Tahaei

Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference (2024.findings-eacl)

Challenge:	Existing methods for dynamic inference are limited by hardware inefficiencies or performance degradation.
Approach:	They propose a framework for depth-based dynamic inference that freezes the pre-trained model and inserts additional transformer layers at selected exit points.
Outcome:	The proposed framework outperforms state-of-the-art methods such as Flextron and Layerskip on multiple models at various scales, as well as other leading compression techniques across a variety of benchmarks.

Copied to clipboard

Parsa Kavehzadeh, Mojtaba Valipour, Marzieh Tahaei, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh

Do we need Label Regularization to Fine-tune Pre-trained Language Models? (2023.eacl-main)

Challenge:	Large language models excel at understanding and generating human-like text, but their widespread deployment can be prohibitively expensive.
Approach:	They propose a method that makes large language models dynamic without Pre-Training . they use modularity in networks and sort sub-models based on computation/accuracy in a nested manner.
Outcome:	The proposed method can make large language models dynamic without pre-training and replace standard fine-tuning with sorted fine- tuning.

Copied to clipboard

Ivan Kobyzev, Aref Jafari, Mehdi Rezagholizadeh, Tianda Li, Alan Do-Omri, Peng Lu, Pascal Poupart, Ali Ghodsi

Kronecker Decomposition for GPT Compression (2022.acl-short)

Challenge:	Knowledge Distillation (KD) is a label regularization technique that can be replaced with lighter teacher-free variants such as the label-smoothing technique.
Approach:	They propose to use knowledge distillation to train student models by deploying the teacher network during training.
Outcome:	The proposed method can be replaced with lighter teacher-free variants on PLMs with more than 600 distinct trials and ran each configuration five times.

Copied to clipboard

Ali Edalati, Marzieh Tahaei, Ahmad Rashid, Vahid Nia, James Clark, Mehdi Rezagholizadeh

Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity (2024.findings-emnlp)

Challenge:	GPT is an auto-regressive Transformer-based pre-trained language model . but its huge size can be prohibitive for deploying on low capacity devices .
Approach:	They use a Kronecker decomposition technique to compress GPT models . they use ILKD to refine the model on downstream tasks .
Outcome:	The proposed model outperforms the existing DistilGPT2 model on language modeling and general language understanding evaluation benchmark tasks.

Copied to clipboard

Michael Metel, Peng Lu, Boxing Chen, Mehdi Rezagholizadeh, Ivan Kobyzev

Challenge:	Speculative decoding uses a small draft model to generate a single input token, instead of sequentially generating tokens until completion.
Approach:	They propose a method that generates varying draft models adapted to the input context using simple rules.
Outcome:	The proposed method is competitive with the current SOTA for self-speculative decoding while being a truly plug-and-play method.

EWEK-QA : Enhanced Web and Efficient Knowledge Graph Retrieval for Citation-based Question Answering Systems (2024.acl-long)

Copied to clipboard

Mohammad Dehghan, Mohammad Alomrani, Sunyam Bagga, David Alfonso-Hermelo, Khalil Bibi, Abbas Ghaddar, Yingxue Zhang, Xiaoguang Li, Jianye Hao, Qun Liu, Jimmy Lin, Boxing Chen, Prasanna Parthasarathi, Mahdi Biparva, Mehdi Rezagholizadeh

Natural Language Interfaces

Challenge:	citation-based QA systems are suffering from two shortcomings . they usually rely only on web as a source of extracted knowledge and external knowledge sources can hamper the efficiency of the system.
Approach:	They propose to use a web-based knowledge graph retrieval solution to enrich extracted knowledge fed to a citation-based QA system.
Outcome:	The proposed model outperforms open-source state-of-the-art models in 7 quantitative and human evaluation tasks.

Attribute Controlled Dialogue Prompting (2023.findings-acl)

Copied to clipboard

Runcheng Liu, Ahmad Rashid, Ivan Kobyzev, Mehdi Rezagholizadeh, Pascal Poupart

Dialogue Systems & Conversational Agents

Challenge:	Prompt-tuning is an increasingly popular parameter-efficient method for adapting large pretrained language models to downstream tasks.
Approach:	They propose an instance-specific prompt-tuning algorithm for dialog generation that generates prompts based on instance-level control code rather than the conversation history.
Outcome:	The proposed prompt-tuning module is a fraction of the size of the pretrained language model and saves memory and expensive storage space.

AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing (2023.findings-acl)

Copied to clipboard

Asaad Alghamdi, Xinyu Duan, Wei Jiang, Zhenhai Wang, Yimeng Wu, Qingrong Xia, Zhefeng Wang, Yi Zheng, Mehdi Rezagholizadeh, Baoxing Huai, Peilun Cheng, Abbas Ghaddar

MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation (2021.acl-long)

Challenge:	Developing monolingual large Pre-trained Language Models (PLMs) is shown to be very successful in handling different tasks in Natural Language Processing (NLP).
Approach:	They present AraMUS, the largest Arabic PLM with 11B parameters trained on 529GB of high-quality Arabic textual data.
Outcome:	The proposed model achieves state-of-the-art performance on a diverse set of Arabic classification and generative tasks.

Copied to clipboard

Ahmad Rashid, Vasileios Lioutas, Mehdi Rezagholizadeh

Practical Takes on Federated Learning with Pretrained Language Models (2023.findings-eacl)

Challenge:	Recent studies have shown that the trillion parameter mark for pre-trained language models is not achievable without a change in training paradigm.
Approach:	They propose a text-based adversarial training algorithm which enhances the performance of knowledge distillation by maximizing the divergence between teacher and student logits.
Outcome:	The proposed algorithm outperforms competing adversarial learning and data augmentation baselines on the GLUE benchmark.

Copied to clipboard

Ankur Agarwal, Mehdi Rezagholizadeh, Prasanna Parthasarathi

DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2023.eacl-main)

Challenge:	federated learning with pretrained language models for language tasks entails data privacy constraints when learning from diverse data domains.
Approach:	They propose to use pretrained language models to learn from diverse data domains . they elaborate hypotheses over the components in federated NLP architectures based on three tasks .
Outcome:	The proposed model can generalize by adapting to the different domains.

Copied to clipboard

Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, Ali Ghodsi

KroneckerBERT: Significant Compression of Pre-trained Language Models Through Kronecker Decomposition and Knowledge Distillation (2022.naacl-main)

Challenge:	Pre-training/fine-tuning of pre-training models has become more expensive and resource-hungry.
Approach:	They propose a low-rank adaptation technique that trains LoRA blocks for a range of ranks instead of a single rank.
Outcome:	The proposed method trains LoRA blocks for a range of ranks instead of a single rank . it can train dynamic search-free models with DyLoRA at least 4 to 7 times faster than LoRA .

Copied to clipboard

Marzieh Tahaei, Ella Charlaix, Vahid Nia, Ali Ghodsi, Mehdi Rezagholizadeh