Papers by Pascal Poupart
WatClaimCheck: A new Dataset for Claim Entailment and Inference (2022.acl-long)
Copied to clipboard
| Challenge: | a dataset for automated fact checking is presented . premise articles are used to verify the veracity of claims . |
| Approach: | They propose a dataset for automated fact checking and an evaluation of state of the art algorithms. |
| Outcome: | The proposed model improves retrieval quality of passages in premise articles . the proposed model predicts claim veracity by inference from premise article . |
Contrastive Deterministic Autoencoders For Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. |
| Approach: | They propose to modify a deterministic model designed for images to avoid posterior collapse by controlling the entropy of the aggregate posterior to make it Gaussian. |
| Outcome: | The proposed models outperform a broad range of VAE models on text generation and downstream tasks from representations while avoiding reparametrization steps. |
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation (KD) do not mitigate the noise in the teacher’s output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. |
| Approach: | They propose a method that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. |
| Outcome: | The proposed method achieves state-of-the-art performance on NLU and computer vision tasks. |
CILDA: Contrastive Data Augmentation Using Intermediate Layer Knowledge Distillation (2022.coling-1)
Copied to clipboard
Md Akmal Haidar, Mehdi Rezagholizadeh, Abbas Ghaddar, Khalil Bibi, Phillippe Langlais, Pascal Poupart
| Challenge: | Knowledge distillation (KD) is an efficient framework for compressing large-scale pre-trained language models. |
| Approach: | They propose a data augmentation technique tailored for knowledge distillation based on contrastive loss to improve masked adversarial data augmented by intermediate layer matching. |
| Outcome: | The proposed technique outperforms state-of-the-art methods on the GLUE benchmark and in an out-of domain evaluation. |
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation (2022.findings-naacl)
Copied to clipboard
Md Akmal Haidar, Nithin Anchuri, Mehdi Rezagholizadeh, Abbas Ghaddar, Philippe Langlais, Pascal Poupart
| Challenge: | Existing methods for intermediate layer knowledge distillation suffer from computational burdens and engineering efforts for setting up a proper layer mapping. |
| Approach: | They propose a method where intermediate layers from teacher and student models are randomly selected to be distilled into intermediate layers of student models. |
| Outcome: | The proposed method outperforms state-of-the-art intermediate layer knowledge distillation methods on GLUE tasks and out-of domain test sets. |
Variational Attention for Sequence-to-Sequence Models (C18-1)
Copied to clipboard
| Challenge: | Existing variational autoencoders encode data to latent variables and then decode them into target data. |
| Approach: | They propose a variational attention mechanism where the attention vector is also modeled as Gaussian distributed random variables. |
| Outcome: | The proposed method reduces the variational latent space bypassing phenomenon as it increases diversity of generated sentences. |
Do we need Label Regularization to Fine-tune Pre-trained Language Models? (2023.eacl-main)
Copied to clipboard
Ivan Kobyzev, Aref Jafari, Mehdi Rezagholizadeh, Tianda Li, Alan Do-Omri, Peng Lu, Pascal Poupart, Ali Ghodsi
| Challenge: | Knowledge Distillation (KD) is a label regularization technique that can be replaced with lighter teacher-free variants such as the label-smoothing technique. |
| Approach: | They propose to use knowledge distillation to train student models by deploying the teacher network during training. |
| Outcome: | The proposed method can be replaced with lighter teacher-free variants on PLMs with more than 600 distinct trials and ran each configuration five times. |
Attribute Controlled Dialogue Prompting (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt-tuning is an increasingly popular parameter-efficient method for adapting large pretrained language models to downstream tasks. |
| Approach: | They propose an instance-specific prompt-tuning algorithm for dialog generation that generates prompts based on instance-level control code rather than the conversation history. |
| Outcome: | The proposed prompt-tuning module is a fraction of the size of the pretrained language model and saves memory and expensive storage space. |