Papers by Subhabrata Mukherjee
Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have focused on compressing pre-trained language models (PLMs) however, few studies have examined the impact of compression on generalizability and robustness of compressed models for out-of-distribution data. |
| Approach: | They propose to use knowledge distillation and pruning to reduce model generalization and generalization on out-of-distribution data. |
| Outcome: | The proposed compression techniques overfit on shortcut samples and generalize poorly on hard ones. |
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation (2023.findings-acl)
Copied to clipboard
Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks Lakshmanan, V.S., Ahmed Hassan Awadallah, Sebastien Bubeck, Jianfeng Gao
| Challenge: | Existing MoE designs do not consider computational constraints (e.g., FLOPs, latency) Existing works in MoE consider homogeneous design where the same number of experts of the same size are placed uniformly throughout the network. |
| Approach: | They propose a framework for designing heterogeneous MoEs under computational constraints. |
| Outcome: | The proposed framework achieves 4x inference speedup and FLOPs reduction over manual models and within 1 BLEU point of MoE SwitchTransformer over benchmark datasets for NMT. |
Self-Training with Weak Supervision (2021.naacl-main)
Copied to clipboard
| Challenge: | State-of-the-art deep neural networks require large amounts of labeled training data that is expensive to obtain or not available for many tasks. |
| Approach: | They propose a weak supervision framework that leverages all available data for a given task . they leverage task-specific unlabeled data through self-training with a model that predicts pseudo-labels for instances that may not be covered by weak rules . |
| Outcome: | The proposed framework improves on state-of-the-art datasets on six benchmark tasks. |
OpenKI: Integrating Open Information Extraction and Knowledge Bases with Relation Inference (N19-1)
Copied to clipboard
| Challenge: | Existing methods for knowledge extraction and alignment are limited in quality and performance. |
| Approach: | They propose to integrate OpenIE extractions in the form of (subject, predicate, object) triples with Knowledge Bases (KB) |
| Outcome: | The proposed method improves state-of-the-art for OpenIE extractions and boosts performance on OpenIE from semi-structured data. |
Red Queen: Exposing Latent Multi-Turn Risks in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing jailbreak strategies are single-turn with explicit malicious intent, failing to reflect real-world scenarios where interactions can be multi-turn and users can conceal intents. |
| Approach: | They propose a jailbreak attack that constructs a multi-turn scenario and conceals malicious intent under the guise of preventing harm. |
| Outcome: | The proposed attack achieves 87.6% attack success rate (ASR) on GPT-4o and 77.1% on Llama3-70B compared to previous jailbreak attacks . |
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained language models focuses on reducing the size of these models into shallow ones. |
| Approach: | They propose a knowledge distillation technique that leverages teacher internal representations to reduce the size of pre-trained language models. |
| Outcome: | The proposed method outperforms previous methods in multilingual Named Entity Recognition (NER) it reduces the size of teacher models by 35x while retaining 95% of its F1 score. |
DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning (D18-1)
Copied to clipboard
| Challenge: | Recent work on automated fact-checking does not consider external evidence, but requires rich lexicons. |
| Approach: | They propose a neural network model that aggregates external evidence and language . they also derive informative features for generating user-comprehensible explanations . |
| Outcome: | The proposed model aggregates signals from external evidence articles, language and trustworthiness of their sources without human intervention. |
Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer (2020.acl-main)
Copied to clipboard
| Challenge: | Multilingual word embeddings embed words from many languages into a single semantic space such that words with similar meanings are close to each other regardless of the language. |
| Approach: | They propose to use multilingual word embeddings to align embeddable words from multiple languages into a single semantic space so that words with similar meanings are close to each other regardless of the language. |
| Outcome: | The proposed model can be used to learn gender bias in multilingual representations and to improve transfer learning. |
Self-training with Few-shot Rationalization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work focused on training largescale and complex neural network models, but they are opaque in terms of their decision-making process. |
| Approach: | They propose a multi-task teacher-student framework for self-training pre-trained language models with limited task-specific labels and annotated rationales. |
| Outcome: | The proposed model improves performance in low-resource settings by making it aware of its rationalized predictions. |
Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent few-shot learning methods focus on improving downstream task performance, but there is limited understanding of the adversarial robustness of such methods. |
| Approach: | They evaluate prompt-based FSL methods against fully fine-tuned models to better understand the impact of various factors towards robustness. |
| Outcome: | The proposed methods show that they are less robust in the face of adversarial perturbations than fully fine-tuned models. |
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning (2022.emnlp-main)
Copied to clipboard
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao
| Challenge: | Standard fine-tuning of large pre-trained language models requires updating hundreds of millions to billions of parameters and storing a large copy of the PLM weights for every task. |
| Approach: | They propose a parameter-efficient fine-tuning technique where small trainable components are injected into the PLM and updated during fine-uning. |
| Outcome: | The proposed method outperforms SOTA parameter-efficient fine-tuning and full model fine-uning on GLUE development set with RoBERTa-large encoder. |
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training (2023.acl-long)
Copied to clipboard
| Challenge: | Modern Natural Language Generation models come with massive computational and storage requirements. |
| Approach: | They propose a method that applies word-level knowledge distillation to multiple PTs generated by both teacher and student. |
| Outcome: | The proposed techniques can be used to compress natural language models while preserving their performance. |
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners (2022.findings-naacl)
Copied to clipboard
| Challenge: | LiST is an efficient method for fine-tuning large pre-trained language models in few-shot learning settings. |
| Approach: | They propose a method for efficient fine-tuning of large pre-trained language models in few-shot settings using self-training and meta-learning. |
| Outcome: | The proposed method outperforms GPT-3 in-context learning by 33% on few-shot tasks. |
MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning (2021.naacl-main)
Copied to clipboard
Mengzhou Xia, Guoqing Zheng, Subhabrata Mukherjee, Milad Shokouhi, Graham Neubig, Ahmed Hassan Awadallah
| Challenge: | Recent work shows that multilingual representations are disjointed across languages, bringing additional challenges for transfer onto extremely low-resource languages. |
| Approach: | They propose a meta-learning based framework that learns to transform representations judiciously from auxiliary languages to a target one and brings their representation spaces closer for effective transfer. |
| Outcome: | The proposed framework learns to transform representations from auxiliary languages to a target language and brings their representation spaces closer for effective transfer. |
STANCY: Stance Classification Based on Consistency Cues (D19-1)
Copied to clipboard
| Challenge: | Recent work has shown that stance classification is a critical step for information credibility and automated fact-checking. |
| Approach: | They propose a neural network model for stance classification leveraging BERT representations and augmenting them with a novel consistency constraint. |
| Outcome: | The proposed model outperforms existing methods on a Perspectrum dataset and shows that it is more accurate than existing methods. |
Smart To-Do: Automatic Generation of To-Do Items from Emails (2020.acl-main)
Copied to clipboard
| Challenge: | Using neural text generation, we generate To-Do items from emails where the sender has promised to perform an action. |
| Approach: | They propose a task and dataset for automatically generating To-Do items from emails where the sender has promised to perform an action. |
| Outcome: | The proposed task obtains BLEU and ROUGE scores of 0.23 and 0.63 for the task. |