Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have focused on compressing pre-trained language models (PLMs) however, few studies have examined the impact of compression on generalizability and robustness of compressed models for out-of-distribution data. |
| Approach: | They propose to use knowledge distillation and pruning to reduce model generalization and generalization on out-of-distribution data. |
| Outcome: | The proposed compression techniques overfit on shortcut samples and generalize poorly on hard ones. |
Similar Papers
Pretrained Transformers Improve Out-of-Distribution Robustness (2020.acl-main)
Copied to clipboard
| Challenge: | Pretrained Transformers are more effective at detecting anomalous or OOD examples, while many previous models are frequently worse than chance. |
| Approach: | They construct a new robustness benchmark with real distribution shifts to measure out-of-distribution generalization for seven NLP datasets and compare them to previous models. |
| Outcome: | The proposed model generalizations for seven datasets show that pretrained Transformers are significantly less effective at detecting anomalous or OOD examples, while many previous models are often worse than chance. |
Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on compression prioritizes preserving perplexity, which is analogous to training loss. |
| Approach: | They examine the impact of model compression along four dimensions: degeneration harm, representational harm, dialect bias, and language modeling and downstream task performance. |
| Outcome: | The proposed compression methods can lead to unexpected consequences, the authors show . quantization preserves bias while pruning degrades quickly. |
Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some performance benefits. |
| Approach: | They propose to augment the distillation with generated unlabelled examples that match the target distribution and upsamples data points among the training set that are similar to the target. |
| Outcome: | The proposed method outperforms previous robustness solutions on the task of natural language inference (NLI) it also improves performance on OOD domains even beyond the target domain. |
Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm (2022.acl-long)
Copied to clipboard
Shaoyi Huang, Dongkuan Xu, Ian Yen, Yijue Wang, Sung-En Chang, Bingbing Li, Shiyang Chen, Mimi Xie, Sanguthevar Rajasekaran, Hang Liu, Caiwen Ding
| Challenge: | Conventional wisdom in pruning Transformer-based language models is that it reduces model expressiveness, but new research shows pruning increases risk of overfitting when performed at the fine-tuning phase. |
| Approach: | They propose to reduce pruning risk under pretrain-and-finetune paradigm . they propose to use knowledge distillation to improve pruning performance . |
| Outcome: | The proposed method outperforms the leading competitors on the GLUE benchmark. |
Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning strategies struggle to enhance robustness against adversarial attacks when continually increasing model sparsity and require a retraining process. |
| Approach: | They propose a pruning strategy that replicates embedding space and feature space of dense language models and aims to conserve more pre-trained knowledge during the pruning process. |
| Outcome: | The proposed pruning strategy replicates embedding space and feature space of dense language models, aiming to conserve more pre-trained knowledge during the pruning process. |
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies on compression of pretrained language models usually use preserved accuracy as the metric for evaluation. |
| Approach: | They propose two new metrics that measure how closely a compressed model mimics the original model. |
| Outcome: | The proposed metrics measure how closely a compressed model (i.e., student) mimics the original model (e.g., teacher). |
GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-Distribution Generalization Perspective (2023.findings-acl)
Copied to clipboard
Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, Yue Zhang
| Challenge: | Pre-trained language models (PLMs) have improved generalization performance but the out-of-distribution (OOD) generalization problem remains a challenge in many NLP tasks. |
| Approach: | They propose to create a benchmark for evaluating out-of-distribution (OOD) generalization in NLP models. |
| Outcome: | The proposed benchmarks highlight the importance of OOD robustness and provide insights on how to measure it and improve it. |
Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in hardware, modeling, and optimization for deep neural networks have led to improvements in memory and inference efficiency. |
| Approach: | They propose to combine sharpness-aware minimization with various model compression methods to improve model compressibility. |
| Outcome: | Empirically, optimizing for flatter minima leads to greater compressibility of parameters compared to vanilla Adam when fine-tuning BERT models, with little to no loss in accuracy on the GLUE text classification and SQuAD question answering benchmarks. |
Out-of-Distribution Generalization in Natural Language Processing: Past, Present, and Future (2023.emnlp-main)
Copied to clipboard
Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, Yue Zhang
| Challenge: | Existing literature on the generalization of machine learning models to out-of-distribution data is lacking. |
| Approach: | They propose to present the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
| Outcome: | The proposed survey provides the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
Are Compressed Language Models Less Subgroup Robust? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to reduce model size and latency while retaining overall performance are not known about their impact on subgroup robustness. |
| Approach: | They investigate the effects of model compression on subgroup robustness of BERT language models. |
| Outcome: | The proposed compression methods do not worsen the performance on minority subgroups. |