Compressing Pre-trained Language Models by Matrix Decomposition (2020.aacl-main)
Copied to clipboard
| Challenge: | Large pre-trained language models reach state-of-the-art results when fine-tuned individually; They also come with a significant memory and computational requirements, calling for methods to reduce model sizes (green AI). |
| Approach: | They propose a two-stage model-compression method to reduce a model’s inference time cost by decompressing the model into smaller matrices and performing feature distillation on the internal representation. |
| Outcome: | The proposed method reduces the number of parameters by 0.4x and increases inference speed by 1.45x while preserving the information contained within the model. |
Similar Papers
KroneckerBERT: Significant Compression of Pre-trained Language Models Through Kronecker Decomposition and Knowledge Distillation (2022.naacl-main)
Copied to clipboard
| Challenge: | a recent study shows that over-parameterized pre-trained language models are unsuitable for low-capacity devices. |
| Approach: | They propose a transformer-based pre-trained language model that is overparameterized . they use a two-stage knowledge distillation scheme to train the model . |
| Outcome: | The proposed model outperforms state-of-the-art models on well-known NLP benchmarks. |
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2021.tacl-1)
Copied to clipboard
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, Marianne Winslett
| Challenge: | Popular pre-trained Transformers have improved performance for various NLP tasks by sizable margins, but are too resource-hungry and computation-intensive to suit low-capacity devices or applications with strict latency requirements. |
| Approach: | They present a literature review of the compression of Transformers, focusing on the popular BERT model, which has attracted considerable research attention. |
| Outcome: | The proposed models improve Sentiment analysis, paraphrase detection, machine reading comprehension, question answering, text summarization, and other tasks by sizable margins. |
Compression of Generative Pre-trained Language Models via Quantization (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to compress generative pre-trained language models fail on generative tasks due to homogeneous word embeddings and limited memory. |
| Approach: | They propose a token-level contrastive distillation method to learn distinguishable word embeddings and a module-wise dynamic scaling method to make quantizers adaptive to different modules. |
| Outcome: | The proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. |
Kronecker Decomposition for GPT Compression (2022.acl-short)
Copied to clipboard
| Challenge: | GPT is an auto-regressive Transformer-based pre-trained language model . but its huge size can be prohibitive for deploying on low capacity devices . |
| Approach: | They use a Kronecker decomposition technique to compress GPT models . they use ILKD to refine the model on downstream tasks . |
| Outcome: | The proposed model outperforms the existing DistilGPT2 model on language modeling and general language understanding evaluation benchmark tasks. |
Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization (2023.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models (PLMs) have shown remarkable performance in various natural language processing tasks, outperforming small PLMs by a large margin. |
| Approach: | They propose to scale up parameters of pre-trained language models only during fine-tuning to benefit from over-parameterization. |
| Outcome: | The proposed approach can significantly boost the fine-tuning performance of small PLMs and even help small PDMs outperform 3 parameterized larger ones. |
Language Model Decomposition: Quantifying the Dependency and Correlation of Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (LMs) have led to significant improvements on various NLP tasks in past years, but a theoretical framework for studying their relationships is still missing. |
| Approach: | They propose to use language model decomposition to represent a set of pre-trained LMs and derive a closed-form solution. |
| Outcome: | The proposed model is based on a language model decomposition (LMD) and its variants. |
The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on LLM compression focuses on general metrics like perplexity or downstream task accuracy. |
| Approach: | They propose to quantify the effect of pruning and quantization on model quality . they use the LAMA and LM-Harness benchmarks to quantify compression techniques . |
| Outcome: | The proposed compression techniques provide faster inference, smaller memory footprints, and enables local deployment. |
Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for pre-trained language models (PLMs) use parameter reduction techniques. |
| Approach: | They propose a pre-trained language model compression approach based on the matrix product operator from quantum many-body physics. |
| Outcome: | The proposed approach can decompose an original matrix into central tensors and auxiliary tenses . it can be applied to the original or compressed PLMs in a general way, with a lighter network . |
When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit excellent performance in various tasks, but memory requirements present a challenge when deploying on memory-limited devices. |
| Approach: | They propose a framework to compress LLM after quantization further, achieving about 2.2x compression ratio. |
| Outcome: | The proposed model can achieve 40% reduction in memory size with negligible loss in accuracy and inference speed. |
Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization (2024.findings-emnlp)
Copied to clipboard
Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, Kehai Chen, Min Zhang
| Challenge: | Large language models require a balance between efficiency and performance. |
| Approach: | They propose a low-rank compression technique that reduces non-essential parameters by decomposing weight matrices into products of two low-ranked matrici. |
| Outcome: | The proposed method outperforms existing pruning and low-rank compression techniques in maintaining model performance at the same compression ratio. |