Zhengyan Zhang, Baitao Gong, Yingfa Chen, Xu Han, Guoyang Zeng, Weilin Zhao, Yanxu Chen, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing efforts to compress medium-sized models for specific tasks have limited results. |
| Approach: | They propose a task-agnostic compression toolkit for big models that implements quantization, pruning, distillation and MoEfication methods. |
| Outcome: | The proposed tool improves performance on a model with 3 billion parameters by 12x . it also outperforms the original model on three typical NLP benchmarks. |
Similar Papers
DisComp: A Two-Stage Prompt Optimization Framework Combining Task-Agnostic and Task-Aware Compression (2025.findings-naacl)
Copied to clipboard
| Challenge: | Extended prompts can lead to substantial computational overhead and increased hardware demands, limiting the scalability and efficiency of large language models. |
| Approach: | They propose a two-stage prompt compression framework that combines task-agnostic and task-based strategies to efficiently compress prompt length without compromising performance. |
| Outcome: | The proposed framework outperforms task-agnostic and task-specific compression methods on three benchmark datasets and is up to 6.56 faster at inference compared to the best token-level compression method. |
PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
Zhuocheng Gong, Jiahao Liu, Qifan Wang, Yang Yang, Jingang Wang, Wei Wu, Yunsen Xian, Dongyan Zhao, Rui Yan
| Challenge: | Quantization is a viable solution for pre-trained language models, but most existing methods are task-specific and require customized training and quantization with a large number of trainable parameters. |
| Approach: | They propose a "quantize before fine-tuning" framework that allows for quantization with a large number of trainable parameters on each individual task. |
| Outcome: | The proposed framework is compatible with quantization-aware training and post-training quantization and corrects quantization errors. |
BMInf: An Efficient Toolkit for Big Model Inference and Tuning (2022.acl-demo)
Copied to clipboard
Xu Han, Guoyang Zeng, Weilin Zhao, Zhiyuan Liu, Zhengyan Zhang, Jie Zhou, Jun Zhang, Jia Chao, Maosong Sun
| Challenge: | Recent years, pre-trained language models (PLMs) have achieved promising results on various NLP tasks. |
| Approach: | They propose an open-source toolkit for big model inference and tuning which can support big model tuning at extremely low computation cost. |
| Outcome: | The proposed toolkit can support big model inference and tuning at extremely low computation cost. |
LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit (2024.emnlp-industry)
Copied to clipboard
Ruihao Gong, Yang Yong, Shiqiao Gu, Yushi Huang, Chengtao Lv, Yunchen Zhang, Dacheng Tao, Xianglong Liu
| Challenge: | Existing quantization techniques have been categorized as 'simple' and 'highly efficient' however, their configurations vary from each other and cannot be fairly compared . |
| Approach: | They propose a plug-and-play compression toolkit to explore the impact of quantization. |
| Outcome: | The proposed toolkit explores the impact of quantization on large language models. |
Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in hardware, modeling, and optimization for deep neural networks have led to improvements in memory and inference efficiency. |
| Approach: | They propose to combine sharpness-aware minimization with various model compression methods to improve model compressibility. |
| Outcome: | Empirically, optimizing for flatter minima leads to greater compressibility of parameters compared to vanilla Adam when fine-tuning BERT models, with little to no loss in accuracy on the GLUE text classification and SQuAD question answering benchmarks. |
DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods rely on information entropy as the metric to compress lexical units, but ignore attention-critical tokens and information . recent advent of In-Context Learning (ICL), Chain-of-Thought (CoT), and Retrieval Augmented Generation (RAG) technologies has significantly invigorated the landscape of applications based on Large Language Models (LLMs). |
| Approach: | They propose a dynamic attention-aware approach to task-agnostic prompt compression . they integrate entropy and attention information to achieve fine-grained prompt compression. |
| Outcome: | Experiments show that the proposed approach improves across tasks and LLMs. |
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024.findings-acl)
Copied to clipboard
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Dongmei Zhang
| Challenge: | Existing approaches to compress prompts only leverage unidirectional context, causing suboptimal results. |
| Approach: | They propose a task-agnostic prompt compression method that takes tokens from context . they use a Transformer encoder to capture all essential information needed for prompt compression . |
| Outcome: | The proposed method is 3x-6x faster than existing prompt compression methods and faster than baselines. |
When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit excellent performance in various tasks, but memory requirements present a challenge when deploying on memory-limited devices. |
| Approach: | They propose a framework to compress LLM after quantization further, achieving about 2.2x compression ratio. |
| Outcome: | The proposed model can achieve 40% reduction in memory size with negligible loss in accuracy and inference speed. |
Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt compression reduces inference time and costs while maintaining informativeness for different usage scenarios. |
| Approach: | They propose a framework that adapts a smaller language model to compress prompts for a larger model on a new task without additional training. |
| Outcome: | The proposed framework outperforms two baseline models in four tasks . iteratively generates and selects effective compressed prompts as task-specific demonstrations . |
Compression of Generative Pre-trained Language Models via Quantization (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to compress generative pre-trained language models fail on generative tasks due to homogeneous word embeddings and limited memory. |
| Approach: | They propose a token-level contrastive distillation method to learn distinguishable word embeddings and a module-wise dynamic scaling method to make quantizers adaptive to different modules. |
| Outcome: | The proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. |