Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained sequence-to-sequence models have advanced the state of the art on text generation tasks. |
| Approach: | They introduce a modular encoder-decoder framework for flexible sequence-to-sequence model compression. |
| Outcome: | The proposed framework can achieve flexible compression ratios from 1.1x to 6x with little to moderate relative performance drop. |
Similar Papers
Emergent Modularity in Pre-trained Transformers (2023.findings-acl)
Copied to clipboard
Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Jie Zhou
| Challenge: | Existing studies on pre-trained Transformers show that they learn fine-grained neuron functions. |
| Approach: | They examine the presence of modularity in pre-trained Transformers . they focus on Mixture-of-Experts, a promising candidate for modularity . |
| Outcome: | The proposed structure stabilizes at the early stage, which is faster than neuron stabilization. |
Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules (2024.emnlp-main)
Copied to clipboard
| Challenge: | Empirical results show that MoMs consistently outperform vanilla transformers . |
| Approach: | They propose an architecture that allows for a mixture-of-modules computation that uses a finite set of modules defined by multi-head attention and feed-forward networks. |
| Outcome: | The proposed architecture outperforms vanilla Transformers and their variants in multiple ways. |
Distillation of encoder-decoder transformers for sequence labelling (2023.findings-eacl)
Copied to clipboard
| Challenge: | despite the strong trend in NLP to explore the use of large language models, there is still limited work evaluating prompting and decoding mechanisms for SL tasks. |
| Approach: | They propose a hallucination-free framework for sequence tagging that is especially suited for distillation. |
| Outcome: | The proposed framework performs well across multiple sequence labelling datasets and in a few-shot learning scenario. |
On the Way to Lossless Compression of Language Transformers: Exploring Cross-Domain Properties of Quantization (2024.lrec-main)
Copied to clipboard
Nikita Martynov, Aleksei Goncharov, Gleb Kumichev, Evgeniy Egorov, Stanislav Vladimirovich Pavlov, Mikhail Sergeevich Durinov, Aleksandr Sergeevich Zuev, Egor Anatolievich Filimonov
| Challenge: | Modern Natural Language Processing models have a huge capacity, but this makes it difficult to employ. |
| Approach: | They propose a method to quantize at least 95% of Transformer weights without access to task-specific data so the drop in performance does not exceed 0.02%. |
| Outcome: | The proposed method quantizes 95% of Transformer weights and corresponding activations to INT8 without access to task-specific data so the drop in performance does not exceed 0.02%. |
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in transformer quantization have shown remarkable improvement in many Natural Language Processing tasks and beyond. |
| Approach: | They propose a novel quantization scheme for transformers that can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. |
| Outcome: | The proposed methods achieve state-of-the-art results on the GLUE benchmark using BERT, while preserving memory and accuracy. |
Efficient Transformer Knowledge Distillation: A Performance Review (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Pretrained transformer language models have been gaining popularity in the field of natural language processing . however, there is no study into the intersection of these two fields . |
| Approach: | They propose a method to extract knowledge from transformers to produce high-performing efficient attention models with low costs. |
| Outcome: | The proposed model compression method preserves up to 98.6% of original model performance across short-context tasks and up to 95.8% on long-concept Named Entity Recognition tasks while decreasing inference times by up to 57%. |
Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent transformer language models achieve outstanding results on many downstream tasks, but their enormous size often makes them impractical on memory-constrained devices. |
| Approach: | They propose an offline compression approach that reduces the complexity of the model by enabling collaboration between modules. |
| Outcome: | The proposed approach outperforms commonly used factorization-based offline compression methods on various NLP tasks. |
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2021.tacl-1)
Copied to clipboard
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, Marianne Winslett
| Challenge: | Popular pre-trained Transformers have improved performance for various NLP tasks by sizable margins, but are too resource-hungry and computation-intensive to suit low-capacity devices or applications with strict latency requirements. |
| Approach: | They present a literature review of the compression of Transformers, focusing on the popular BERT model, which has attracted considerable research attention. |
| Outcome: | The proposed models improve Sentiment analysis, paraphrase detection, machine reading comprehension, question answering, text summarization, and other tasks by sizable margins. |
Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Existing methods for quantization-aware training and quantization for learning have limitations in dealing with accumulative quantization errors. |
| Approach: | They propose a method that minimizes accumulative quantization errors and outperforms baselines by distilling knowledge from a fine-tuned teacher network. |
| Outcome: | The proposed method minimizes accumulative quantization errors and outperforms baselines on the XGLUE benchmark. |
On Sparsifying Encoder Outputs in Sequence-to-Sequence Models (2021.findings-acl)
Copied to clipboard
| Challenge: | Using sequence-to-sequence models, encoder outputs are usually transferred to the decoder for generation, but in this study, encoded outputs can be compressed to shorten the sequence for decoding. |
| Approach: | They propose to use a stochastic gate-based algorithm to mask encoder outputs to shorten the sequence delivered for decoding. |
| Outcome: | The proposed model can be used to shorten encoder outputs to short a sequence . the proposed model yields a speedup of up to 1.65 on document summarization and 1.20 on character-based machine translation tasks. |