| Challenge: | Prior work has proposed to augment Transformer model with the capability of skimming tokens to improve its computational efficiency. |
| Approach: | They propose to add a parameterized predictor before each layer that learns to make the skimming decision. |
| Outcome: | The proposed model achieves 10.97x speedup on GLUE benchmark compared with BERT-base baseline with less than 1% accuracy degradation. |
Similar Papers
AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to train transformers with millions of parameters require large storage. |
| Approach: | They propose a transformer-based adapter architecture that adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer. |
| Outcome: | The proposed model significantly reduces trainable parameters with minimal performance loss compared to fine-tuned models. |
Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for NLP tasks require fine-tuning, but it is computationally infeasible. |
| Approach: | They propose an approach that inexpensively estimates a ranking of the expected performance of a given set of transformer language models for a specific task. |
| Outcome: | The proposed model improves the Pearson correlation coefficient between the true model ranks and the estimate. |
AdapterDrop: On the Efficiency of Adapters in Transformers (2021.emnlp-main)
Copied to clipboard
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, Iryna Gurevych
| Challenge: | Recent approaches to transformer models are expensive to fine-tune, slow for inference, and have large storage requirements. |
| Approach: | They propose a method to remove adapters from transformer layers during training and inference . they show that AdapterDrop can dynamically reduce computational overhead . |
| Outcome: | The proposed approach reduces computational overhead while maintaining performance over multiple tasks with minimal loss of performance. |
Sparsifying Transformer Models with Trainable Representation Pooling (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to sparsify attention in the Transformer model are based on quadratic memory complexity and a lack of information for each word. |
| Approach: | They propose a method to sparsify attention in a Transformer model by learning to select the most-informative token representations during the training process. |
| Outcome: | The proposed model performs better than the current SOTA model while being 1.8 faster during training, 4.5 faster inference and 13 more efficient in the decoder. |
Token-Wise Kernels (TWiKers) for Vicinity-Aware Attention in Transformers (2026.findings-eacl)
Copied to clipboard
| Challenge: | Token-Wise Kernels (TWiKers) are a novel enhancement to transformers that learn token-specific convolutional kernels applied to the keys or values. |
| Approach: | They propose a transformer enhancement that learns token-specific convolutional kernels applied to the keys or values. |
| Outcome: | The proposed transformers learn token-specific convolutional kernels applied to the keys or values . the results show that content words retain self-focus while function words shift attention toward their neighbors . |
TADA: Efficient Task-Agnostic Domain Adaptation for Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained transformer-based language models are limited in their expressiveness and domain knowledge. |
| Approach: | They propose a task-agnostic domain adaptation method which is modular, parameter-efficient, and data-efficient. |
| Outcome: | The proposed method is efficient and modular, parameter-efficient, and data-efficient. |
Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent improvements in NLP tasks can be attributed to the Transformer model. |
| Approach: | They propose to use parameter-sharing methods to reduce parameter budgets in generative models by using sandwich-style parameter sharing and self-attentive embedding factorization. |
| Outcome: | The proposed model outperforms the current RNN model even with significantly fewer parameters. |
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022.emnlp-main)
Copied to clipboard
| Challenge: | Fig. 1 shows how feed-forward network (FFN) layers are utilized to build LMs. |
| Approach: | They reverse-engineer the operation of feed-forward network layers to find out how they work . they show that each update can be decomposed to sub-updates corresponding to single parameter vectors . |
| Outcome: | The proposed model reduces the toxicity of GPT2 by almost 50% and improves computation efficiency with a simple early exit rule, saving 20% of computation on average. |
Bag of Tricks for Optimizing Transformer Efficiency (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Improving Transformer efficiency has become increasingly attractive in recent years. |
| Approach: | They propose to combine pruning, quantization, new architectures and training strategies to improve Transformer efficiency. |
| Outcome: | The proposed methods improve the inference efficiency of a strong Transformer system by 3.80x on CPU and 2.52x on GPU. |
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in transformer quantization have shown remarkable improvement in many Natural Language Processing tasks and beyond. |
| Approach: | They propose a novel quantization scheme for transformers that can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. |
| Outcome: | The proposed methods achieve state-of-the-art results on the GLUE benchmark using BERT, while preserving memory and accuracy. |