SHARCS: Efficient Transformers Through Routing with Dynamic Width Sub-networks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Several methods have been proposed to improve the inference efficiency of transformer-based models. |
| Approach: | They propose a new adaptive inference method that takes into account the hardness of input samples. |
| Outcome: | The proposed model outperforms or complements existing per-sample adaptive inference methods in terms of accuracy vs. FLOPs and can be applied to compressed and efficient transformer encoders to further improve their efficiency. |
Similar Papers
AdapterDrop: On the Efficiency of Adapters in Transformers (2021.emnlp-main)
Copied to clipboard
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, Iryna Gurevych
| Challenge: | Recent approaches to transformer models are expensive to fine-tune, slow for inference, and have large storage requirements. |
| Approach: | They propose a method to remove adapters from transformer layers during training and inference . they show that AdapterDrop can dynamically reduce computational overhead . |
| Outcome: | The proposed approach reduces computational overhead while maintaining performance over multiple tasks with minimal loss of performance. |
Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained sequence-to-sequence models have advanced the state of the art on text generation tasks. |
| Approach: | They introduce a modular encoder-decoder framework for flexible sequence-to-sequence model compression. |
| Outcome: | The proposed framework can achieve flexible compression ratios from 1.1x to 6x with little to moderate relative performance drop. |
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in transformer quantization have shown remarkable improvement in many Natural Language Processing tasks and beyond. |
| Approach: | They propose a novel quantization scheme for transformers that can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. |
| Outcome: | The proposed methods achieve state-of-the-art results on the GLUE benchmark using BERT, while preserving memory and accuracy. |
Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent improvements in NLP tasks can be attributed to the Transformer model. |
| Approach: | They propose to use parameter-sharing methods to reduce parameter budgets in generative models by using sandwich-style parameter sharing and self-attentive embedding factorization. |
| Outcome: | The proposed model outperforms the current RNN model even with significantly fewer parameters. |
Transformers as Transducers (2025.tacl-1)
Copied to clipboard
| Challenge: | Using finite transducers, we find that transformers can express large classes of (total functional) transductions. |
| Approach: | They extend existing RASP programming language to sequence-to-sequence transductions and introduce two new extensions. |
| Outcome: | The proposed model can express surprisingly large classes of (total functional) transductions. |
DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing interpretability methods have been proposed to interpret the inner workings of Transformer models at different levels of precision and complexity. |
| Approach: | They propose a method to analyze encoder-decoder Transformers by using the decoder module Model Output encoder to cross-attend representations of intermediate encoder activations instead of using the default output. |
| Outcome: | The proposed method maps uninterpretable representations to human-interpreted sequences of words or symbols, shedding new light on the information flow in this popular but understudied class of models. |
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers (2021.findings-acl)
Copied to clipboard
Tianchu Ji, Shraddhan Jain, Michael Ferdman, Peter Milder, H. Andrew Schwartz, Niranjan Balasubramanian
| Challenge: | Recent work shows that attention can be pruned to zeros with minimal loss in accuracy. |
| Approach: | They propose a pruning technique which quantizes attention to a 3-bit format without retraining . they find that 80% of attention values can be pruned to zeros with minimal loss in accuracy . |
| Outcome: | The proposed approach produces only a few unique attention values with minimal loss in accuracy. |
On Sparsifying Encoder Outputs in Sequence-to-Sequence Models (2021.findings-acl)
Copied to clipboard
| Challenge: | Using sequence-to-sequence models, encoder outputs are usually transferred to the decoder for generation, but in this study, encoded outputs can be compressed to shorten the sequence for decoding. |
| Approach: | They propose to use a stochastic gate-based algorithm to mask encoder outputs to shorten the sequence delivered for decoding. |
| Outcome: | The proposed model can be used to shorten encoder outputs to short a sequence . the proposed model yields a speedup of up to 1.65 on document summarization and 1.20 on character-based machine translation tasks. |
On the Benefits of Learning to Route in Mixture-of-Experts Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing Mixture-of-Expert (MoE) models allow us to scale up model sizes while keeping the amount of compute time fixed. |
| Approach: | They propose to use a router to route inputs to experts in a layer to scale up model sizes while keeping the amount of compute time fixed. |
| Outcome: | The proposed model scales up with the help of a router that routes input tokens to experts in a layer and shows that it is more efficient than a non-trainable router. |
Optimizing Deeper Transformers on Small Datasets (2021.acl-long)
Copied to clipboard
Peng Xu, Dhruv Kumar, Wei Yang, Wenjie Zi, Keyi Tang, Chenyang Huang, Jackie Chi Kit Cheung, Simon J.D. Prince, Yanshuai Cao
| Challenge: | a common belief that training deep transformers from scratch requires large datasets is wrong . however, with proper initialization and optimization, the benefits of very deep transformer can carry over to challenging tasks with small datasets. |
| Approach: | They train 48 layers of transformers from pre-trained RoBERTa and 24 relation-aware layers from scratch. |
| Outcome: | The proposed scheme achieves state-of-the-art performance on a text-to-sql parsing benchmark . it uses 24 fine-tuned layers from pre-trained RoBERTa and 24 relation-aware layers from scratch . |