HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020.acl-main)
Copied to clipboard
| Challenge: | Extensive experiments on four machine translation tasks demonstrate that HAT can discover efficient models for different hardware (CPU, GPU, IoT device). |
| Approach: | They propose to construct a large design space with arbitrary encoder-decoder attention and heterogeneous layers and then train a SuperTransformer that efficiently produces many SubTransformers with weight sharing. |
| Outcome: | The proposed framework can find efficient models for different hardware (CPU, GPU, IoT device) it achieves 3 speedup, 3.7 smaller size over baseline Transformer; 2.7 speed up, 3.6 smaller sizes over Evolved Transformer with 12,041 less search cost and no performance loss. |
Similar Papers
Transformers: State-of-the-Art Natural Language Processing (2020.emnlp-demos)
Copied to clipboard
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander Rush
| Challenge: | Transformers is an open-source library that aims to open up advances in natural language processing to the wider machine learning community. |
| Approach: | they propose an open-source library that aims to open up advances in machine learning to the wider community. |
| Outcome: | Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. |
Token and Head Adaptive Transformers for Efficient Natural Language Processing (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT have shown significant accuracy improvements on various tasks, but their computational cost and memory footprint are prohibitive. |
| Approach: | They propose to extend Length Adaptive Transformer to extend the model to a token and head pruning scheme to optimize pruning efficiency. |
| Outcome: | The proposed model can compress and accelerate BERT-based models by fine-tuning and a token and head pruning scheme. |
The Green KNIGHT: Green Machine Translation with Knowledge-Distilled, Narrow, Inexpensive, Greedy, Hybrid Transformers (2025.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art neural machine translation models deliver high-quality translations at the expense of high inference latency and energy consumption. |
| Approach: | They propose a hardware-agnostic collection of recipes to optimize translation speed and energy consumption. |
| Outcome: | The Green KNIGHT optimizes translation speed and energy consumption with a moderate trade-off in quality. |
Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model (2022.naacl-industry)
Copied to clipboard
Gongzheng Li, Yadong Xi, Jingzhen Ding, Duan Wang, Ziyang Luo, Rongsheng Zhang, Bai Liu, Changjie Fan, Xiaoxi Mao, Zeng Zhao
| Challenge: | Recent studies show that transformer-based models are effective over many tasks, but they are expensive to deploy in the industrial application. |
| Approach: | They propose a transformer-based inference solution that optimizes kernels for long inputs and large hidden sizes and a flexible CUDA memory manager to reduce the memory footprint when deploying a large model. |
| Outcome: | The proposed solution achieves an average speedup of 1.40-4.20x on the transformer decoder layer with an A100 GPU. |
EFTNAS: Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Depending on the size of transformer-based models, they can be restricted from deployment in resource-constrained environments. |
| Approach: | They propose to combine neural architecture search and network pruning techniques to generate and train weight-sharing super-networks that contain efficient transformer-based models. |
| Outcome: | The proposed model achieves high-performing, high-performance subnetworks on the general language understanding evaluation and the Stanford Question Answering Dataset. |
HyperMixer: An MLP-based Low Cost Alternative to Transformers (2023.acl-long)
Copied to clipboard
Florian Mai, Arnaud Pannatier, Fabio Fehr, Haolin Chen, Francois Marelli, Francois Fleuret, James Henderson
| Challenge: | Existing MLP-based architectures that combine multiple features are expensive and require a lot of training data. |
| Approach: | They propose a simple MLP-based model which allows token mixing by dynamically applying hypernetworks to each feature independently. |
| Outcome: | The proposed model performs better than Transformers and lowers costs in terms of processing time, training data, and hyperparameter tuning. |
HashFormers: Towards Vocabulary-independent Pre-trained Transformers (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding. |
| Approach: | They propose a family of vocabulary-independent pre-trained transformers that support unlimited vocabulary . they propose to map each token to its corresponding embedding by default . |
| Outcome: | The proposed models are more memory efficient than existing models while achieving comparable performance on multiple text classification tasks. |
AdapterHub: A Framework for Adapting Transformers (2020.emnlp-demos)
Copied to clipboard
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych
| Challenge: | AdapterHub framework enables dynamic “stiching-in” of pre-trained adapters for different tasks and languages. |
| Approach: | They propose a framework that allows dynamic "stiching-in" of pre-trained adapters for different tasks and languages. |
| Outcome: | The proposed framework allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages. |
Jump to Conclusions: Short-Cutting Transformers with Linear Transformations (2024.lrec-main)
Copied to clipboard
| Challenge: | Transformer-based language models create hidden representations of inputs at every layer, but only use final-layer representations for prediction. |
| Approach: | They propose a method for casting hidden representations as final representations, bypassing transformer computation in-between. |
| Outcome: | The proposed method produces more accurate predictions from hidden layers across various model scales, architectures, and data distributions. |
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in transformer quantization have shown remarkable improvement in many Natural Language Processing tasks and beyond. |
| Approach: | They propose a novel quantization scheme for transformers that can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. |
| Outcome: | The proposed methods achieve state-of-the-art results on the GLUE benchmark using BERT, while preserving memory and accuracy. |