Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model

Challenge:	Recent studies show that transformer-based models are effective over many tasks, but they are expensive to deploy in the industrial application.
Approach:	They propose a transformer-based inference solution that optimizes kernels for long inputs and large hidden sizes and a flexible CUDA memory manager to reduce the memory footprint when deploying a large model.
Outcome:	The proposed solution achieves an average speedup of 1.40-4.20x on the transformer decoder layer with an A100 GPU.

Similar Papers

Overview

Challenge

Approach

Outcome

LightSeq: A High Performance Inference Library for Transformers (2021.naacl-industry)

Copied to clipboard

Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, Lei Li

Language Models

Challenge:	Existing inference frameworks for natural language processing are not the best choice for online service of sequence processing problems.
Approach:	They propose a highly efficient inference library for Transformer models that includes GPU optimization techniques to streamline computation and reduce memory footprint.
Outcome:	The proposed library achieves 14x speedup compared with TensorFlow and 1.4x speed up compared to a concurrent CUDA implementation.

Bag of Tricks for Optimizing Transformer Efficiency (2021.findings-emnlp)

Copied to clipboard

Ye Lin, Yanyang Li, Tong Xiao, Jingbo Zhu

Semantic Text Processing

Challenge:	Improving Transformer efficiency has become increasingly attractive in recent years.
Approach:	They propose to combine pruning, quantization, new architectures and training strategies to improve Transformer efficiency.
Outcome:	The proposed methods improve the inference efficiency of a strong Transformer system by 3.80x on CPU and 2.52x on GPU.

Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2021.tacl-1)

Copied to clipboard

Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, Marianne Winslett

Language Models

Challenge:	Popular pre-trained Transformers have improved performance for various NLP tasks by sizable margins, but are too resource-hungry and computation-intensive to suit low-capacity devices or applications with strict latency requirements.
Approach:	They present a literature review of the compression of Transformers, focusing on the popular BERT model, which has attracted considerable research attention.
Outcome:	The proposed models improve Sentiment analysis, paraphrase detection, machine reading comprehension, question answering, text summarization, and other tasks by sizable margins.

AdapterDrop: On the Efficiency of Adapters in Transformers (2021.emnlp-main)

Copied to clipboard

Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, Iryna Gurevych

Language Models

Challenge:	Recent approaches to transformer models are expensive to fine-tune, slow for inference, and have large storage requirements.
Approach:	They propose a method to remove adapters from transformer layers during training and inference . they show that AdapterDrop can dynamically reduce computational overhead .
Outcome:	The proposed approach reduces computational overhead while maintaining performance over multiple tasks with minimal loss of performance.

Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024.findings-acl)

Copied to clipboard

Lukas Garbaciauskas, Max Ploner, Alan Akbik

Language Models

Challenge:	Existing models for NLP tasks require fine-tuning, but it is computationally infeasible.
Approach:	They propose an approach that inexpensively estimates a ranking of the expected performance of a given set of transformer language models for a specific task.
Outcome:	The proposed model improves the Pearson correlation coefficient between the true model ranks and the estimate.

TranSFormer: Slow-Fast Transformer for Machine Translation (2023.findings-acl)

Copied to clipboard

Bei Li, Yi Jing, Xu Tan, Zhen Xing, Tong Xiao, Jingbo Zhu

Language Models

Challenge:	Prior work has focused on treating subwords as basic units in developing such systems.
Approach:	They propose a slow-fast two-stream learning model that uses a “slow” branch to deal with subword sequences and a "fast" branch to cope with longer character sequences.
Outcome:	The proposed model shows consistent BLEU improvements (larger than 1 BLUE point) on several machine translation benchmarks.

Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model (2020.coling-main)

Copied to clipboard

Sungrae Park, Geewook Kim, Junyeop Lee, Junbum Cha, Ji-Hoon Kim, Hwalsuk Lee

Semantic Text Processing

Challenge:	Existing approaches to character-level language modeling have suffered from high learning complexity caused by inherently long character sequences.
Approach:	They propose a method that efficiently reduces the computational cost and parameter size of Transformer by splitting feature space into multiple groups, factorizing the calculation paths, and reducing computations for the group interaction.
Outcome:	The proposed model reduces the computational cost and parameter size of Transformer on two benchmark tasks, enwik8 and text8, and it performs well.

The NLP Task Effectiveness of Long-Range Transformers (2023.eacl-main)

Copied to clipboard

Guanghui Qin, Yukun Feng, Benjamin Van Durme

Language Models

Challenge:	Existing benchmarks on long-range attention models have not been sufficient to develop efficient Transformers and their practical application on complex NLP tasks.
Approach:	They propose to benchmark 7 Transformer variants on 5 difficult NLP tasks and 7 datasets to examine their capacity for long-range attention.
Outcome:	The proposed models have advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error.

HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020.acl-main)

Copied to clipboard

Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, Song Han

Language Models

Challenge:	Extensive experiments on four machine translation tasks demonstrate that HAT can discover efficient models for different hardware (CPU, GPU, IoT device).
Approach:	They propose to construct a large design space with arbitrary encoder-decoder attention and heterogeneous layers and then train a SuperTransformer that efficiently produces many SubTransformers with weight sharing.
Outcome:	The proposed framework can find efficient models for different hardware (CPU, GPU, IoT device) it achieves 3 speedup, 3.7 smaller size over baseline Transformer; 2.7 speed up, 3.6 smaller sizes over Evolved Transformer with 12,041 less search cost and no performance loss.

An Architecture for Accelerated Large-Scale Inference of Transformer-Based Language Models (2021.naacl-industry)

Copied to clipboard

Amir Ganiev, Colton Chapin, Anderson De Andrade, Chen Liu

Language Models

Challenge:	a recent paper shows that attention-based language models can be used to train, evaluate, and perform inference on predictive models.
Approach:	They develop a machine learning architecture that can scale to a large volume of requests . they use a BERT model that is fine-tuned for emotion analysis .
Outcome:	The proposed architecture can scale to a large volume of requests with a minimum of 96 hours of running time.