Accelerating BERT Inference for Sequence Labeling via Early-Exit (2021.acl-long)
Copied to clipboard
| Challenge: | Existing early-exit mechanisms are designed for sequence-level tasks, rather than sequence labeling. |
| Approach: | They propose to extend sentence-level early-exit to accelerate inference of PTMs . they propose a token-level mechanism that allows partial tokens to exit early at different layers . |
| Outcome: | The proposed approach can save up to 66%75% inference cost with minimal performance degradation. |
Similar Papers
PCEE-BERT: Accelerating BERT Inference via Patient and Confident Early Exiting (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are the state-of-the-art (SOTA) models for natural language processing (NLP). |
| Approach: | They propose a patient and confident early exiting BERT (PCEE-BERT) that can work with different PLMs and popular model compression methods. |
| Outcome: | The proposed method outperforms existing models on the GLUE benchmarks and achieves different speed-up ratios. |
BADGE: Speeding Up BERT Inference after Deployment via Block-wise Bypasses and Divergence-based Early Exiting (2023.acl-industry)
Copied to clipboard
| Challenge: | Recent years have witnessed the rise of many pre-trained language models (PLMs) such as GPT (Radford et al., 2019) and XLNet (Yang e.t al, 2019). |
| Approach: | They propose a framework which consists of two off-the-shelf methods for improving PLMs’ early exiting. |
| Outcome: | The proposed method can reduce the average latency of pre-trained language models and work with other inference speed-up methods like model pruning. |
LeeBERT: Learned Early Exit for BERT with cross-level optimization (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are resource exhaustive and computationally expensive for industrial scenarios. |
| Approach: | They propose a learning scheme to learn from each other to speed up inference . they ask each exit to learn the weights of different loss terms, instead of learning only from the last layer . |
| Outcome: | The proposed scheme improves state-of-the-art (SOTA) early exit methods for pre-trained models on the GLUE benchmark. |
LECO: Improving Early Exiting via Learned Exits and Comparison-based Exiting Mechanism (2023.acl-srw)
Copied to clipboard
| Challenge: | Recent work on dynamic early exiting has neglected the intermediate exits’ architectural designs. |
| Approach: | They propose a framework for learning exits and COmparison-based early exiting to improve PTMs’ early exit performance. |
| Outcome: | The proposed framework achieves the SOTA performance on multi-exit BERT training and dynamic early exiting on pre-trained models. |
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models such as BERT are notorious for being slow in both training and inference. |
| Approach: | They propose a method to accelerate BERT inference by inserting extra classification layers between each transformer layer of BERT. |
| Outcome: | The proposed method saves up to 40% inference time with minimal degradation in model quality. |
Small and Practical BERT Models for Sequence Labeling (D19-1)
Copied to clipboard
| Challenge: | Existing models for morphosyntactic tagging have focused on building separate models for each language or for a small group of related languages. |
| Approach: | They propose a scheme to train a single multilingual sequence labeling model that is small and fast enough to run on a CPU. |
| Outcome: | The proposed model outperforms state-of-the-art models on low-resource languages and low-level models on codemixed inputs. |
A Global Past-Future Early Exit Method for Accelerating Inference of Pre-trained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to accelerate inference speed of pre-trained language models are limited to local representations of exit layer . current models are associated with large memory requirement and high computational cost, which slow down inference and further encumber the application of PLMs. |
| Approach: | They propose a method to exit early without passing through all inference layers . they take into consideration all the linguistic information embedded in the past layers a global perspective . |
| Outcome: | The proposed method outperforms existing methods by a large margin . it uses linguistic information embedded in the past layers and future features . the proposed method is scalable and cost-effective . |
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods to make exiting decisions are limited to classification tasks . large-scale pre-trained language models such as BERT have brought performance gain but at the cost of heavy computational burden. |
| Approach: | They propose a fine-tuning strategy and a learning-to-exit module to accelerate BERT inference . they propose to make trade-offs between model quality and efficiency by early exiting . |
| Outcome: | The proposed approach improves early exiting for BERT, with better trade-offs . it can be combined with other acceleration methods, and the proposed strategy can be applied to regression tasks. |
TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models (PLMs) are expensive in inference, making them impractical in resource-limited real-world applications. |
| Approach: | They propose a dynamic token reduction approach to accelerate PLMs' inference by adapting the layer number of each token to avoid redundant calculation. |
| Outcome: | The proposed approach speeds up BERT by 2-5 times and improves performance in long-text tasks with less computation. |
CeeBERT: Cross-Domain Inference in Early Exit BERT (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained Language Models suffer in inference latency due to their large size. |
| Approach: | They propose an online learning algorithm that dynamically determines early exits of samples based on the level of confidence observed at intermediate layers. |
| Outcome: | The proposed algorithm can speed up the BERT/ALBERT models by 2 - 3.1 with minimal drop in accuracy. |