GAML-BERT: Improving BERT Early Exiting by Gradient Aligned Mutual Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve the early exiting of natural language processing (NLP) are notoriously gigantic and slow in both training and inference. |
| Approach: | They propose a framework for improving the early exiting of BERT by asking each exit to distill knowledge from each other. |
| Outcome: | The proposed framework outperforms the state-of-the-art (SOTA) BERT early exiting methods on the GLUE benchmark. |
Similar Papers
LeeBERT: Learned Early Exit for BERT with cross-level optimization (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are resource exhaustive and computationally expensive for industrial scenarios. |
| Approach: | They propose a learning scheme to learn from each other to speed up inference . they ask each exit to learn the weights of different loss terms, instead of learning only from the last layer . |
| Outcome: | The proposed scheme improves state-of-the-art (SOTA) early exit methods for pre-trained models on the GLUE benchmark. |
BADGE: Speeding Up BERT Inference after Deployment via Block-wise Bypasses and Divergence-based Early Exiting (2023.acl-industry)
Copied to clipboard
| Challenge: | Recent years have witnessed the rise of many pre-trained language models (PLMs) such as GPT (Radford et al., 2019) and XLNet (Yang e.t al, 2019). |
| Approach: | They propose a framework which consists of two off-the-shelf methods for improving PLMs’ early exiting. |
| Outcome: | The proposed method can reduce the average latency of pre-trained language models and work with other inference speed-up methods like model pruning. |
PCEE-BERT: Accelerating BERT Inference via Patient and Confident Early Exiting (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are the state-of-the-art (SOTA) models for natural language processing (NLP). |
| Approach: | They propose a patient and confident early exiting BERT (PCEE-BERT) that can work with different PLMs and popular model compression methods. |
| Outcome: | The proposed method outperforms existing models on the GLUE benchmarks and achieves different speed-up ratios. |
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods to make exiting decisions are limited to classification tasks . large-scale pre-trained language models such as BERT have brought performance gain but at the cost of heavy computational burden. |
| Approach: | They propose a fine-tuning strategy and a learning-to-exit module to accelerate BERT inference . they propose to make trade-offs between model quality and efficiency by early exiting . |
| Outcome: | The proposed approach improves early exiting for BERT, with better trade-offs . it can be combined with other acceleration methods, and the proposed strategy can be applied to regression tasks. |
LECO: Improving Early Exiting via Learned Exits and Comparison-based Exiting Mechanism (2023.acl-srw)
Copied to clipboard
| Challenge: | Recent work on dynamic early exiting has neglected the intermediate exits’ architectural designs. |
| Approach: | They propose a framework for learning exits and COmparison-based early exiting to improve PTMs’ early exit performance. |
| Outcome: | The proposed framework achieves the SOTA performance on multi-exit BERT training and dynamic early exiting on pre-trained models. |
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models such as BERT are notorious for being slow in both training and inference. |
| Approach: | They propose a method to accelerate BERT inference by inserting extra classification layers between each transformer layer of BERT. |
| Outcome: | The proposed method saves up to 40% inference time with minimal degradation in model quality. |
CeeBERT: Cross-Domain Inference in Early Exit BERT (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained Language Models suffer in inference latency due to their large size. |
| Approach: | They propose an online learning algorithm that dynamically determines early exits of samples based on the level of confidence observed at intermediate layers. |
| Outcome: | The proposed algorithm can speed up the BERT/ALBERT models by 2 - 3.1 with minimal drop in accuracy. |
Accelerating BERT Inference for Sequence Labeling via Early-Exit (2021.acl-long)
Copied to clipboard
| Challenge: | Existing early-exit mechanisms are designed for sequence-level tasks, rather than sequence labeling. |
| Approach: | They propose to extend sentence-level early-exit to accelerate inference of PTMs . they propose a token-level mechanism that allows partial tokens to exit early at different layers . |
| Outcome: | The proposed approach can save up to 66%75% inference cost with minimal performance degradation. |
LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference (2026.acl-industry)
Copied to clipboard
Shashank Kapadia, Deep Narayan Mishra, Sujal Reddy Alugubelli, Haoan Wang, Saipraveen Vabbilisetty, Rishi Bhatia, Anupriya Sharma
| Challenge: | Layer-aligned distillation and convergence-based early exit are dominant computational efficiency paradigms for transformer inference. |
| Approach: | They propose a training objective that aligns intermediate student layers to teacher representations and reconciles this incompatibility with standard distillation. |
| Outcome: | The proposed model achieves 1.61 measured wall-clock speedup with 91.9% of samples exiting by layer 7 and 1.80 theoretical layer reduction, where standard distilled models achieve zero effective speedup. |
The Right Tool for the Job: Matching Model and Instance Complexities (2020.acl-main)
Copied to clipboard
| Challenge: | a large increase in the size of NLP models can increase production costs and reduce adoption on real-time devices. |
| Approach: | They propose a modification to contextual representation fine-tuning which allows for an early exit from neural network calculations for simple instances and late exit for hard instances. |
| Outcome: | The proposed method produces models which are up to five times faster than the state of the art while preserving their accuracy. |