Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to speed up inference latency have shown performance degradation caused by a state copying mechanism or numerous exit paths. |
| Approach: | They propose a framework that allocates adaptive computation paths for each token based on the complexity of generating the subsequent token. |
| Outcome: | The proposed framework outperforms existing frameworks on extensive generation tasks. |
Similar Papers
A Global Past-Future Early Exit Method for Accelerating Inference of Pre-trained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to accelerate inference speed of pre-trained language models are limited to local representations of exit layer . current models are associated with large memory requirement and high computational cost, which slow down inference and further encumber the application of PLMs. |
| Approach: | They propose a method to exit early without passing through all inference layers . they take into consideration all the linguistic information embedded in the past layers a global perspective . |
| Outcome: | The proposed method outperforms existing methods by a large margin . it uses linguistic information embedded in the past layers and future features . the proposed method is scalable and cost-effective . |
Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to generate draft tokens in large language models are expensive and resource-intensive. |
| Approach: | They propose an approach to generate draft tokens using a segment of the LLM and a self-distillation method to enhance the quality of draft token. |
| Outcome: | The proposed approach generates draft tokens using a segment of the LLM and a self-distillation method to improve quality and speed up generation. |
FREE: Fast and Robust Vision Language Models with Early Exits (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have shown remarkable performance improvements in Vision-language tasks, but their large size poses challenges for real-world applications. |
| Approach: | They propose an adversarial approach to train exit classifiers in Vision-Language Models that uses a transformer layer and a classifier to perform input-adaptive inference. |
| Outcome: | The proposed approach speeds up inference speed with minimal drop in performance by 1.51 while retaining comparable performance. |
River-LLM: Large Language Model Seamless Exit Based on KV Share (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to reduce latency and speed up early exits are costly and impose significant cost and energy consumption. |
| Approach: | They propose a lightweight KV-Shared Exit River framework that allows the backbone’s missing KV cache to be naturally generated and preserved during the exit process. |
| Outcome: | The proposed framework achieves 1.71 to 2.16 speedup while maintaining high generation quality. |
A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation (2022.findings-acl)
Copied to clipboard
Tianxiang Sun, Xiangyang Liu, Wei Zhu, Zhichao Geng, Lingling Wu, Yilong He, Yuan Ni, Guotong Xie, Xuanjing Huang, Xipeng Qiu
| Challenge: | Existing methods to measure instance difficulty use generalization and threshold-tuning . a new approach to learn to exit is based on hash functions to assign tokens to a fixed exiting layer. |
| Approach: | They propose a Hash-based Early Exiting approach that replaces learn-to-exit modules with hash functions to assign each token to a fixed exiting layer. |
| Outcome: | The proposed approach improves on learning to exit and predicting instance difficulty. |
BADGE: Speeding Up BERT Inference after Deployment via Block-wise Bypasses and Divergence-based Early Exiting (2023.acl-industry)
Copied to clipboard
| Challenge: | Recent years have witnessed the rise of many pre-trained language models (PLMs) such as GPT (Radford et al., 2019) and XLNet (Yang e.t al, 2019). |
| Approach: | They propose a framework which consists of two off-the-shelf methods for improving PLMs’ early exiting. |
| Outcome: | The proposed method can reduce the average latency of pre-trained language models and work with other inference speed-up methods like model pruning. |
Early Exit with Disentangled Representation and Equiangular Tight Frame (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing early exit paradigm relies on training parametrical internal classifiers to complete specific tasks. |
| Approach: | They propose a method to decouple two distinct types of representation and introduce a non-parametric tight frame classifier for improvement. |
| Outcome: | Experiments on monolingual and multilingual tasks show that the proposed method improves over existing methods. |
SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration (2026.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models. |
| Approach: | They propose a self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision and adaptively bounds speculation length based on token-wise decoding difficulty. |
| Outcome: | The proposed framework suppresses spurious confidence and bounds speculation length based on token-wise decoding difficulty. |
Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing autoregressive models generate tokens sequentially and are memory-bound, resulting in a memory-based inference stage that is memory-limited. |
| Approach: | They propose an approach to accelerate the inference speed of large language models with billions of parameters by integrating semi-autoregressive inference and speculative decoding capabilities. |
| Outcome: | The proposed approach has demonstrated inference speedups of 2.7x-4.0x on humanEval-X while maintaining output quality. |
DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Encoder-decoder transformer models suffer from high inference latency due to auto-regressive decoding . Typically, the decoder takes up most of the latency because of the auto-decoding - a problem that is not solved by the current model. |
| Approach: | They propose an approach to perform Dynamic Early Exit on Decoder to reduce inference latency by 20%-74% by using a multi-exit encoder-decoder transformer model trained with deep supervision. |
| Outcome: | The proposed model reduces inference latency by 20%-74% with comparable or even higher accuracy compared to baseline models. |