| Challenge: | Existing decoder-only transformers fail to preserve initial token-level information in deeper layers. |
| Approach: | They propose a new architecture that incorporates value residual connections in addition to hidden state residuals. |
| Outcome: | The proposed architecture reduces KV cache size by nearly half with only a small performance penalty and can be integrated with other KV-efficient methods. |
Similar Papers
RealFormer: Transformer Likes Residual Attention (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing techniques to create Residual Attention Layer Transformer networks outperform the canonical Transformer on a wide spectrum of tasks. |
| Approach: | They propose a technique to create Residual Attention Layer Transformer networks that outperform the canonical Transformer on a wide spectrum of tasks. |
| Outcome: | The proposed technique outperforms the canonical Transformer on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQUAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP. |
Rewiring the Transformer with Depth-Wise LSTMs (2024.lrec-main)
Copied to clipboard
| Challenge: | Stacking non-linear layers allows deep neural networks to model complicated functions . but residual connections within each layer fail to fuse information from previous layers effectively . |
| Approach: | They propose a Transformer with depth-wise LSTMs connecting cascading Transformer layers and sub-layers. |
| Outcome: | The proposed model improves in English-German / French and multilingual tasks with BLEU. |
ResFormer: All-Time Reservoir Memory for Long Sequence Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models with quadratic time and memory complexity restrict input length . however, analyzing extensive sequential contexts is challenging . |
| Approach: | They propose a neural network architecture that captures contextual dependencies in linear time and a nonlinear readout to model short-term dependencies within sentences. |
| Outcome: | The proposed model outperforms baseline models on EmoryNLP datasets and on IEMOCAP and MultiWOZ datasets. |
Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent improvements in NLP tasks can be attributed to the Transformer model. |
| Approach: | They propose to use parameter-sharing methods to reduce parameter budgets in generative models by using sandwich-style parameter sharing and self-attentive embedding factorization. |
| Outcome: | The proposed model outperforms the current RNN model even with significantly fewer parameters. |
Lipschitz Constrained Parameter Initialization for Deep Transformers (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies show that deep Transformers have difficulty in training even with residual connection and layer normalization. |
| Approach: | They propose a method that leverages the Lipschitz constraint on the initialization of Transformer parameters to ease the optimization difficulties caused by its multi-layer encoder/decoder structure. |
| Outcome: | The proposed model outperforms previous RNN/CNN models but fails to converge with the original computation order. |
Your Transformer is Secretly Linear (2024.acl-long)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | a novel linear characteristic exclusive to transformer decoders is revealed: embedding transformations between sequential layers exhibit almost perfect linearity. |
| Approach: | They propose a cosine-similarity-based regularization to reduce layer linearity in transformer decoders. |
| Outcome: | The proposed method improves performance metrics on Tiny Stories and SuperGLUE but also decreases the linearity of the models. |
Speeding up Transformer Decoding via an Attention Refinement Network (2022.coling-1)
Copied to clipboard
| Challenge: | Extensive experiments on ten WMT machine translation tasks show that the proposed model yields an average of 1.35x faster (with almost no decrease in BLEU) |
| Approach: | They propose a weighted residual network which reconstructs attention by reusing the features across layers. |
| Outcome: | The proposed model is 1.35x faster than the state-of-the-art inference model on translation tasks compared to AAN and SAN models with fewer parameter numbers . |
PartialFormer: Modeling Part Instead of Whole for Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing feed-forward neural networks have significant computational and parametric overhead. |
| Approach: | They propose a parameter-efficient Transformer architecture that utilizes multiple smaller FFNs to reduce parameters and computation while maintaining essential hidden dimensions. |
| Outcome: | The proposed architecture reduces computational and parameter overhead while maintaining essential hidden dimensions. |
RingFormer: Rethinking Recurrent Transformer with Adaptive Level Signals (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Transformers have shown strong performance in processing sequential data, but their parameters are larger . a novel approach to reduce the model parameters while maintaining high performance is proposed . |
| Approach: | They propose a transformer-based model that processes input repeatedly in a circular, ring-like manner. |
| Outcome: | The proposed approach reduces model parameters while maintaining high performance . the proposed approach is validated in the experiments. |
Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention (D19-1)
Copied to clipboard
| Challenge: | Existing methods to improve NLP convergence and computational overhead are limited by stacking more layers. |
| Approach: | They propose a depth-scaled initialization method which reduces parameter variance at initialization and reduces output variance of residual connections to ease gradient back-propagation. |
| Outcome: | The proposed method outperforms the base model on translation tasks with five translation directions while matching the decoding speed of the baseline model. |