| Challenge: | Large Language Models (LLMs) suffer from huge number of parameters, which restricts their deployment on edge devices. |
| Approach: | They propose two methods that share parameters across attention heads to reduce memory usage and reduce performance drop by using coarse-grained weight sharing rules. |
| Outcome: | The proposed methods reuse pre-trained weights without retraining and then share, denoted as PostShare. |
Similar Papers
Cross-layer Attention Sharing for Pre-trained Large Language Models (2026.tacl-1)
Copied to clipboard
Yongyu Mu, Yuzhang Wu, Yuchun Fan, Chenglong Wang, Hengyu Li, Jiali Zeng, Qiaozhi He, Murun Yang, Fandong Meng, Jie Zhou, Tong Xiao, Jingbo Zhu
| Challenge: | Existing studies focus on compressing the Key-Value cache or grouping attention heads, while overlooking redundancy between layers. |
| Approach: | They propose a lightweight substitute for self-attention in well-trained LLMs that uses feed-forward networks to align attention heads between adjacent layers and low-rank matrices to approximate differences in layer-wise attention weights. |
| Outcome: | The proposed model reduces redundancy by sharing weights across layers while maintaining high response quality while reducing redundant calculations within 53% 84% of the total layers. |
Does Self-Attention Need Separate Weights in Transformers? (2025.naacl-industry)
Copied to clipboard
| Challenge: | Experimental results show a 66.53% reduction in parameter size within the attention block and competitive accuracy improvements of 3.55% and 0.89% over symmetric and pairwise attention-based models, respectively. |
| Approach: | They propose a simplified approach where a single weight matrix is used for Keys, Queries, and Values instead of separate matrices for each. |
| Outcome: | The proposed approach outperforms the BERT baseline on GLUE tasks even outperforming the standard BERT model in handling noisy and out-of-domain data. |
IAM: Efficient Inference through Attention Mapping between Different-scale LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a challenge due to their internal reasoning processes. |
| Approach: | They propose an algorithm that can optimize attention matrices by performing attention mapping between small and large LLMs. |
| Outcome: | The proposed framework can reduce KV cache usage by 22.1% and accelerate prefill by 15% without sacrificing performance. |
SharVeT: Similarity-aware Parameter Sharing with Vector-based Tuning for Efficient LLM Compression (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for parameter sharing rely on naive grouping and fail to correct sharing-induced discrepancies. |
| Approach: | They propose a parameter sharing framework that performs similarity-based grouping to ensure accurate sharing and allocates parameters adaptively to preserve diversity within each group. |
| Outcome: | The proposed framework outperforms existing methods, achieving 32.1% lower perplexity and 23.3% higher few-shot reasoning accuracy. |
Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained language models have produced performance gains in various tasks but come with large computational requirements. |
| Approach: | They propose an alternative module that uses only a single shared projection matrix and multiple head embeddings (MHE) they demonstrate that MHE attention is substantially more memory efficient compared to alternative attention mechanisms. |
| Outcome: | The proposed model is more memory efficient compared to the current model while achieving high retention ratio on several downstream tasks. |
A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent studies have shown that sharing key-value (KV) cache across layers is effective in efficient inference of large language models. |
| Approach: | They propose a unified framework that covers several recent methods and their novel variants to investigate cross-layer KV sharing. |
| Outcome: | The proposed framework achieves higher throughput and better performance when reducing the size of the key-value cache by 2 while maintaining competitive performance. |
LongHeads: Multi-Head Attention is Secretly a Long Context Processor (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models struggle to process lengthy inputs due to limited length generalization and attention’s quadratic computational demands. |
| Approach: | They propose a training-free framework that allows each head to attend to important context chunks instead of allowing each head a full sentence . |
| Outcome: | The proposed framework unlocks multi-head attention's untapped potential by allowing each head to attend to important context chunks instead of the full sentence. |
ALPS: Attention Localization and Pruning Strategy for Efficient Adaptation of Large Language Models (2025.findings-acl)
Copied to clipboard
Hao Chen, Haoze Li, Zhiqing Xiao, Lirong Gao, Qi Zhang, Xiaomeng Hu, Ningtao Wang, Xing Fu, Junbo Zhao
| Challenge: | Prior research has focused on optimizing general-purpose large language models to downstream tasks . however, these approaches inherently introduce data dependency, which hinders generalization and reusability. |
| Approach: | They propose an algorithm that localizes the most task-sensitive attention heads and prunes by restricting attention training updates to these heads, thereby reducing alignment costs. |
| Outcome: | The proposed algorithm achieves 2% performance improvement over baselines on three tasks while localizing the most task-sensitive attention heads. |
Probing and Boosting Large Language Models Capabilities via Attention Heads (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to identifying capabilities rely on external signals with limited structural grounding . emergence of specific capabilities remains poorly understood . |
| Approach: | They propose a lightweight approach that links LLM capabilities to internal components by identifying correspondences at the level of attention heads. |
| Outcome: | The proposed approach improves accuracy on MMLU and BBH by 1 to 1.5 points over gradient-based method and 5 to 6 points over other intermediate-state baselines. |
Efficient Sparse Attention needs Adaptive Token Release (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide array of text-centric tasks, however, their ‘large’ scale introduces significant computational and storage challenges, particularly in managing the key-value states of the transformer, which limits their wider applicability. |
| Approach: | They propose to release resources from caches and rebuild key-value states by a lightweight controller module to approximate an ideal top-K sparse attention. |
| Outcome: | The proposed method achieves a significant throughput improvement of 221.8% over full attention and a model with 7 billion tokens. |