Papers with MHA
A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models are often inefficient for real-world deployment due to expensive inference costs. |
| Approach: | They propose to use knowledge distillation to transfer the knowledge of the original model to a smaller, more efficient student model. |
| Outcome: | The proposed method is the best for multi-lingual and multilingual student architectures. |
Cascaded Head-colliding Attention (2021.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks for natural language processing ignore interactions among different heads, which wastes the capacity of the model. |
| Approach: | They propose a model which explicitly models interactions between attention heads through a hierarchical variational distribution. |
| Outcome: | The proposed model outperforms the baseline model on Wikitext-103 and WMT14 EN-DE on language modeling and translation tasks. |
RoBSA: RoPE-based Blockwise Sparse Multi-head Latent Attention (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have advanced in recent years, scaling up in both parameter count and context length. |
| Approach: | They propose a method to compute attention over a subset of context tokens and to implement token selection in a blockwise manner. |
| Outcome: | The proposed method reduces end-to-end inference latency by up to 2.55x with minimal accuracy loss compared to full attention in long-context scenarios for very large models. |
EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models have shown remarkable results on various NLP tasks. |
| Approach: | They propose to improve the feed-forward network (FFN) in BERT with a higher computational cost than improving the multi-head attention (MHA). |
| Outcome: | The proposed model is 6.9 smaller and 4.4 faster than BERTBASE and has competitive performances on GLUE and SQuAD Benchmarks. |
Enhancing Long-range Dependency with State Space Model and Kolmogorov-Arnold Networks for Aspect-based Sentiment Analysis (2025.coling-main)
Copied to clipboard
| Challenge: | Aspect-based Sentiment Analysis (ABSA) evaluates sentiments toward specific aspects of entities within the text. |
| Approach: | They propose a method to enhance long-range dependencies between aspect and opinion words in ABSA by combining attention mechanisms with a syntax-based Graph Convolutional Network and a Mamba-Transformer module. |
| Outcome: | The proposed model outperforms state-of-the-art models on three benchmark datasets. |
Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptation (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Inference optimizations such as quantization, pruning, format and datatype conversion, model export, and serialization can lead to functional degradations in language model task performance. |
| Approach: | They propose a lightweight and dataset-agnostic method to recover model accuracies from quantization, pruning, format and datatype conversion, model export, and serialization errors. |
| Outcome: | The proposed method recovers model accuracies by 5-17% on MHA and GQA models. |
BlockPruner: Fine-grained Pruning for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have significant computational and memory costs associated with training and inference. |
| Approach: | They propose a training-free structured pruning approach that targets redundancies in MHA and MLP blocks. |
| Outcome: | The proposed pruning approach achieves more granular and effective pruning compared to state-of-the-art pruning methods. |
Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated exceptional performance across diverse natural language processing tasks. |
| Approach: | They propose a method for converting multi-head attention into grouped-query attention with any compression ratio of KV heads. |
| Outcome: | The proposed method can compress up to 87.5% KV heads of LLaMA2-7B model and 75% Kv heads of Sheared-LLa MA-1.3B with acceptable performance degradation. |
When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved impressive performance across NLP tasks. |
| Approach: | They propose to use long-context SFT to improve short-contemporary performance . they also decouple and analyze two key components, Multi-Head Attention and Feed-Forward Network . |
| Outcome: | The proposed model improves short-context performance, contrary to pretraining. |
Generating Fluent Adversarial Examples for Natural Languages (P19-1)
Copied to clipboard
| Challenge: | Current methods for building adversarial attackers for NLP are inefficient as the gradient is discarded. |
| Approach: | They propose an adversarial attacker which performs Metropolis-Hastings sampling with the guidance of gradients to solve these problems. |
| Outcome: | The proposed algorithm outperforms the baseline model on attacking capability on IMDB and SNLI. |
Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations (2024.findings-acl)
Copied to clipboard
| Challenge: | Structured pruning is a feasible solution for end-side LLM deployment . however, achieving a high compression ratio for scaled-up LLMs remains a challenge . |
| Approach: | They propose a task-agnostic structured pruning approach coupled with a compact Transformer architecture to prune LLMs into an intra-module low-rank architecture. |
| Outcome: | The proposed approach reduces transitional activations inside multi-head attention (MHA) and multi-layer perceptron (MLP) modules while preserving inter-module activations sensitive to perturbations. |
Transforming Visual Scene Graphs to Image Captions (2023.acl-long)
Copied to clipboard
Xu Yang, Jiawei Peng, Zihua Wang, Haiyang Xu, Qinghao Ye, Chenliang Li, Songfang Huang, Fei Huang, Zhangzikang Li, Yu Zhang
| Challenge: | Existing approaches to generate captions using image captioning are based on multi-head attention (MHA) |
| Approach: | They propose to transform scene graphs into more descriptive captions by using multi-head attention to build a Graph Neural Network (GNN) . they construct a Mixture-of-Expert (MOE)-based decoder where each expert is built on MHA for discriminating the graph embeddings to generate different kinds of words. |
| Outcome: | The proposed framework can generate captions from multiple visual features and objects . it is based on a mixture-of-expert (MOE)-based decoder based upon MHA . |
Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained language models have produced performance gains in various tasks but come with large computational requirements. |
| Approach: | They propose an alternative module that uses only a single shared projection matrix and multiple head embeddings (MHE) they demonstrate that MHE attention is substantially more memory efficient compared to alternative attention mechanisms. |
| Outcome: | The proposed model is more memory efficient compared to the current model while achieving high retention ratio on several downstream tasks. |
Finding the Pillars of Strength for Multi-Head Attention (2023.acl-long)
Copied to clipboard
| Challenge: | Recent studies have revealed some issues of Multi-Head Attention (MHA) e.g., redundancy and over-parameterization. |
| Approach: | They propose to train attention heads with a self-supervised group constraint to focus on an essential but distinctive feature subset. |
| Outcome: | The proposed method achieves significant performance gains on three well-established tasks while significantly compressing parameters. |
Multi-matrix Factorization Attention (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing variants for Multi-Head Attention (MHA) fail to maintain strong performance under stringent Key-Value cache (KV cache) constraints. |
| Approach: | They propose to use multi-matrix factorization attention and MFA-Key-reuse attention architectures to increase model capacity under tight KV cache constraints. |
| Outcome: | The proposed architecture outperforms existing methods while reducing KV cache usage by 56% and 93.7% in large-scale experiments. |
Dissecting Persona-Driven Reasoning in Language Models via Activation Patching (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit remarkable versatility in adopting diverse personas. |
| Approach: | They examine how assigning a persona influences a model’s reasoning on an objective task by activation patching . |
| Outcome: | The early Multi-Layer Perceptron (MLP) layers attend to syntactic structure of input and process its semantic content. |
VeriLocc: End-to-End Cross-Architecture Register Allocation via LLM (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern GPUs evolve rapidly, yet production compilers still rely on hand-crafted register allocation heuristics that require substantial re-tuning for each hardware generation. |
| Approach: | They propose a framework that combines large language models with compiler techniques to enable generalizable and verifiable register allocation across GPU architectures. |
| Outcome: | The framework outperforms expert-tuned libraries on matrix multiplication and multi-head attention and a verifier-guided regeneration loop to ensure correctness. |
Towards Economical Inference: Enabling DeepSeek’s Multi-Head Latent Attention in Any Transformer-based LLMs (2025.acl-long)
Copied to clipboard
Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Shenlixing Shenlixing, Chenzhan Chenzhan, Xipeng Qiu, Qi Zhang, Tao Gui
| Challenge: | Multi-head Latent Attention (MLA) is an innovative architecture designed to ensure efficient and economical inference by significantly compressing the Key-Value (KV) cache into a latent vector. |
| Approach: | They propose a data-efficient fine-tuning method for transitioning from MHA to MLA using a latent vector cache. |
| Outcome: | The proposed architecture reduces the KV cache size of Llama2-7B by 92.19%, with only 1% drop in LongBench performance. |
A BERTology View of LLM Orchestrations: Token- and Layer-Selective Probes for Efficient Single-Pass Classification (2026.acl-long)
Copied to clipboard
| Challenge: | Modern LLM deployments are rarely a single model in isolation. |
| Approach: | They propose a model that reuses computation already paid for by the serving LLM . they instantiate a template with pooling, a scoring-attention gate, and a downcast multi-head self-attention probe . |
| Outcome: | The proposed model improves safety and sentiment benchmarks on dense and mixture-of-experts architectures while preserving near-serving latency. |