Information Aggregation for Multi-Head Attention with Routing-by-Agreement (N19-1)
Copied to clipboard
| Challenge: | Existing studies focus on extracting informative or distinct partial-representations from different subspaces, while few studies have paid attention to the aggregation of the extracted partial-Representations. |
| Approach: | They propose to use a routing-by-agreement algorithm to improve multi-head attention by iteratively updating the proportion of how much a part should be assigned to a whole based on agreement between parts and wholes. |
| Outcome: | The proposed algorithm improves the information aggregation for multi-head attention over the standard linear transformation on linguistic probing and machine translation tasks. |
Similar Papers
Multi-Head Attention with Disagreement Regularization (D18-1)
Copied to clipboard
| Challenge: | Existing methods to encourage diversity among multi-head attention are limited. |
| Approach: | They propose a disagreement regularization term to encourage diversity among attention heads . they validated their approach on EnglishGerman and ChineseEnglish translation tasks . |
| Outcome: | The proposed approach improves translation performance across language pairs on English-German and Chinese-English translation tasks. |
Interpretable Multi-headed Attention for Abstractive Summarization at Controllable Lengths (2020.coling-main)
Copied to clipboard
| Challenge: | Abstractive summarization at controllable lengths is a challenging task in natural language processing . high variance in screen-sizes often require extensive human supervision to perform these modifications. |
| Approach: | They propose a supervised method to construct abstractive summaries of a text document at controllable lengths using an interpretable multi-headed attention mechanism. |
| Outcome: | The proposed method outperforms baselines on two low-resource datasets in English by 14.70%. |
Multi-Level Attention Aggregation for Language-Agnostic Speaker Replication (2024.eacl-short)
Copied to clipboard
| Challenge: | Recent advances in speech synthesis research have enabled the generation of natural-sounding speech, which has prompted a notable shift in TTS research towards the synthesis of speech in the voices of both seen and unseen speakers. |
| Approach: | They propose a multi-level attention aggregation approach that probes and amplifies various speaker-specific attributes in a hierarchical manner. |
| Outcome: | The proposed model achieves substantial speaker similarity and generalizes to out-of-domain (OOD) cases. |
Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated exceptional performance across diverse natural language processing tasks. |
| Approach: | They propose a method for converting multi-head attention into grouped-query attention with any compression ratio of KV heads. |
| Outcome: | The proposed method can compress up to 87.5% KV heads of LLaMA2-7B model and 75% Kv heads of Sheared-LLa MA-1.3B with acceptable performance degradation. |
Mixed Multi-Head Self-Attention for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Recent advances in neural machine translation have been made in the field of multi-head self-attention and there is no explicit mechanism to ensure that different attention heads capture different features. |
| Approach: | They propose a novel multi-head self-attention model which models not only global and local attention but also forward and backward attention in different attention heads. |
| Outcome: | The proposed model improves on WAT17 English-Japanese and IWSLT14 German-English translation tasks without increasing the number of parameters. |
Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translation (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to address speech tasks with a self-attention mechanism are expensive and lead to information loss. |
| Approach: | They propose a Transformer-based model which uses different attention mechanisms on each head to bias the self-attention towards the extraction of more diverse token interactions. |
| Outcome: | The proposed model outperforms baseline models by 0.7 BLEU in the speech task. |
Finding the Pillars of Strength for Multi-Head Attention (2023.acl-long)
Copied to clipboard
| Challenge: | Recent studies have revealed some issues of Multi-Head Attention (MHA) e.g., redundancy and over-parameterization. |
| Approach: | They propose to train attention heads with a self-supervised group constraint to focus on an essential but distinctive feature subset. |
| Outcome: | The proposed method achieves significant performance gains on three well-established tasks while significantly compressing parameters. |
Mixture of Attention Heads: Selecting Attention Heads Per Token (2022.emnlp-main)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) networks have been proposed as an efficient way to scale up model capacity and implement conditional computing. |
| Approach: | They propose a new architecture that combines multi-head attention with the MoE mechanism and a sparsely gated architecture that allows for faster computations. |
| Outcome: | The proposed architecture can scale up the number of attention heads and the number parameters while preserving computational efficiency. |
Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained language models have produced performance gains in various tasks but come with large computational requirements. |
| Approach: | They propose an alternative module that uses only a single shared projection matrix and multiple head embeddings (MHE) they demonstrate that MHE attention is substantially more memory efficient compared to alternative attention mechanisms. |
| Outcome: | The proposed model is more memory efficient compared to the current model while achieving high retention ratio on several downstream tasks. |
Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference (2020.emnlp-main)
Copied to clipboard
Bang An, Jie Lyu, Zhenyi Wang, Chunyuan Li, Changwei Hu, Fei Tan, Ruiyi Zhang, Yifan Hu, Changyou Chen
| Challenge: | Existing studies show that multi-head attention is an effective module in deep neural networks, but there are no explicit mechanisms guaranteeing this property. |
| Approach: | They propose a non-parametric approach that explicitly improves the repulsiveness in multi-head attention and consequently strengthens model’s expressiveness. |
| Outcome: | The proposed approach improves the repulsiveness in multi-head attention and strengthens model’s expressiveness. |