Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (P19-1)
Copied to clipboard
| Challenge: | et al., 2017) show that multi-head attention is important for neural machine translation. |
| Approach: | They evaluate the contribution made by individual attention heads to the overall performance of the Transformer model and analyze the roles played by them in the encoder. |
| Outcome: | The proposed pruning method removes the vast majority of heads without affecting performance. |
Similar Papers
Differentiable Subset Pruning of Transformer Heads (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent work shows that a large proportion of the heads in a Transformer’s multi-head attention mechanism can be safely pruned away without significantly harming the performance of the model. |
| Approach: | They propose a method that prunes a Transformer's multi-head attention mechanism away without significantly harming its performance. |
| Outcome: | The proposed method improves on natural language inference and machine translation tasks while offering precise control of sparsity level. |
On the weak link between importance and prunability of attention heads (2020.emnlp-main)
Copied to clipboard
| Challenge: | a large fraction of attention heads can be randomly pruned with limited effect on accuracy, a new study finds . a second study finds no advantage in pruning attention heads identified to be important based on the location of a head . |
| Approach: | They examine the importance of pruning attention heads on a Transformer-based model . they find no advantage in pruning attention head positions on the BERT model based on location . |
| Outcome: | The results show that pruning strategies on Transformer and BERT models are not important based on location . the results suggest that interpretation of attention heads does not strongly inform pruning strategies. |
Mixed Multi-Head Self-Attention for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Recent advances in neural machine translation have been made in the field of multi-head self-attention and there is no explicit mechanism to ensure that different attention heads capture different features. |
| Approach: | They propose a novel multi-head self-attention model which models not only global and local attention but also forward and backward attention in different attention heads. |
| Outcome: | The proposed model improves on WAT17 English-Japanese and IWSLT14 German-English translation tasks without increasing the number of parameters. |
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that attention heads learn simple positional patterns . |
| Approach: | They propose to replace all but one attention head of each encoder layer with simple fixed – non-learnable – attentive patterns that are solely based on position and do not require external knowledge. |
| Outcome: | The proposed model improves translation quality and improves BLEU scores by up to 3 points in low-resource scenarios. |
Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks (2021.acl-long)
Copied to clipboard
| Challenge: | Prior research has found that only a few attention heads are important in each mono-lingual NLP task and pruning the remaining heads leads to comparable or improved performance of the model. |
| Approach: | They examine the relative importance of attention heads in Transformer-based models to aid their interpretability in cross-lingual and multi-lingual tasks. |
| Outcome: | The proposed model performs better with the remaining heads pruned than with the other models, the authors show . |
Losing Heads in the Lottery: Pruning Transformer Attention in Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent research shows that attention heads are not confident in their decisions and can be pruned. |
| Approach: | They apply the lottery ticket hypothesis to prune heads in early training . they find that the pruned model is 1.5 times faster at inference . |
| Outcome: | The proposed method is 1.5 times faster at inference, but at the cost of longer training. |
Finding the Pillars of Strength for Multi-Head Attention (2023.acl-long)
Copied to clipboard
| Challenge: | Recent studies have revealed some issues of Multi-Head Attention (MHA) e.g., redundancy and over-parameterization. |
| Approach: | They propose to train attention heads with a self-supervised group constraint to focus on an essential but distinctive feature subset. |
| Outcome: | The proposed method achieves significant performance gains on three well-established tasks while significantly compressing parameters. |
Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translation (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to address speech tasks with a self-attention mechanism are expensive and lead to information loss. |
| Approach: | They propose a Transformer-based model which uses different attention mechanisms on each head to bias the self-attention towards the extraction of more diverse token interactions. |
| Outcome: | The proposed model outperforms baseline models by 0.7 BLEU in the speech task. |
Alleviating the Inequality of Attention Heads for Neural Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | Recent studies show that the attention heads in Transformer are not equal. |
| Approach: | They propose a masking method to mask attention heads in Transformer . they empirically validate the inequality and propose 'head mask' method to avoid bottleneck . |
| Outcome: | The proposed masking method improves translation performance on multiple languages . it can be used to remove a small subset of heads without affecting performance . |
Hard-Coded Gaussian Attention for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has questioned the importance of multi-headed attention in achieving high translation quality. |
| Approach: | They develop a “hard-coded” attention variant without any learned parameters. |
| Outcome: | The proposed model reduces BLEU scores by adding a single learned cross attention head to an otherwise hard-coded Transformer. |