Papers by David Talbot
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (P19-1)
Copied to clipboard
| Challenge: | et al., 2017) show that multi-head attention is important for neural machine translation. |
| Approach: | They evaluate the contribution made by individual attention heads to the overall performance of the Transformer model and analyze the roles played by them in the encoder. |
| Outcome: | The proposed pruning method removes the vast majority of heads without affecting performance. |