Papers by Shengqiang Zhang
Attention Temperature Matters in Abstractive Summarization Distillation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent progress of abstractive text summarization relies on large pre-trained sequence-to-sequence Transformer models, which are computationally expensive. |
| Approach: | They propose to distill large Transformer summarization models into smaller ones with minimal performance loss by manipulating attention temperatures in Transformers. |
| Outcome: | The proposed method outperforms vanilla pseudo-labeling based methods on three summarization datasets and is shorter and more abstractive. |