Papers by Ali Edalati
Kronecker Decomposition for GPT Compression (2022.acl-short)
Copied to clipboard
| Challenge: | GPT is an auto-regressive Transformer-based pre-trained language model . but its huge size can be prohibitive for deploying on low capacity devices . |
| Approach: | They use a Kronecker decomposition technique to compress GPT models . they use ILKD to refine the model on downstream tasks . |
| Outcome: | The proposed model outperforms the existing DistilGPT2 model on language modeling and general language understanding evaluation benchmark tasks. |