Papers by Ligeng Zhu
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020.acl-main)
Copied to clipboard
| Challenge: | Extensive experiments on four machine translation tasks demonstrate that HAT can discover efficient models for different hardware (CPU, GPU, IoT device). |
| Approach: | They propose to construct a large design space with arbitrary encoder-decoder attention and heterogeneous layers and then train a SuperTransformer that efficiently produces many SubTransformers with weight sharing. |
| Outcome: | The proposed framework can find efficient models for different hardware (CPU, GPU, IoT device) it achieves 3 speedup, 3.7 smaller size over baseline Transformer; 2.7 speed up, 3.6 smaller sizes over Evolved Transformer with 12,041 less search cost and no performance loss. |