Papers by Vijaykrishnan Narayanan
Token and Head Adaptive Transformers for Efficient Natural Language Processing (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT have shown significant accuracy improvements on various tasks, but their computational cost and memory footprint are prohibitive. |
| Approach: | They propose to extend Length Adaptive Transformer to extend the model to a token and head pruning scheme to optimize pruning efficiency. |
| Outcome: | The proposed model can compress and accelerate BERT-based models by fine-tuning and a token and head pruning scheme. |