Papers by Deepanshu Gupta
EELBERT: Tiny Models through Dynamic Embeddings (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Empirical evaluation shows that the input embedding layer occupies a large portion of the model size. |
| Approach: | They propose an approach for compression of transformer-based models with minimal impact on downstream tasks by replacing the input embedding layer with dynamic embeddable computations. |
| Outcome: | Empirical evaluation shows that the proposed model is 15x smaller (1.2 MB) compared to the traditional model. |