Papers by Vasileios Lioutas
Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Word-embeddings are vital components of natural language processing (NLP) but they consume a lot of memory which poses a challenge for edge deployment. |
| Approach: | They propose an embedding compression method based on matrix decomposition and knowledge distillation that initializes weights of pre-trained word-embeddings and fine-tunes end-to-end. |
| Outcome: | The proposed method has higher BLEU score on translation and lower perplexity on language modeling compared to complex, difficult to tune methods. |
Towards Zero-Shot Knowledge Distillation for Natural Language Processing (2021.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions. |
| Approach: | They propose to use teacher training data for model compression . they investigate six tasks and find they can achieve between 75% and 92% of the teacher’s classification score while compressing the model 30 times. |
| Outcome: | The proposed solution achieves between 75% and 92% of the teacher’s classification score while compressing the model 30 times. |
MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that the trillion parameter mark for pre-trained language models is not achievable without a change in training paradigm. |
| Approach: | They propose a text-based adversarial training algorithm which enhances the performance of knowledge distillation by maximizing the divergence between teacher and student logits. |
| Outcome: | The proposed algorithm outperforms competing adversarial learning and data augmentation baselines on the GLUE benchmark. |