Papers by Gyeongman Kim
Distilling Linguistic Context for Language Model Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge distillation is a major technique for deploying vast language models in resource-strapped environments. |
| Approach: | They propose a method that transfers contextual knowledge via Word Relation and Layer Transforming Relation. |
| Outcome: | The proposed method is able to transfer contextual knowledge without restrictions on architectural changes between teacher and student on language understanding tasks. |
PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have raised concerns about inference costs, increasing the need for research into model compression. |
| Approach: | They propose a method that utilizes prompt tuning to enable generative language models to transfer student-friendly knowledge. |
| Outcome: | Extensive experiments on instruction-following datasets show that PromptKD achieves state-of-the-art performance while adding only 0.0007% of the teacher’s parameters as prompts. |