Papers by Yuqiao Wen
f-Divergence Minimization for Sequence-Level Knowledge Distillation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation approaches focus on minimizing a generalized f-divergence function. |
| Approach: | They propose a framework which formulates sequence-level knowledge distillation as minimizing a generalized f-divergence function. |
| Outcome: | The proposed framework outperforms existing methods and reduces intractable divergence to word-level losses. |
An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing benchmark datasets for open-domain dialogue generation are advancing the field . overlapping between training and test sets can cause fake performance . |
| Approach: | They analyze dailyDialog and OpenSubtitles to find out how overlapping can be exploited to obtain fake state-of-the-art performance. |
| Outcome: | The proposed datasets are cleaned and set up for future research. |
Knowledge Distillation for Language Models (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a teacher to a student . this tutorial will cover topics ranging from LLM sequence compression to LLM self-distillation . |
| Approach: | They propose to introduce intermediate-layer matching and prediction matching . they will then present advanced techniques such as reinforcement learning-based KD and multi-teacher distillation . |
| Outcome: | This tutorial aims to provide participants with a comprehensive understanding of the techniques and applications of knowledge distillation for language models. |