Papers by Nithin Anchuri
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation (2022.findings-naacl)
Copied to clipboard
Md Akmal Haidar, Nithin Anchuri, Mehdi Rezagholizadeh, Abbas Ghaddar, Philippe Langlais, Pascal Poupart
| Challenge: | Existing methods for intermediate layer knowledge distillation suffer from computational burdens and engineering efforts for setting up a proper layer mapping. |
| Approach: | They propose a method where intermediate layers from teacher and student models are randomly selected to be distilled into intermediate layers of student models. |
| Outcome: | The proposed method outperforms state-of-the-art intermediate layer knowledge distillation methods on GLUE tasks and out-of domain test sets. |