Papers by Jemin Park
Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation approaches to NMT often rely on heuristics when deciding which teacher layers to distill from. |
| Approach: | They propose an approach to align student attention heads with their teacher counterparts by heuristics to solve a feature mapping problem. |
| Outcome: | The proposed strategy shows gains of +3.61 and +0.63 BLEU points for WMT-2022 DeDsb and WMT-2014 EnDe compared to baselines. |