Papers by Alan Do-Omri
Do we need Label Regularization to Fine-tune Pre-trained Language Models? (2023.eacl-main)
Copied to clipboard
Ivan Kobyzev, Aref Jafari, Mehdi Rezagholizadeh, Tianda Li, Alan Do-Omri, Peng Lu, Pascal Poupart, Ali Ghodsi
| Challenge: | Knowledge Distillation (KD) is a label regularization technique that can be replaced with lighter teacher-free variants such as the label-smoothing technique. |
| Approach: | They propose to use knowledge distillation to train student models by deploying the teacher network during training. |
| Outcome: | The proposed method can be replaced with lighter teacher-free variants on PLMs with more than 600 distinct trials and ran each configuration five times. |