Intermediate Layer Distillation with the Reused Teacher Classifier: A Study on the Importance of the Classifier of Attention-based Models

Challenge:	Existing methods underestimate the importance of utilizing the teacher's discriminative classifier and face challenges in establishing proper layer mappings.
Approach:	They propose to reuse pre-trained teacher classifiers to improve student performance . they use projectors to match hidden size of the teacher model to student .
Outcome:	The proposed method outperforms existing methods on 97.7% of the teacher BERT base without additional trainable parameters.

Similar Papers

Overview

Challenge

Approach

Outcome

Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective (2023.findings-eacl)

Copied to clipboard

Jongwoo Ko, Seungjoon Park, Minchan Jeong, Sukjin Hong, Euijai Ahn, Du-Seong Chang, Se-Young Yun

Language Models

Challenge:	Existing methods for knowledge distillation (KD) are prone to overfitting to training datasets . recent advances in NLP have shown that using PLMs such as BERT and RoBERTa on downstream tasks is effective.
Approach:	They propose a consistency-regularized knowledge distillation method which mitigates overfitting of existing methods.
Outcome:	The proposed method outperforms existing methods on the GLUE benchmark and synthetic datasets.

Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation (2021.emnlp-main)

Copied to clipboard

Yimeng Wu, Mehdi Rezagholizadeh, Abbas Ghaddar, Md Akmal Haidar, Ali Ghodsi

Language Models

Challenge:	Existing methods for intermediate layer matching are limited due to huge over-parameterization .
Approach:	They propose to match intermediate layers of teacher and student in output space via attention-based layer projection.
Outcome:	The proposed method outperforms existing methods on GLUE tasks.

BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover’s Distance (2020.emnlp-main)

Copied to clipboard

Jianquan Li, Xiaokang Liu, Honghong Zhao, Ruifeng Xu, Min Yang, Yaohong Jin

Language Models

Challenge:	Pre-trained language models have been proposed and applied to many NLP tasks, yielding state-of-the-art performance, but high storage and computational costs obstruct them to be effectively deployed on resource-constrained devices and real-time applications.
Approach:	They propose a BERT distillation method which allows each intermediate student layer to learn from any intermediate teacher layers.
Outcome:	The proposed method can learn from different teacher layers adaptively for different NLP tasks.

How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives (2023.acl-short)

Copied to clipboard

Xinpeng Wang, Leonie Weissweiler, Hinrich Schütze, Barbara Plank

Language Models

Challenge:	Recent studies show that intermediate layer distillation (ILD) objectives improve model compression, but a comprehensive evaluation of distillation objectives in both task-specific and task-agnostic settings is lacking.
Approach:	They propose to use knowledge distillation to improve model compression by transferring knowledge from one model to another.
Outcome:	The proposed framework improves on the task of QNLI with lower teacher layers and higher teacher layers.

Contrastive Distillation on Intermediate Representations for Language Model Compression (2020.emnlp-main)

Copied to clipboard

Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, Jingjing Liu

Semantic Text Processing

Challenge:	Existing methods to compress language models use a simple L_2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one.
Approach:	They propose a method that uses knowledge distillation to distill knowledge through intermediate layers of the teacher via a contrastive objective.
Outcome:	The proposed method outperforms state-of-the-art methods on the GLUE benchmark.

Patient Knowledge Distillation for BERT Model Compression (D19-1)

Copied to clipboard

Siqi Sun, Yu Cheng, Zhe Gan, Jingjing Liu

Semantic Text Processing

Challenge:	Pre-trained language models such as BERT have proven to be highly effective for natural language processing tasks, but the high demand for computing resources hinders their application in practice.
Approach:	They propose to compress an original large model (teacher) into an equally-effective lightweight shallow network (student) Empirically, this translates into improved results on multiple NLP tasks with a significant gain in training efficiency, without sacrificing model accuracy.
Outcome:	The proposed model reduces the computational cost of training models using the teacher model into a lightweight shallow network.

TinyBERT: Distilling BERT for Natural Language Understanding (2020.findings-emnlp)

Copied to clipboard

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu

Language Models

Challenge:	Pre-trained language models are computationally expensive and difficult to efficiently execute on resource-restricted devices.
Approach:	They propose a Transformer distillation method that performs Transformer distillations at pre-training and task-specific learning stages.
Outcome:	The proposed method accelerates inference and reduces model size while maintaining accuracy.

RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation (2022.findings-naacl)

Copied to clipboard

Md Akmal Haidar, Nithin Anchuri, Mehdi Rezagholizadeh, Abbas Ghaddar, Philippe Langlais, Pascal Poupart

Semantic Text Processing

Challenge:	Existing methods for intermediate layer knowledge distillation suffer from computational burdens and engineering efforts for setting up a proper layer mapping.
Approach:	They propose a method where intermediate layers from teacher and student models are randomly selected to be distilled into intermediate layers of student models.
Outcome:	The proposed method outperforms state-of-the-art intermediate layer knowledge distillation methods on GLUE tasks and out-of domain test sets.

Pre-training Distillation for Large Language Models: A Design Space Exploration (2025.acl-long)

Copied to clipboard

Hao Peng, Xin Lv, Yushi Bai, Zijun Yao, Jiajie Zhang, Lei Hou, Juanzi Li

Semantic Text Processing

Challenge:	Knowledge distillation (KD) aims to transfer knowledge from a large teacher model to a smaller student model for model compression.
Approach:	They extend knowledge distillation to the pre-training phase of large language models . they first conduct an experiment using a teacher LLM to distill a 1.9B student LLM .
Outcome:	The proposed model can be used to distill a 1.9B student model using a teacher LLM.

Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers (2020.emnlp-main)

Copied to clipboard

Yimeng Wu, Peyman Passban, Mehdi Rezagholizadeh, Qun Liu

Language Models

Challenge:	Existing knowledge distillation techniques are not suitable for deep learning tasks due to memory constraints.
Approach:	They propose to combine knowledge from a large teacher network into a student network (S) they propose to use a combinatorial mechanism to inject layer-level supervision from T to S .
Outcome:	The proposed model outperforms existing models in PortugueseEnglish, TurkishEnglish and EnglishGerman directions and students trained using it have 50% fewer parameters and can deliver comparable results to 12-layer teachers.