Papers with teacher
Noisy Self-Knowledge Distillation for Text Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to summarize text using a single reference and noisy datasets are ill-suited to summarising on single reference datasets. |
| Approach: | They propose to use self-knowledge distillation to improve text summarization by generating smoothed labels for students and teachers to reduce model uncertainty. |
| Outcome: | The proposed framework improves on pretrained and non-pretrained models on three benchmarks. |
Self-Training with Weak Supervision (2021.naacl-main)
Copied to clipboard
| Challenge: | State-of-the-art deep neural networks require large amounts of labeled training data that is expensive to obtain or not available for many tasks. |
| Approach: | They propose a weak supervision framework that leverages all available data for a given task . they leverage task-specific unlabeled data through self-training with a model that predicts pseudo-labels for instances that may not be covered by weak rules . |
| Outcome: | The proposed framework improves on state-of-the-art datasets on six benchmark tasks. |
Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for training language-vision models only consider monolingual learning, especially English. |
| Approach: | They propose to extend an English language-vision model into a multilingual and code-mixed model by using knowledge distillation techniques. |
| Outcome: | The proposed model outperforms existing models on multilingual and code-mixed VQA datasets on eleven languages. |
How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies show that intermediate layer distillation (ILD) objectives improve model compression, but a comprehensive evaluation of distillation objectives in both task-specific and task-agnostic settings is lacking. |
| Approach: | They propose to use knowledge distillation to improve model compression by transferring knowledge from one model to another. |
| Outcome: | The proposed framework improves on the task of QNLI with lower teacher layers and higher teacher layers. |
Learning a Matching Model with Co-teaching for Multi-turn Response Selection in Retrieval-based Dialogue Systems (P19-1)
Copied to clipboard
| Challenge: | Existing methods for learning a robust matching model from noisy training data are retrieval-based or generation-based. |
| Approach: | They propose a general co-teaching framework that learns matching models from noisy training data. |
| Outcome: | The proposed learning framework can improve existing models on two public data sets. |
Distilling Calibrated Knowledge for Stance Detection (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for stance detection ignore meaningful signals among categories offered by hard labels. |
| Approach: | They propose to use knowledge distillation to calibrate teacher predictions in each generation step. |
| Outcome: | The proposed method can calibrate teacher predictions in each generation step and improves stance detection accuracy. |
Patient Knowledge Distillation for BERT Model Compression (D19-1)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT have proven to be highly effective for natural language processing tasks, but the high demand for computing resources hinders their application in practice. |
| Approach: | They propose to compress an original large model (teacher) into an equally-effective lightweight shallow network (student) Empirically, this translates into improved results on multiple NLP tasks with a significant gain in training efficiency, without sacrificing model accuracy. |
| Outcome: | The proposed model reduces the computational cost of training models using the teacher model into a lightweight shallow network. |
Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training (D19-1)
Copied to clipboard
| Challenge: | Current weakly supervised approaches for learning aspect classifiers require many fine-grained aspect labels, which are labor-intensive to obtain. |
| Approach: | They propose a weakly supervised approach that leverages seed words for aspect detection . they propose supervised student-teacher approach that uses teacher to train student models . |
| Outcome: | The proposed approach outperforms previous weakly supervised approaches by 14.1 F1 points on average in six domains of product reviews and six multilingual datasets of restaurant reviews. |
SmartAD: Capacity-Aligned Agent Distillation for Small Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) show strong reasoning and decision-making ability, but their high inference cost motivates transferring agentic skills to small language models. |
| Approach: | They propose a capacity-aligned agent distillation framework that trains SLMs on full reason–act–observe trajectories from a tool-using teacher. |
| Outcome: | The proposed framework outperforms all baselines on multi-hop QA and math benchmarks with 1.5B and 3B models. |