| Challenge: | Existing distillation methods that focus on encoder-only LMs fail to handle the distillation of encoder decoder LM. |
| Approach: | They propose a method that finetunes pretrained language models (LMs) they propose 'MiniEnD' that allows for task-agnostic distillation of LMs. |
| Outcome: | The proposed distillation method is generally effective and competitive compared to other alternatives. |
Similar Papers
A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models are often inefficient for real-world deployment due to expensive inference costs. |
| Approach: | They propose to use knowledge distillation to transfer the knowledge of the original model to a smaller, more efficient student model. |
| Outcome: | The proposed method is the best for multi-lingual and multilingual student architectures. |
Distillation of encoder-decoder transformers for sequence labelling (2023.findings-eacl)
Copied to clipboard
| Challenge: | despite the strong trend in NLP to explore the use of large language models, there is still limited work evaluating prompting and decoding mechanisms for SL tasks. |
| Approach: | They propose a hallucination-free framework for sequence tagging that is especially suited for distillation. |
| Outcome: | The proposed framework performs well across multiple sequence labelling datasets and in a few-shot learning scenario. |
Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing methods for transferring knowledge from BERT into a model with large parameters are not efficient due to their large-scale and high computational cost. |
| Approach: | They propose a sentence representation approximating oriented distillation framework that can distill pre-trained BERT into a simple LSTM based model without specifying tasks. |
| Outcome: | The proposed model outperforms other distillation methods and larger models on multiple NLP tasks with efficiency well-improved. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023.findings-acl)
Copied to clipboard
Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister
| Challenge: | Deploying large language models (LLMs) is difficult because they are memory inefficient and compute-intensive for practical applications. |
| Approach: | They propose a mechanism that fine tunes or distills small models that outperform LLMs . they use human labels to fine tune models or LLM-generated labels to train models . |
| Outcome: | The proposed method outperforms LLMs by using fewer training examples compared to few-shot prompted models using substantially smaller model sizes. |
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains (2021.findings-acl)
Copied to clipboard
| Challenge: | Large pre-trained models suffer from domain shift and are not optimal for specific domains. |
| Approach: | They propose a general approach to developing small, fast and effective pretrained models for specific domains by adapting off-the-shelf general pretrained model and performing task-agnostic knowledge distillation in target domains. |
| Outcome: | The proposed approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3 smaller and 5.1 faster than the BRT BASE. |
Decoupling Generalization and Adaptation in Meta-Learning for Large Language Models (2026.acl-short)
Copied to clipboard
| Challenge: | Adapting large language models to specific downstream tasks requires multi-step fine-tuning with substantial training data, incurring significant computational overhead. |
| Approach: | They propose a framework that separates learning generalizable initializations and adaptation through dedicated parameter spaces. |
| Outcome: | The proposed framework outperforms existing meta-learning and standard multi-task baselines on common-sense reasoning, mathematics, logic, medical and coding benchmarks. |
Distilling LLM Reasoning into Dense Encoders: Bridging the Accuracy-Efficiency Gap in Recommendation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing distillation approaches target Small Language Models (SLMs) or Conventional Recommendation Models, but face a critical trade-off between computational cost and semantic reasoning capacity. |
| Approach: | They propose a framework that establishes a text encoder as the optimal student architecture for scalable recommendation. |
| Outcome: | Experiments on four datasets show that the proposed framework outperforms state-of-the-art models and achieves significantly reduced latency. |
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained language models focuses on reducing the size of these models into shallow ones. |
| Approach: | They propose a knowledge distillation technique that leverages teacher internal representations to reduce the size of pre-trained language models. |
| Outcome: | The proposed method outperforms previous methods in multilingual Named Entity Recognition (NER) it reduces the size of teacher models by 35x while retaining 95% of its F1 score. |
MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Pre-trained Language Models (LMs) are an integral part of natural language processing but their usability is constrained by computational and time complexity and their increasing size. |
| Approach: | They propose a technique for converting knowledge of fully parameterised LMs into a compact recursive student. |
| Outcome: | The proposed models match the performance of bloated models with negligible performance losses. |