| Challenge: | Large language models require high computational resources which limits their deployment in real-world applications. |
| Approach: | They propose to distill large language models into smaller language models by either knowledge distillation or task distillation. |
| Outcome: | The proposed model outperforms or performs comparable to over 20x bigger LLMs on language inference benchmarks and BIG-bench tasks. |
Similar Papers
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation (2024.emnlp-main)
Copied to clipboard
KaShun Shum, Minrui Xu, Jianshu Zhang, Zixin Chen, Shizhe Diao, Hanze Dong, Jipeng Zhang, Muhammad Raza
| Challenge: | Experimental results show that a well-calibrated model is more reliable than a fine-tuned model due to “tuning-induced mis-calibration”. |
| Approach: | They propose a method which utilizes a small portion of teacher’s knowledge to obtain a reliable language model in a cost-efficient way. |
| Outcome: | The proposed method reduces the computational burden by utilizing teacher's knowledge to obtain a reliable language model in a cost-efficient way. |
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023.findings-acl)
Copied to clipboard
Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister
| Challenge: | Deploying large language models (LLMs) is difficult because they are memory inefficient and compute-intensive for practical applications. |
| Approach: | They propose a mechanism that fine tunes or distills small models that outperform LLMs . they use human labels to fine tune models or LLM-generated labels to train models . |
| Outcome: | The proposed method outperforms LLMs by using fewer training examples compared to few-shot prompted models using substantially smaller model sizes. |
Distilling Rule-based Knowledge into Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have broadened their applicability across diverse realworld scenarios. |
| Approach: | They propose to encode rule-based knowledge into large language models by using strong in-context abilities to extract the knowledge from the textual rules and then explicitly encode it into the parameters of LLMs. |
| Outcome: | The proposed learning paradigm is much more efficient than example-based learning in both sample size and generalization ability. |
Effective Distillation of Table-based Reasoning Ability from LLMs (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing work on table-based reasoning distillation has focused on smaller models with limited performance. |
| Approach: | They propose a table-based reasoning distillation approach to distill LLMs into smaller models . their results show that a 220 million parameter model fine-tuned using distilled data improves performance . |
| Outcome: | The proposed model improves on a scientific table-to-text generation dataset and surpasses specific LLMs. |
On the Generalization vs Fidelity Paradox in Knowledge Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a key technique for compressing large language models into smaller ones while preserving performance. |
| Approach: | They propose to use knowledge distillation to compress large language models into smaller ones while preserving performance. |
| Outcome: | The proposed technique improves the performance of smaller models by 10% while providing only marginal benefits for larger models. |
Pre-training Distillation for Large Language Models: A Design Space Exploration (2025.acl-long)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a large teacher model to a smaller student model for model compression. |
| Approach: | They extend knowledge distillation to the pre-training phase of large language models . they first conduct an experiment using a teacher LLM to distill a 1.9B student LLM . |
| Outcome: | The proposed model can be used to distill a 1.9B student model using a teacher LLM. |
Beyond One-Step Distillation: Bridging the Capacity Gap in Small Language Models via Multi-Step Knowledge Transfer (2026.eacl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel across diverse tasks but remain too large for efficient on-device deployment. |
| Approach: | They revisit multi-step knowledge distillation as an effective remedy . they demonstrate that MSKD improves ROUGE-L and perplexity over single-step approaches . |
| Outcome: | The proposed approach improves ROUGE-L and perplexity over single-step approaches . large language models are too large for efficient on-device deployment, the authors show . |
Cost-effective Distillation of Large Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods require pretraining of the teacher on task-specific datasets, which can be costly for large and unstable for small datasets. |
| Approach: | They propose an approach to improve knowledge distillation by a loss-agnostic approach to task and model architecture. |
| Outcome: | The proposed method achieves competitive results across a range of tasks, especially for tasks with smaller datasets. |
LLMR: Knowledge Distillation with a Large Language Model-Induced Reward (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models have demonstrated remarkable performance in various NLP tasks, but are typically computationally expensive and difficult to be deployed in resource-constrained environments. |
| Approach: | They propose a knowledge distillation method based on a reward function induced from large language models. |
| Outcome: | The proposed method outperforms traditional methods on multiple datasets and tasks. |