Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Instruction tuning aims to align large language models (LLMs) with open-domain instructions and human-preferred responses. |
| Approach: | They propose a multi-round distillation framework that uses an oracle LLM to select instructions that are difficult for a student LLM. |
| Outcome: | The proposed framework outperforms large language models and user-tuned models on several widely recognized benchmarks and multiple student LLMs. |
Similar Papers
DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Models (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing studies on distilled lightweight LLMs have focused on transferring knowledge from a larger model (the teacher) to a smaller model (sector). |
| Approach: | They propose a family of distilled, lightweight LLMs derived from Qwen2.5 models. |
| Outcome: | Experimental results show that the distilled models have significantly stronger instruction-following capabilities than the original models. |
Demystifying Instruction Mixing for Fine-tuning Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Instruction tuning is effective for aligning large language models with human instructions, but the procedure to optimizing the mixing of instruction datasets is still unclear. |
| Approach: | They categorize instructions into three primary types: NLP downstream tasks, coding, and general chat. |
| Outcome: | The proposed method improves performance of large language models (LLMs) but it is difficult to combine different instruction datasets to optimize overall performance. |
LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions (2024.eacl-long)
Copied to clipboard
| Challenge: | Large language models with instruction tuning are resource-intensive . a recent study suggests that the performance of LLMs scales proportionally with the size of the model. |
| Approach: | They propose to distill knowledge from instruction-tuned LLMs into much smaller ones . they develop a large set of 2.58M instructions based on existing and newly-generated instructions . |
| Outcome: | The proposed models are comparable to strong baselines while being much smaller in size. |
FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation (2024.emnlp-main)
Copied to clipboard
KaShun Shum, Minrui Xu, Jianshu Zhang, Zixin Chen, Shizhe Diao, Hanze Dong, Jipeng Zhang, Muhammad Raza
| Challenge: | Experimental results show that a well-calibrated model is more reliable than a fine-tuned model due to “tuning-induced mis-calibration”. |
| Approach: | They propose a method which utilizes a small portion of teacher’s knowledge to obtain a reliable language model in a cost-efficient way. |
| Outcome: | The proposed method reduces the computational burden by utilizing teacher's knowledge to obtain a reliable language model in a cost-efficient way. |
Calibrated Progressive Distillation: Co-Designing Curriculum and Target Mixing for Knowledge Distillation of Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation address the teacher–student capacity gap by mixing teacher and student distributions in the distillation target or using curriculum learning to sequence training from easy to hard examples. |
| Approach: | They propose a white-box KD framework that co-designs curriculum scheduling and target mixing through a unified difficulty-aware principle. |
| Outcome: | The proposed framework outperforms existing methods while reducing training runtime by over 10%. |
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023.findings-acl)
Copied to clipboard
Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister
| Challenge: | Deploying large language models (LLMs) is difficult because they are memory inefficient and compute-intensive for practical applications. |
| Approach: | They propose a mechanism that fine tunes or distills small models that outperform LLMs . they use human labels to fine tune models or LLM-generated labels to train models . |
| Outcome: | The proposed method outperforms LLMs by using fewer training examples compared to few-shot prompted models using substantially smaller model sizes. |
Lion: Adversarial Distillation of Proprietary Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for transferring knowledge from a proprietary large language model to an open-source LLM have been criticized for not incorporating any feedback to boost student model's proficiency iteratively. |
| Approach: | They propose a novel adversarial distillation framework that integrates feedback to identify challenging instructions where the student model’s performance falls short. |
| Outcome: | The proposed framework surpasses existing state-of-the-art LLMs by 55.4% in challenging zero-shot reasoning benchmarks and 16.7% on AGIEval. |
Staged Knowledge Distillation Through Least-to-Most Prompting: Optimizing Teacher Guidance via Difficulty-Aware Training (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) enables the compression of large language models (LLMs) conventional methods suffer from training-inference mismatches and suboptimal performance due to expensive student-generated outputs. |
| Approach: | They propose a method that combines a CL strategy and adaptive loss design to reduce training mismatches and suboptimal performance. |
| Outcome: | L2M-KD outperforms existing white-box KD methods on instruction-following tasks . it outperformed existing methods, achieving superior student model performance with reduced overhead . |
Summarize-Exemplify-Reflect: Data-driven Insight Distillation Empowers LLMs for Few-shot Tabular Classification (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show the promise of large language models for few-shot tabular classification but highlight challenges due to the variability in structured data. |
| Approach: | They propose a framework that distills data into actionable insights to enable robust and effective classification by large language models. |
| Outcome: | The proposed framework integrates rule summarization, strategic exemplification, and insight reflection through deep collaboration between LLMs and data modeling techniques. |
Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a promising solution for large language models, but their deployment remains computationally expensive. |
| Approach: | They propose a framework which iteratively balances training data within a fixed computational budget and enables the transfer of knowledge from expensive teacher LLMs to smaller student models. |
| Outcome: | The proposed framework achieves state-of-the-art performance across diverse long-tailed datasets, enhancing both the efficiency and efficacy of the distilled models. |