BayesKD: Bayesian Knowledge Distillation for Compact LLMs in Constrained Fine-tuning Scenarios (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized various domains with their remarkable capabilities, but their massive parameter sizes pose significant challenges for fine-tuning and inference. |
| Approach: | They propose a Bayesian Knowledge Distillation framework for compact Large Language Models in resource-constrained fine-tuning scenarios that employs Logits Dual-Scaling, Knowledge Alignment Module, and Bayes Distillations Optimization. |
| Outcome: | The proposed framework outperforms baseline methods on various state-of-the-art LLMs, including LLaMA, Qwen2, Bloom, and Vicuna. |
Similar Papers
Streamlining LLMs: Adaptive Knowledge Distillation for Tailored Language Models (2025.naacl-srw)
Copied to clipboard
| Challenge: | Large language models (LLMs) have transformative potential across industries, e.g., enhancing customer service, revolutionizing medical diagnostics, or identifying crises in news articles. |
| Approach: | They propose to distill compact, parameter-efficient tailored language models from LLMs for domain-specific tasks with comparable performance. |
| Outcome: | The proposed framework outperforms knowledge distillation frameworks in the crisis domain, where labeled data is scarce. |
Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) face high computational demands at inference time due to high computational costs. |
| Approach: | They propose a cost-effective and high-throughput solution for large language models . PGKD distills the knowledge of LLMs into smaller, task-specific models based on teacher-student knowledge distillation . |
| Outcome: | PGKD outperforms BERT-based models and other knowledge distillation methods on multi-class classification datasets. |
ToDi: Token-wise Distillation via Fine-Grained Divergence Control (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) offer impressive performance but are impractical for resource-constrained deployment due to high latency and energy consumption. |
| Approach: | They propose a method that adaptively combines FKL and RKL per token using a sigmoid-based weighting function derived from the teacher-student probability log-ratio. |
| Outcome: | The proposed method outperforms baselines using uniform or less granular strategies across instruction-following benchmarks. |
Dual-Space Knowledge Distillation for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) have strong generalization abilities due to their huge model capacities. |
| Approach: | They propose a dual-space knowledge distillation framework that unifies the output spaces of the two models for KD. |
| Outcome: | The proposed framework outperforms existing white-box KD frameworks on task-agnostic instruction-following benchmarks and can automatically align representations of two models with different vocabularies. |
Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs (2025.acl-long)
Copied to clipboard
Anshumann Anshumann, Mohd Abbas Zaidi, Akhil Kedia, Jinwoo Ahn, Taehwak Kwon, Kangwook Lee, Haejun Lee, Joohyung Lee
| Challenge: | Knowledge distillation is a cost-effective technique to distill knowledge in Large Language Models, if the teacher output logits can be pre-computed and cached. |
| Approach: | They propose an importance-sampling-based method which provides unbiased estimates, preserves the gradient in expectation, and requires storing significantly sparser logits. |
| Outcome: | The proposed method enables faster training of student models with marginal overhead (10%) compared to cross-entropy based training, while maintaining competitive performance compared with full distillation. |
FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation (2024.emnlp-main)
Copied to clipboard
KaShun Shum, Minrui Xu, Jianshu Zhang, Zixin Chen, Shizhe Diao, Hanze Dong, Jipeng Zhang, Muhammad Raza
| Challenge: | Experimental results show that a well-calibrated model is more reliable than a fine-tuned model due to “tuning-induced mis-calibration”. |
| Approach: | They propose a method which utilizes a small portion of teacher’s knowledge to obtain a reliable language model in a cost-efficient way. |
| Outcome: | The proposed method reduces the computational burden by utilizing teacher's knowledge to obtain a reliable language model in a cost-efficient way. |
MCC-KD: Multi-CoT Consistent Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable abilities in complex reasoning through chain of thought (CoT) prompting. |
| Approach: | They propose to generate multiple rationales for each question and enforce consistency among their predictions by minimizing the bidirectional KL-divergence between the answer distributions. |
| Outcome: | The proposed model achieves superior performance on in-distribution and commonsense reasoning benchmarks. |
Staged Knowledge Distillation Through Least-to-Most Prompting: Optimizing Teacher Guidance via Difficulty-Aware Training (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) enables the compression of large language models (LLMs) conventional methods suffer from training-inference mismatches and suboptimal performance due to expensive student-generated outputs. |
| Approach: | They propose a method that combines a CL strategy and adaptive loss design to reduce training mismatches and suboptimal performance. |
| Outcome: | L2M-KD outperforms existing white-box KD methods on instruction-following tasks . it outperformed existing methods, achieving superior student model performance with reduced overhead . |
PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have raised concerns about inference costs, increasing the need for research into model compression. |
| Approach: | They propose a method that utilizes prompt tuning to enable generative language models to transfer student-friendly knowledge. |
| Outcome: | Extensive experiments on instruction-following datasets show that PromptKD achieves state-of-the-art performance while adding only 0.0007% of the teacher’s parameters as prompts. |
BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation (2025.coling-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a method for reducing model size while preserving performance. |
| Approach: | They propose a method to distill large language models at the logit level by transferring knowledge from a large teacher model to a smaller student model. |
| Outcome: | The proposed method outperforms supervised fine-tuning, vanilla KL loss and five other distillation methods on 13 datasets. |