Flipping Knowledge Distillation: Leveraging Small Models’ Expertise to Enhance LLMs in Text Matching (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities in acquiring diverse knowledge, making them highly effective across a wide range of tasks. |
| Approach: | They propose a flipped knowledge distillation paradigm where LLM learns from SLM . they propose to reinterpret LLMs as encoder-decoder models using LoRA . |
| Outcome: | The proposed model has been deployed in an online application environment and validated on financial and healthcare benchmarks and real-world applications. |
Similar Papers
QCRD: Quality-guided Contrastive Rationale Distillation for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on smaller, task-specific models enhanced by distilling knowledge from LLMs, but the diversity and quality of negative knowledge remains understudied. |
| Approach: | They propose a quality-guided contrastive rationale distillation framework that aims to enhance reasoning capabilities through contrastive knowledge learning. |
| Outcome: | The proposed method consistently outperforms existing distillation techniques yielding higher-quality rationales. |
Beyond One-Step Distillation: Bridging the Capacity Gap in Small Language Models via Multi-Step Knowledge Transfer (2026.eacl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel across diverse tasks but remain too large for efficient on-device deployment. |
| Approach: | They revisit multi-step knowledge distillation as an effective remedy . they demonstrate that MSKD improves ROUGE-L and perplexity over single-step approaches . |
| Outcome: | The proposed approach improves ROUGE-L and perplexity over single-step approaches . large language models are too large for efficient on-device deployment, the authors show . |
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023.findings-acl)
Copied to clipboard
Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister
| Challenge: | Deploying large language models (LLMs) is difficult because they are memory inefficient and compute-intensive for practical applications. |
| Approach: | They propose a mechanism that fine tunes or distills small models that outperform LLMs . they use human labels to fine tune models or LLM-generated labels to train models . |
| Outcome: | The proposed method outperforms LLMs by using fewer training examples compared to few-shot prompted models using substantially smaller model sizes. |
Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning (2024.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that fine-tuning of large language models for specific tasks can be challenging . distribution shift during fine-timing can lead to performance degradation in general task capabilities . |
| Approach: | They propose a new approach that bridges the distribution gap between task datasets and LLMs by guiding fine-tuning with a distilled dataset generated by the model itself. |
| Outcome: | The proposed approach achieves comparable or superior performance on downstream tasks compared to the vanilla approach. |
Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel on English NLU tasks, yet struggle to extend their NLU capabilities to underrepresented languages. |
| Approach: | They integrate machine translation models (MT) directly into LLM backbones via sample-efficient self-distillation. |
| Outcome: | The proposed model outperforms translation-test models on 127 low-resource languages. |
Balancing Forget Quality and Model Utility: A Reverse KL-Divergence Knowledge Distillation Approach for Better Unlearning in LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for unlearning large language models struggle with forget quality and model utility, leading to over-unlearning or partial unlearning. |
| Approach: | They propose a method that uses reverse KL-divergence based knowledge distillation for unlearning to achieve significant forget quality while maintaining model utility. |
| Outcome: | The proposed method outperforms existing methods in forget quality and model utility with larger unlearning datasets. |
Effective Distillation of Table-based Reasoning Ability from LLMs (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing work on table-based reasoning distillation has focused on smaller models with limited performance. |
| Approach: | They propose a table-based reasoning distillation approach to distill LLMs into smaller models . their results show that a 220 million parameter model fine-tuned using distilled data improves performance . |
| Outcome: | The proposed model improves on a scientific table-to-text generation dataset and surpasses specific LLMs. |
Distilling Rule-based Knowledge into Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have broadened their applicability across diverse realworld scenarios. |
| Approach: | They propose to encode rule-based knowledge into large language models by using strong in-context abilities to extract the knowledge from the textual rules and then explicitly encode it into the parameters of LLMs. |
| Outcome: | The proposed learning paradigm is much more efficient than example-based learning in both sample size and generalization ability. |
Streamlining LLMs: Adaptive Knowledge Distillation for Tailored Language Models (2025.naacl-srw)
Copied to clipboard
| Challenge: | Large language models (LLMs) have transformative potential across industries, e.g., enhancing customer service, revolutionizing medical diagnostics, or identifying crises in news articles. |
| Approach: | They propose to distill compact, parameter-efficient tailored language models from LLMs for domain-specific tasks with comparable performance. |
| Outcome: | The proposed framework outperforms knowledge distillation frameworks in the crisis domain, where labeled data is scarce. |
LLMR: Knowledge Distillation with a Large Language Model-Induced Reward (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models have demonstrated remarkable performance in various NLP tasks, but are typically computationally expensive and difficult to be deployed in resource-constrained environments. |
| Approach: | They propose a knowledge distillation method based on a reward function induced from large language models. |
| Outcome: | The proposed method outperforms traditional methods on multiple datasets and tasks. |